Hello We have restarted our work on updated KNOX 1.4.0 to KNOX 1.6.1 in IBM Hadoop Execution Engine. Although we are able to run the gateway service comfortably, we are not able to use the NGINX re-routes that we used to do earlier. I will try to explain this below:
HEE: Hadoop Execution Engine (manages remote Hadoop clusters. Supports running remote jobs, running Spark sessions using LIVY or JEG, remote HDFS connection, remote HIVE connections, etc.) CPD: IBM Cloud Pak for Data (IBM Machine Learning Platform that integrates with other IBM addon services like HEE) Now, in order to run HEE with CPD, we have to undergo a registration process that enables a CPD cluster to recognize HEE on a remote Hadoop Cluster. As part of this process, we rewrite the KNOX gateway URL using NGINX and following this, every CURL to the KNOX GATEWAY in HEE is done using the rewritten NGINX URL. This is where we see a problem. We also send some headers along with the CURL request to this rewritten NGINX URL and this used to work fine until KNOX 1.4.0. Now, what I suspect is that there has been some change in KNOX 1.6.1 on a granular level in the way headers are handled and this I say based on: https://github.com/apache/knox/pull/374/files This is a critical upgrade for us and any help from your side will be very much appreciated. I am also available for a quick 15-30 minute call where I can demo the issue to anyone who can help. Please let me know if there is anything else needed from my side to understand the HEE structure or the way KNOX is used in it. PS. I am not sure if I am subscribed to the mailing list. Could use some help confirming that. Regards. Akshay Kotecha Jain (AK) Senior Software Developer, IBM Cloud -- Regards Akshay Kotecha Jain
