dhruve commented on issue #24785: [SPARK-27937][CORE] Revert partial logic for auto namespace discovery URL: https://github.com/apache/spark/pull/24785#issuecomment-500053084 I have updated the jira with the use case where the current behavior of spark fails to launch an application. - If you get a token from a specific NN, it is valid with or without HA. HDFS takes care of this. I don't think that Spark needs to do this explicitly. May be an hdfs expert can give a better perspective. cc @kihwal @daryn-sharp - From the discussion in SPARK-24149, the reason why this change was introduced seems to address a very specific use case (while it is common to have tables and partitions across different namespaces, this one addresses a very specific use case: where partitions are stored across unrelated namespaces, using federation and luckily on the same cluster ) which I am not sure of is very common. > the use case for this change, for instance, is when you have a partitioned table, when the partitions are on different namespaces and there is no viewFS configured
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
