mgaido91 commented on issue #24785: [SPARK-27937][CORE] Revert partial logic for auto namespace discovery URL: https://github.com/apache/spark/pull/24785#issuecomment-500478249 Answering to your points: > If the data is stored on a different cluster, this solution doesn't work. Well, that's quite expected since Spark runs on a specific hadoop cluster. For that case, you still have to do it manually, as before the patch you'd like to revert. So, no change on this. > We now get tokens for namespaces which the user isn't going to read/write from. As you mentioned, there is also a configuration for that. So if you want, you can limit your app to get only the tokens you need. > If hadoop already figures out getting tokens for different namespaces using viewfs, that is a better choice. Yes, but this is not related with Spark code and this patch, it is rather related to how you configure and use your cluster. You're definitely allowed to use viewfs and no extra token is taken in that case... > It fails to launch spark with existing HDFS deployments while trying to create a path from NameServiceIDs. Then this is a bug, if you have a fix for this, that's great.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
