abhishekd0907 commented on issue #24934: [SPARK-28124] [SS] SQS source for Structured Streaming URL: https://github.com/apache/spark/pull/24934#issuecomment-519208775 > > @steveloughran I'm not sure if I should use hadoop 2.9. Spark parent already uses 2.7 as default hadoop version. Adding hadoop 2.9 as a dependency will increase the work of excluding / shading classes to avoid classpath conflicts & seems risky to me. Is it possible for you to backport the changes you want to hadoop 2.7 ? > > 2.7 is EOL, 2.8 is going to exactly one release as soon as we can find a volunteer and that's it tool. Neither are even qualified to work on Java 8 properly, so they need to be retired. It is time to move on. Sorry. > > Bahir sounds like a home; I'll help review patches there. Make it all conditional on recent Hadoop 3.x+ and I can expose the relevant credential accessor there, maybe even think of a way of getting the AssumedRole delegation tokens to work with this too. @steveloughran Making it all conditional on Hadoop 3 will prevent Spark users who currently use default Hadoop version (2.7) from using it. We can make it all conditional on Hadoop 3 with Spark 3.0 onwards when Hadoop 3 becomes the default Hadoop Version for Spark. Right now, I would like my code to work for all spark users, even those who use spark with the default Hadoop version. You can mention your other concerns on the [BAHIR pull request](https://github.com/apache/bahir/pull/91)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
