steveloughran commented on issue #24934: [SPARK-28124] [SS] SQS source for Structured Streaming URL: https://github.com/apache/spark/pull/24934#issuecomment-517849008 > @steveloughran I'm not sure if I should use hadoop 2.9. Spark parent already uses 2.7 as default hadoop version. Adding hadoop 2.9 as a dependency will increase the work of excluding / shading classes to avoid classpath conflicts & seems risky to me. Is it possible for you to backport the changes you want to hadoop 2.7 ? 2.7 is EOL, 2.8 is going to exactly one release as soon as we can find a volunteer and that's it tool. Neither are even qualified to work on Java 8 properly, so they need to be retired. It is time to move on. Sorry. Bahir sounds like a home; I'll help review patches there. Make it all conditional on recent Hadoop 3.x+ and I can expose the relevant credential accessor there, maybe even think of a way of getting the AssumedRole delegation tokens to work with this too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
