kskalski commented on issue #4841: Issues wth Hadoop batch indexing using S3A in 0.10.1 and hadoop client 2.7.3 URL: https://github.com/apache/incubator-druid/issues/4841#issuecomment-414960119 FIY, I tried using hadoop-aws.jar from Hadoop 2.9 release, but it required a lot of other jars to be replaced, both in hadoop-dependencies and from Druid's extensions/druid-hdfs-storage/ (e.g. aws-java-sdk used by hadoop's 2.9 is already newer than Druid 0.12.1's version, so I had to make both directories converge). Ultimately S3A started working, but the way to achieve it was quite messy. Not sure if shading would work, I couldn't build the custom jar, because Hadoop 2.7 requires dependencies (protoc) that are no longer available on OS at my disposal. For now I settled with using S3N, but it forces provising AWS secret credentials in plain text within config / task spec, so it's a pain.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
