I greatly appreciate your guidance. Now all the tests pass well! Thanks, Dongjin
On Thu, Jul 26, 2018 at 3:27 AM Jihoon Son <ghoon...@gmail.com> wrote: > Hi Dongjin, > > as Charles said, you should set the region for aws sdk. Here is a snippet > of our Travis configuration. > > > # other modules test > > - sudo: false > > env: > > - NAME="other modules test" > > - AWS_REGION=us-east-1 # set a aws region for unit tests > > install: echo "MAVEN_OPTS='-Xmx3000m'" > ~/.mavenrc && mvn install > -q -ff -DskipTests -B > > before_script: > > - unset _JAVA_OPTIONS > > script: echo "MAVEN_OPTS='-Xmx512m'" > ~/.mavenrc && mvn test -B > -Pparallel-test -Dmaven.fork.count=2 -pl '!processing,!server' > > Jihoon > > On Wed, Jul 25, 2018 at 8:29 AM Charles Allen > <charles.al...@snap.com.invalid> wrote: > > > IMHO the best way to do it would be to setup the test so that it sets the > > region just for the test, and let production systems use whichever region > > provider fits the way they typically do region configs. That way people > > running the tests don't have to worry about anything regarding how to > setup > > a region, and people deploying cloud services don't have surprise > behavior > > for Druid compared to other usages of the java sdk. > > > > On Tue, Jul 24, 2018 at 9:48 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > Hello. I encountered a problem building druid. *In short, > > > `TestAWSCredentialsProvider` fails like the following*: > > > > > > ``` > > > mvn -pl extensions-core/s3-extensions test > > > > > > ... > > > > > > > > > > > > testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider) > > > Time elapsed: 6.615 sec <<< ERROR! > > > com.amazonaws.SdkClientException: Unable to find a region via the > region > > > provider chain. Must provide an explicit region in the builder or setup > > > environment to supply a region. > > > at > > > > > > > > > io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98) > > > > > > testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider) > > Time > > > elapsed: 4.345 sec <<< ERROR! > > > com.amazonaws.SdkClientException: Unable to find a region via the > region > > > provider chain. Must provide an explicit region in the builder or setup > > > environment to supply a region. > > > at > > > > > > > > > io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67) > > > ``` > > > > > > After digging the code, I found following: > > > > > > 1. `S3StorageDruidModule` does not provide a way to explictily set > > > `Region`, but detects the region automatically with its default > > > credential/region provider chain. > `S3StorageDruidModule#getAmazonS3Client > > > -> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;` > > > 2. The default region provider chain[^1][^2] tries to determine region > in > > > following orders:[^1][^2] > > > a. Environment Variable[^5] > > > b. JVM property[^6] > > > c. AWS Configuration[^7] > > > d. EC2 instance metadata service[^8] > > > > > > If all of the above fails, it throws an Exception. > > > > > > *In short, there is a possibility that `S3StorageDruidModule` can't > > > determine the Region. It would be better to provide a way to set Region > > > explicitly.* > > > > > > How do you think? > > > > > > Best, > > > Dongjin > > > > > > [^1]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java > > > [^2]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java > > > [^3]: > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v2_developer-2Dguide_java-2Ddg-2Dregion-2Dselection.html&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=Ro8j9Jj9lCbFbc0aYDNjwtZuD2a_9fPaYw1lOp7ZQeQ&e= > > > [^4]: The example client in documentation is `AmazonEC2ClientBuilder` > but > > > it also applies to the other clients like `AmazonS3ClientBuilder`. > > > [^5]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java > > > [^6]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java > > > [^7]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java > > > [^8]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > *github: < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=VLpmOvpUoEqDfaI8jAV3WKsttHGasxSzCwmAE-we-CU&e= > > > >github.com/dongjinleekr > > > <http://github.com/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=LnjNxyq941KadHM075fCkotIkN-U4USFaqm2JsKi08Q&e= > > > >slideshare: > > > www.slideshare.net/dongjinleekr > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=hoP7U3GI-oMiDEdyIocsIkBAoupeT_QQ52PBHlEeoUQ&e= > > > >* > > > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > *github: <http://goog_969573159/>github.com/dongjinleekr > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > www.slideshare.net/dongjinleekr > <http://www.slideshare.net/dongjinleekr>* >
