Re: Documentation for maths operations between different types?
Hi , Can someone help me guide through to start with update the Drill documentation by adding the missing functions ? Appreciate the pointers / link to update the Drill doc and how can I gather all the information and create the JIRA for it ? Thanks, Divya On Sat, 17 Aug 2019 at 01:06, Paul Rogers wrote: > Hi Divya, > > Would be great to add to the docs. First step is to gather the info in a > JIRA ticket. Then, someone (thanks for volunteering) can update the various > tables in the docs. > > I believe that the documentation is generated via some process other than > a direct edit of the MD files in the repo. Perhaps someone can fill in the > details about how to update the material. > > When writing the Learning Apache Drill book, Charles found that many > functions are not documented in the "official" documentation. Would be > great to review the function tables in the book and apply any missing > information to the Drill documentation. > > Thanks, > - Paul > > > > On Friday, August 16, 2019, 08:25:14 AM PDT, Divya Gehlot < > divinediv...@gmail.com> wrote: > > Can we add this as part of Drill Function docs , will be useful for other > ppl as well ? > I can help to add , unsure if I have access to edit Drill doc > > Thanks , > Divya > > On Fri, 16 Aug 2019 at 01:04, Paul Rogers > wrote: > > > Hi Dave, > > > > As it turns out, improving the detail in function documentation is a > > long-standing request. The historical answer has been to either 1) read > the > > code, or 2) try it with a test query. > > > > You can use the sqlTypeOf() function to learn the answer to your > question: > > > > SELECT sqlTypeOf(cast(1 AS INT) / cast(2 AS INT)) FROM values(1) > > > > Charles Givre patiently tracked down and documented all the Drill > > functions in his appendix to our book "Learning Apache Drill." But, even > > there, the level of detail you request is missing. > > > > Maybe, once you do the research to find the answers you want, you could > > submit a Documentation JIRA ticket with the results so that it can be > added > > to the documentation. > > > > Thanks, > > - Paul > > > > > > > >On Thursday, August 15, 2019, 03:55:11 AM PDT, Dave Challis < > > dave.chal...@cipher.ai> wrote: > > > > Is there any documentation out there on how mathematical functions are > > handled when operating on different types? > > > > E.g.: > > > > * would integer division of 1 / 2 produce a float or double of 0.5? Or an > > integer of the same type set to 0? > > > > * if two INT are multipled and produce a result larger than INT can > > support, is the result returned as a BIGINT? > > >
Column alias in group by behavior in 1.16
Hi folks, we are testing an upgrade to 1.16 from 1.14 and noticed a behavior change in 1.16 related to the support for allowing select column alias in group by clause. When there is an ambiguity to the alias name, Drill 1.16 is exhibiting a different behavior comparing to expectation. Here is an example to illustrate: *-- Test query 1:* select *MOD(x,10) as x * from (select 14 *as x* UNION ALL select 4 *as x*) TableA *group by x; --- x refers to select alias from column name?* --- returns 2 rows if x refers to from clause table column x --- returns 1 row if x refers to select column alias x *-- Test query 2:* select *MOD(x,10) as x * from (select 14 *as x* UNION ALL select 4 *as x*) TableA *group by MOD(x,10); ** --- x refers to select alias from column name?* --- returns 1 row as x here should refer to the from clause table column x *PostgreSQL 9.6* Test Query 1 --> 2 rows (==> x refers to from clause table column) Test Query 2 --> 1 row (==> x refers to from clause table column) *Drill 1.14 -- same as PostgreSQL 9.6* Test Query 1 --> 2 rows (==> x refers to from clause table column) Test Query 2 --> 1 row (==> x refers to from clause table column) *Drill 1.16 -- different results:* Test Query 1 --> 1 row (==> x refers to select column alias) Test Query 2 --> SQL Error: VALIDATION ERROR: At line 1, column 12: Expression 'TableA.x' is not being grouped (==> also implies that x refers to column alias) Given the logical order of processing for a SQL statement: from --> where --> group by --> ... --> select, ... it would seem that precedence order should be given to the from clause in case of ambiguity. Also to be consistent with test query 2, the alias in group by also refer to from clause before select clause. Love to hear more expert opinions here on this issue. -- Jiang
Hangout Discussion Topics for 08-20-2019
Hello Everyone, Does anyone have any topics for tomorrow's hangout? We will start the hangout at 10 AM PST (link http://meet.google.com/yki-iqdf-tai). Thank you, Jyothsna
Apache Drill Hangout - Aug 20, 2019
Hi Drillers, We will have our bi-weekly hangout tomorrow Aug 20th, at 10 AM PST (link: https://meet.google.com/yki-iqdf-tai ). If there are any topics you would like to discuss during the hangout please respond to this email. Thanks, Gautam
Re: EMC ECS Configuration with Apache Drill
On Mon, Aug 19, 2019 at 11:33 AM Prabu Mohan wrote: > but i am able to connect to ECS via python using boto3 libraries without > any issues, I am able to write files to the bucket and read them back .. > > not sure why i am facing issues with drill though with the same credentials > The key here is your assumption that the same credentials are being passed through Drill to AWS and that there isn't some other consideration that keeps S3 from believing whatever credentials it is getting. That assumption has to be attacked by figuring out experiments that can prove or disprove aspects of it. For instance, if you can get a proxy in the middle of the connection, you should be able to see *exactly* what is on the wire. Likewise if you can get better logging out of Drill.
Re: EMC ECS Configuration with Apache Drill
but i am able to connect to ECS via python using boto3 libraries without any issues, I am able to write files to the bucket and read them back .. not sure why i am facing issues with drill though with the same credentials On Mon, Aug 19, 2019 at 11:53 PM Ted Dunning wrote: > So that looks like the security tokens aren't getting through to AWS > correctly. > > On Mon, Aug 19, 2019 at 11:21 AM Prabu Mohan > wrote: > > > Log info > > > > 2019-08-19 16:23:05,439 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id > > 22a54125-e1e7-29f6-a146-6cfd529b23d1 issued by anonymous: use ecstest > > > > 2019-08-19 16:23:06,691 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] > > ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: > > AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request > > ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error > Message: > > Method Not Allowed > > > > > > > > > > > > Please, refer to logs for more information. > > > > > > > > [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] > > > > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > > AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request > > ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error > Message: > > Method Not Allowed > > > > > > > > > > > > Please, refer to logs for more information. > > > > > > > > [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] > > > > at > > > > > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > > ~[drill-common-1.16.0.jar:1.16.0] > > > > at > > org.apache.drill.exec.work > > .foreman.Foreman$ForemanResult.close(Foreman.java:789) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > > at > > org.apache.drill.exec.work > > > .foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:325) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > > at > > org.apache.drill.exec.work > > .foreman.QueryStateProcessor.planning(QueryStateProcessor.java:221) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > > at > > org.apache.drill.exec.work > > .foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > > at org.apache.drill.exec.work > > .foreman.Foreman.run(Foreman.java:304) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > >at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [na:1.8.0_92] > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [na:1.8.0_92] > > > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92] > > > > Caused by: org.apache.drill.exec.work.foreman.ForemanException: > Unexpected > > exception during fragment initialization: Status Code: 405, AWS Service: > > Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: > > null, AWS Error Message: Method Not Allowed > > > > at org.apache.drill.exec.work > > .foreman.Foreman.run(Foreman.java:305) > > [drill-java-exec-1.16.0.jar:1.16.0] > > > > ... 3 common frames omitted > > > > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status > Code: > > 405, AWS Service: Amazon S3, AWS Request ID: > a1552153:16baf0e3c53:351c7:13, > > AWS Error Code: null, AWS Error Message: Method Not Allowed > > > > at > > > > > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > > > > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > > > > com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > > > > com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) > > ~[aws-java-sdk-1.7.4.jar:na] > > > > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) > > ~[hadoop-aws-2.7.4.jar:na] > > > > at > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) > > ~[hadoop-common-2.7.4.jar:na] > > > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) > > ~[hadoop-common-2.7.4.jar:na] > > > > at > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) > > ~[hadoop-common-2.7.4.jar:na] > > > > at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) > > ~[hadoop-common-2.7.4.jar:na] > > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) > >
Re: EMC ECS Configuration with Apache Drill
So that looks like the security tokens aren't getting through to AWS correctly. On Mon, Aug 19, 2019 at 11:21 AM Prabu Mohan wrote: > Log info > > 2019-08-19 16:23:05,439 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id > 22a54125-e1e7-29f6-a146-6cfd529b23d1 issued by anonymous: use ecstest > > 2019-08-19 16:23:06,691 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] > ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: > AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request > ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: > Method Not Allowed > > > > > > Please, refer to logs for more information. > > > > [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] > > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request > ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: > Method Not Allowed > > > > > > Please, refer to logs for more information. > > > > [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] > > at > > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > > at > org.apache.drill.exec.work > .foreman.Foreman$ForemanResult.close(Foreman.java:789) > [drill-java-exec-1.16.0.jar:1.16.0] > > at > org.apache.drill.exec.work > .foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:325) > [drill-java-exec-1.16.0.jar:1.16.0] > > at > org.apache.drill.exec.work > .foreman.QueryStateProcessor.planning(QueryStateProcessor.java:221) > [drill-java-exec-1.16.0.jar:1.16.0] > > at > org.apache.drill.exec.work > .foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83) > [drill-java-exec-1.16.0.jar:1.16.0] > > at org.apache.drill.exec.work > .foreman.Foreman.run(Foreman.java:304) > [drill-java-exec-1.16.0.jar:1.16.0] > >at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_92] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_92] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92] > > Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected > exception during fragment initialization: Status Code: 405, AWS Service: > Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: > null, AWS Error Message: Method Not Allowed > > at org.apache.drill.exec.work > .foreman.Foreman.run(Foreman.java:305) > [drill-java-exec-1.16.0.jar:1.16.0] > > ... 3 common frames omitted > > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: > 405, AWS Service: Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, > AWS Error Code: null, AWS Error Message: Method Not Allowed > > at > > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > ~[aws-java-sdk-1.7.4.jar:na] > > at > > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > ~[aws-java-sdk-1.7.4.jar:na] > > at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > ~[aws-java-sdk-1.7.4.jar:na] > > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > ~[aws-java-sdk-1.7.4.jar:na] > > at > > com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) > ~[aws-java-sdk-1.7.4.jar:na] > > at > > com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) > ~[aws-java-sdk-1.7.4.jar:na] > > at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) > ~[hadoop-aws-2.7.4.jar:na] > > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) > ~[hadoop-common-2.7.4.jar:na] > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) > ~[hadoop-common-2.7.4.jar:na] > > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) > ~[hadoop-common-2.7.4.jar:na] > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) > ~[hadoop-common-2.7.4.jar:na] > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) > ~[hadoop-common-2.7.4.jar:na] > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) > ~[hadoop-common-2.7.4.jar:na] > > at > > org.apache.drill.exec.store.dfs.DrillFileSystem.(DrillFileSystem.java:93) > ~[drill-java-exec-1.16.0.jar:1.16.0] > > at > > org.apache.drill.exec.util.ImpersonationUtil.lambda$createFileSystem$0(ImpersonationUtil.java:215) > ~[drill-java-exec-1.16.0.jar:1.16.0] > > at java.security.AccessController.doPrivileged(Native Method) > ~[na:1.8.0_92] > > at javax.security.auth.Subject.doAs(Subject.java:422) > ~[na:1.8.0_92] > >
Re: EMC ECS Configuration with Apache Drill
Log info 2019-08-19 16:23:05,439 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 22a54125-e1e7-29f6-a146-6cfd529b23d1 issued by anonymous: use ecstest 2019-08-19 16:23:06,691 [22a54125-e1e7-29f6-a146-6cfd529b23d1:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: Method Not Allowed Please, refer to logs for more information. [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: Method Not Allowed Please, refer to logs for more information. [Error Id: 76b95ca3-0ed7-4463-b3a1-7b0a8a21cd1a on ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) ~[drill-common-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:789) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:325) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:221) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:304) [drill-java-exec-1.16.0.jar:1.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_92] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_92] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92] Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during fragment initialization: Status Code: 405, AWS Service: Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: Method Not Allowed at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:305) [drill-java-exec-1.16.0.jar:1.16.0] ... 3 common frames omitted Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 405, AWS Service: Amazon S3, AWS Request ID: a1552153:16baf0e3c53:351c7:13, AWS Error Code: null, AWS Error Message: Method Not Allowed at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) ~[aws-java-sdk-1.7.4.jar:na] at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) ~[hadoop-aws-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) ~[hadoop-common-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) ~[hadoop-common-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) ~[hadoop-common-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) ~[hadoop-common-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) ~[hadoop-common-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) ~[hadoop-common-2.7.4.jar:na] at org.apache.drill.exec.store.dfs.DrillFileSystem.(DrillFileSystem.java:93) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.util.ImpersonationUtil.lambda$createFileSystem$0(ImpersonationUtil.java:215) ~[drill-java-exec-1.16.0.jar:1.16.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_92] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_92] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) ~[hadoop-common-2.7.4.jar:na] at org.apache.drill.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:213) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:205) ~[drill-java-exec-1.16.0.jar:1.16.0] at
Re: EMC ECS Configuration with Apache Drill
Did you see anything in any logs?
On Sun, Aug 18, 2019 at 10:16 PM Prabu Mohan wrote:
> I am able to connect to the http endpoint using boto3 from python (able to
> retrieve files/store files), from IE with https and port 9021 , it comes
> back with 403 Forbidden indicating that it was able to connect to website
> but does not have permission to view the webpage ( may be due to
> credentials ).
>
> Regards,
> Prabu
>
>
> On Mon, Aug 19, 2019 at 12:59 AM Sorabh Apache >
> wrote:
>
> > Are you able to use the same configured endpoint *http://:9020* from
> your
> > browser ?
> >
> > Thanks,
> >
> > Sorabh.
> >
> > On Mon, Aug 19, 2019 at 12:19 AM Prabu Mohan
> wrote:
>
> > I'm trying to configure Apache Drill with EMC ECS, it was quite easy
> > configuring with AWS S3 and GCP Cloud Storage, but I'm facing issues
> > configuring with EMC ECS,
> >
> > When I use http with port 9020 as endpoint I face this error
> >
> > Error:SYSTEM ERROR: AmazonS3Exception: Status Code: 405, AWS
> > Service:Amazon S3, AWS Request ID:, AWS Error Code:null, AWS Error
> Message:
> > Method Not Allowed
> >
> > core-site.xml
> >
> >
> >
> >
> > fs.s3a.access.key
> > accesskey
> >
> >
> >
> > fs.s3a.secret.key
> > secretkey
> >
> >
> >
> > fs.s3a.endpoint
> > http://:9020
> >
> >
> >
> > fs.s3a.connection.ssl.enabled
> > false
> >
> >
> >
> >
> > and for connecting to the bucket i use the following in the storage
> plugin
> >
> > {
> > "type" : "file",
> > "connection" : "s3a://bucketname/",
> > "config" : {
> > "fs.s3a.impl.disable.cache" : "true"
> > },
> > "workspaces" : {
> > "tmp" : {
> > "location" : "/tmp",
> > "writable" : true,
> > "defaultInputFormat" : null,
> > "allowAccessOutsideWorkspace" : false
> > },
> > "root" : {
> > "location" : "/",
> > "writable" : false,
> > "defaultInputFormat" : null,
> > "allowAccessOutsideWorkspace" : false
> > }
> > },
> > "formats" : {
> > "psv" : {
> > "type" : "text",
> > "extensions" : [ "tbl" ],
> > "delimiter" : "|"
> > },
> > "csv" : {
> > "type" : "text",
> > "extensions" : [ "csv" ],
> > "delimiter" : ","
> > },
> > "tsv" : {
> > "type" : "text",
> > "extensions" : [ "tsv" ],
> > "delimiter" : "\t"
> > },
> > "parquet" : {
> > "type" : "parquet"
> > },
> > "json" : {
> > "type" : "json",
> > "extensions" : [ "json" ]
> > },
> > "avro" : {
> > "type" : "avro"
> > },
> > "sequencefile" : {
> > "type" : "sequencefile",
> > "extensions" : [ "seq" ]
> > },
> > "csvh" : {
> > "type" : "text",
> > "extensions" : [ "csvh" ],
> > "extractHeader" : true,
> > "delimiter" : ","
> > }
> > },
> > "enabled" : true
> > }
> >
> > Any suggestions on how to get this working ?
> >
>
