WHat happens when you just run a query that reference s3 data instead of trying to do the "use s3" command?
On Tue, Apr 13, 2021 at 6:15 AM khushalkj jain <[email protected]> wrote: > Hi All, > Need your help in fixing the below Issue. > I am running drill locally on my MAC in embedded mode. > > *Query:* > > > use s3; > > *Log with Error info :* > > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > > AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; > Error > > Code: 400 Bad Request; Request ID: G9TTDZNV531H5RS9; S3 Extended Request > > ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=) > > Please, refer to logs for more information. > > [Error Id: ab06e603-feea-40cc-933f-85904f731ed8 on 192.168.1.7:31010] > > (org.apache.drill.exec.work.foreman.ForemanException) Unexpected > > exception during fragment initialization: Failed to create > DrillFileSystem > > for proxy user: doesBucketExist on c360-archival: > > com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: > > Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: > > G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=), > > S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=:400 > > Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error > Code: > > 400 Bad Request; Request ID: G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=) > > org.apache.drill.exec.work.foreman.Foreman.run():301 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1130 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():630 > > java.lang.Thread.run():832 > > Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) > > Failed to create DrillFileSystem for proxy user: doesBucketExist on > > c360-archival: com.amazonaws.services.s3.model.AmazonS3Exception: Bad > > Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad > Request; > > Request ID: G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=), > > S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=:400 > > Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error > Code: > > 400 Bad Request; Request ID: G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=) > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():220 > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():205 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():84 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():72 > > > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():232 > > org.apache.calcite.jdbc.DynamicRootSchema.loadSchemaFactory():87 > > org.apache.calcite.jdbc.DynamicRootSchema.getImplicitSubSchema():72 > > org.apache.calcite.jdbc.CalciteSchema.getSubSchema():265 > > > org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getSubSchema():684 > > > org.apache.drill.exec.planner.sql.SchemaUtilites.searchSchemaTree():98 > > org.apache.drill.exec.planner.sql.SchemaUtilites.findSchema():51 > > org.apache.drill.exec.rpc.user.UserSession.setDefaultSchemaPath():225 > > > > org.apache.drill.exec.planner.sql.handlers.UseSchemaHandler.getPlan():43 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():283 > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():163 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():128 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():93 > > org.apache.drill.exec.work.foreman.Foreman.runSQL():593 > > org.apache.drill.exec.work.foreman.Foreman.run():274 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1130 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():630 > > java.lang.Thread.run():832 > > Caused By (org.apache.hadoop.fs.s3a.AWSBadRequestException) > > doesBucketExist on c360-archival: > > com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: > > Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: > > G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=), > > S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=:400 > > Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error > Code: > > 400 Bad Request; Request ID: G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=) > > org.apache.hadoop.fs.s3a.S3AUtils.translateException():224 > > org.apache.hadoop.fs.s3a.Invoker.once():111 > > org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3():265 > > org.apache.hadoop.fs.s3a.Invoker.retryUntranslated():322 > > org.apache.hadoop.fs.s3a.Invoker.retry():261 > > org.apache.hadoop.fs.s3a.Invoker.retry():236 > > org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists():380 > > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize():314 > > org.apache.hadoop.fs.FileSystem.createFileSystem():3303 > > org.apache.hadoop.fs.FileSystem.get():476 > > org.apache.hadoop.fs.FileSystem.get():227 > > org.apache.drill.exec.store.dfs.DrillFileSystem.<init>():94 > > > > > org.apache.drill.exec.util.ImpersonationUtil.lambda$createFileSystem$0():215 > > java.security.AccessController.doPrivileged():691 > > javax.security.auth.Subject.doAs():425 > > org.apache.hadoop.security.UserGroupInformation.doAs():1730 > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():213 > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():205 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():84 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():72 > > > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():232 > > org.apache.calcite.jdbc.DynamicRootSchema.loadSchemaFactory():87 > > org.apache.calcite.jdbc.DynamicRootSchema.getImplicitSubSchema():72 > > org.apache.calcite.jdbc.CalciteSchema.getSubSchema():265 > > > org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getSubSchema():684 > > > org.apache.drill.exec.planner.sql.SchemaUtilites.searchSchemaTree():98 > > org.apache.drill.exec.planner.sql.SchemaUtilites.findSchema():51 > > org.apache.drill.exec.rpc.user.UserSession.setDefaultSchemaPath():225 > > > > org.apache.drill.exec.planner.sql.handlers.UseSchemaHandler.getPlan():43 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():283 > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():163 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():128 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():93 > > org.apache.drill.exec.work.foreman.Foreman.runSQL():593 > > org.apache.drill.exec.work.foreman.Foreman.run():274 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1130 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():630 > > java.lang.Thread.run():832 > > Caused By (com.amazonaws.services.s3.model.AmazonS3Exception) Bad > > Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad > Request; > > Request ID: G9TTDZNV531H5RS9; S3 Extended Request ID: > > > ihj+EsqMcF3qlP2EYHBwuarC5mOqiQ/PvVfgmu722WY8pL5VgRU69gbl4U1B3vpNqYYjcbiejGs=) > > > > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse():1640 > > > > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest():1304 > > > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper():1058 > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute():743 > > > > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer():717 > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute():699 > > com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500():667 > > > > > com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute():649 > > com.amazonaws.http.AmazonHttpClient.execute():513 > > com.amazonaws.services.s3.AmazonS3Client.invoke():4368 > > com.amazonaws.services.s3.AmazonS3Client.invoke():4315 > > com.amazonaws.services.s3.AmazonS3Client.headBucket():1344 > > com.amazonaws.services.s3.AmazonS3Client.doesBucketExist():1284 > > > > org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1():381 > > org.apache.hadoop.fs.s3a.Invoker.once():109 > > org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3():265 > > org.apache.hadoop.fs.s3a.Invoker.retryUntranslated():322 > > org.apache.hadoop.fs.s3a.Invoker.retry():261 > > org.apache.hadoop.fs.s3a.Invoker.retry():236 > > org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists():380 > > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize():314 > > org.apache.hadoop.fs.FileSystem.createFileSystem():3303 > > org.apache.hadoop.fs.FileSystem.get():476 > > org.apache.hadoop.fs.FileSystem.get():227 > > org.apache.drill.exec.store.dfs.DrillFileSystem.<init>():94 > > > > > org.apache.drill.exec.util.ImpersonationUtil.lambda$createFileSystem$0():215 > > java.security.AccessController.doPrivileged():691 > > javax.security.auth.Subject.doAs():425 > > org.apache.hadoop.security.UserGroupInformation.doAs():1730 > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():213 > > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():205 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():84 > > > > > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():72 > > > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():232 > > org.apache.calcite.jdbc.DynamicRootSchema.loadSchemaFactory():87 > > org.apache.calcite.jdbc.DynamicRootSchema.getImplicitSubSchema():72 > > org.apache.calcite.jdbc.CalciteSchema.getSubSchema():265 > > > org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getSubSchema():684 > > > org.apache.drill.exec.planner.sql.SchemaUtilites.searchSchemaTree():98 > > org.apache.drill.exec.planner.sql.SchemaUtilites.findSchema():51 > > org.apache.drill.exec.rpc.user.UserSession.setDefaultSchemaPath():225 > > > > org.apache.drill.exec.planner.sql.handlers.UseSchemaHandler.getPlan():43 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():283 > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():163 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():128 > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():93 > > org.apache.drill.exec.work.foreman.Foreman.runSQL():593 > > org.apache.drill.exec.work.foreman.Foreman.run():274 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1130 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():630 > > java.lang.Thread.run():832 > > > > *I have configured my storage plugin for s3 via UI as follows :* > > { > > "type": "file", > > "connection": "s3a://c360-archival/", > > "config": { > > "fs.s3a.secret.key": "*******", > > "fs.s3a.access.key": "*************", > > "fs.s3a.endpoint": "s3.us-west-2.amazonaws.com", > > "fs.s3a.impl.disable.cache": "true" > > }, > > "workspaces": { > > "tmp": { > > "location": "/tmp", > > "writable": true, > > "defaultInputFormat": null, > > "allowAccessOutsideWorkspace": true > > }, > > "root": { > > "location": "/", > > "writable": true, > > "defaultInputFormat": null, > > "allowAccessOutsideWorkspace": true > > } > > }, > > "formats": { > > "parquet": { > > "type": "parquet" > > }, > > "avro": { > > "type": "avro", > > "extensions": [ > > "avro" > > ] > > }, > > "json": { > > "type": "json", > > "extensions": [ > > "json" > > ] > > }, > > "pcap": { > > "type": "pcap", > > "extensions": [ > > "pcap" > > ] > > }, > > "csvh": { > > "type": "text", > > "extensions": [ > > "csvh" > > ], > > "extractHeader": true > > }, > > "sequencefile": { > > "type": "sequencefile", > > "extensions": [ > > "seq" > > ] > > }, > > "pcapng": { > > "type": "pcapng", > > "extensions": [ > > "pcapng" > > ] > > }, > > "psv": { > > "type": "text", > > "extensions": [ > > "tbl" > > ], > > "fieldDelimiter": "|" > > }, > > "tsv": { > > "type": "text", > > "extensions": [ > > "tsv" > > ], > > "fieldDelimiter": "\t" > > }, > > "csv": { > > "type": "text", > > "extensions": [ > > "csv" > > ] > > }, > > "spss": { > > "type": "spss", > > "extensions": [ > > "sav" > > ] > > }, > > "excel": { > > "type": "excel", > > "extensions": [ > > "xlsx" > > ], > > "lastRow": 1048576 > > }, > > "shp": { > > "type": "shp", > > "extensions": [ > > "shp" > > ] > > }, > > "hdf5": { > > "type": "hdf5", > > "extensions": [ > > "h5" > > ], > > "defaultPath": null > > }, > > "syslog": { > > "type": "syslog", > > "extensions": [ > > "syslog" > > ], > > "maxErrors": 10 > > }, > > "ltsv": { > > "type": "ltsv", > > "extensions": [ > > "ltsv" > > ] > > } > > }, > > "enabled": true > > } > > *core-site.xml* > > > <property> > > <name>fs.s3a.aws.credentials.provider</name> > > <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value> > > </property> >
