Can you try playing around with these properties and see if that helps ?
<property> <name>fs.s3a.connection.establish.timeout</name> <value>5000</value> <description>Socket connection setup timeout in milliseconds.</description> </property> <property> <name>fs.s3a.connection.timeout</name> <value>50000</value> <description>Socket connection timeout in milliseconds.</description> </property> ________________________________ From: Bob Carter <[email protected]> Sent: Thursday, August 3, 2017 12:35 PM To: [email protected] Subject: Timeout error querying S3 Apache Drill 11 Error: SYSTEM ERROR: ConnectionPoolTimeoutException: Timeout waiting for connection from pool [Error Id: c5d6ba1f-2868-48b4-8db0-63e027f15bd2 on db-migrate:31010] (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Unable to execute HTTP request: Timeout waiting for connection from pool org.apache.drill.exec.work.foreman.Foreman.run():297 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():748 Caused By (com.amazonaws.AmazonClientException) Unable to execute HTTP request: Timeout waiting for connection from pool com.amazonaws.http.AmazonHttpClient.executeHelper():454 com.amazonaws.http.AmazonHttpClient.execute():232 com.amazonaws.services.s3.AmazonS3Client.invoke():3528 com.amazonaws.services.s3.AmazonS3Client.invoke():3480 com.amazonaws.services.s3.AmazonS3Client.listObjects():604 org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962 org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734 org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64 org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149 org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396 org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110 org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99 org.apache.drill.exec.ops.QueryContext.getRootSchema():164 org.apache.drill.exec.ops.QueryContext.getRootSchema():153 org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139 org.apache.drill.exec.planner.sql.SqlConverter.<init>():111 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79 org.apache.drill.exec.work.foreman.Foreman.runSQL():1050 org.apache.drill.exec.work.foreman.Foreman.run():280 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():748 Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection():232 org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection():199 sun.reflect.GeneratedMethodAccessor9.invoke():-1 sun.reflect.DelegatingMethodAccessorImpl.invoke():43 java.lang.reflect.Method.invoke():606 com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70 com.amazonaws.http.conn.$Proxy58.getConnection():-1 org.apache.http.impl.client.DefaultRequestDirector.execute():456 org.apache.http.impl.client.AbstractHttpClient.execute():906 org.apache.http.impl.client.AbstractHttpClient.execute():805 com.amazonaws.http.AmazonHttpClient.executeHelper():384 com.amazonaws.http.AmazonHttpClient.execute():232 com.amazonaws.services.s3.AmazonS3Client.invoke():3528 com.amazonaws.services.s3.AmazonS3Client.invoke():3480 com.amazonaws.services.s3.AmazonS3Client.listObjects():604 org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962 org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734 org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64 org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149 org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396 org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110 org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99 org.apache.drill.exec.ops.QueryContext.getRootSchema():164 org.apache.drill.exec.ops.QueryContext.getRootSchema():153 org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139 org.apache.drill.exec.planner.sql.SqlConverter.<init>():111 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79 org.apache.drill.exec.work.foreman.Foreman.runSQL():1050 org.apache.drill.exec.work.foreman.Foreman.run():280 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():748 (state=,code=0) 0: jdbc:drill:zk=local> I add the following to the core-site.xml but it does not help. <property> <name>fs.s3a.connection.maximum</name> <value>100</value> </property>
