Siarhei Krukau created DRILL-6730:
-------------------------------------
Summary: Possible connection leak while accessing S3
Key: DRILL-6730
URL: https://issues.apache.org/jira/browse/DRILL-6730
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Other
Affects Versions: 1.14.0
Environment: h4. Cluster:
3 × {{m5.large}} AWS EC2 instances with Ubuntu 16.04.
Java:
{{openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)}}
Drill:
{{#Generated by Git-Commit-Id-Plugin
#Tue Jul 31 17:18:08 PDT 2018
git.commit.id.abbrev=0508a12
git.commit.user.email=bben-...@mapr.com
git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
git.commit.message.short=[maven-release-plugin] prepare release drill-1.14.0
git.commit.user.name=Ben-Zvi
git.build.user.name=Ben-Zvi
git.commit.id.describe=drill-1.14.0
git.build.user.email=bben-...@mapr.com
git.branch=0508a128853ce796ca7e99e13008e49442f83147
git.commit.time=31.07.2018 @ 16\:50\:38 PDT
git.build.time=31.07.2018 @ 17\:18\:08 PDT
git.remote.origin.url=https\://github.com/apache/drill.git}}
h4. Development:
Java:
{{openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode))}}
Drill:
{{#Generated by Git-Commit-Id-Plugin
#Tue Jul 31 17:18:08 PDT 2018
git.commit.id.abbrev=0508a12
git.commit.user.email=bben-...@mapr.com
git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
git.commit.message.short=[maven-release-plugin] prepare release drill-1.14.0
git.commit.user.name=Ben-Zvi
git.build.user.name=Ben-Zvi
git.commit.id.describe=drill-1.14.0
git.build.user.email=bben-...@mapr.com
git.branch=0508a128853ce796ca7e99e13008e49442f83147
git.commit.time=31.07.2018 @ 16\:50\:38 PDT
git.build.time=31.07.2018 @ 17\:18\:08 PDT
git.remote.origin.url=https\://github.com/apache/drill.git}}
Reporter: Siarhei Krukau
I have an AWS S3 bucket in {{us-east-1}} with hundreds of thousands of Parquet
files named like {{${UUID}.parquet}}. Originally they hold ~50K records and are
~25MiB in size each, but the issue is reproducible even with a single 96KiB
file with two records. The records are thin wrappers around OpenRTB's
[BidRequest|https://docs.openx.com/Content/demandpartners/openrtb_bidrequest.html].
I have configured a Drill cluster (no Hadoop) with three {{m5.large}} (CPU × 2,
RAM × 8GB) instances in the same region, all defaults. But the issue is
reproducible with {{drill-embedded}} as well.
My S3 data source config (both {{drill-embedded}} and cluster):
{{{
"type": "file",
"connection": "s3a://bukket/",
"config": {
"fs.s3a.access.key": "XXXXXXXXXXXXXX9XX9XX",
"fs.s3a.secret.key": "Xx/xxXxxxX9xxxXxxXXxxXxXXx9xxXXXXxXXxxx9",
"fs.s3a.endpoint": "s3.us-east-1.amazonaws.com",
"fs.s3a.connection.maximum": "100",
"fs.s3a.connection.timeout": "10000"
},
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"parquet": {
"type": "parquet"
}
},
"enabled": true
}}}
I omitted non-parquet-related configs for CSV, JSON, and others.
When enabling debug logs and executing a query like {{SELECT * FROM
s3.`slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet` LIMIT 10}} ({{slice}}
is a "directory" name inside S3 bucket) first I see the data from Parquet file
flowing through the wire, but then Drill tries to get byte ranges with HTTP
{{Range}} header:
{{DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: GET
/slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet HTTP/1.1
DEBUG org.apache.http.wire - >> "GET
/slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet HTTP/1.1[\r][\n]"
DEBUG org.apache.http.wire - >> "Host:
bukket.s3.us-east-1.amazonaws.com[\r][\n]"
DEBUG org.apache.http.wire - >> "Authorization: AWS
XXXXXXXXXXXXXX9XX9XX:xxXxXxXxXx9xXXXXxxXXxX9xxXx=[\r][\n]"
DEBUG org.apache.http.wire - >> "User-Agent: aws-sdk-java/1.7.4
Linux/4.17.19-1-MANJARO OpenJDK_64-Bit_Server_VM/25.181-b13/1.8.0_181[\r][\n]"
DEBUG org.apache.http.wire - >> "Range: bytes=2102-98296[\r][\n]"
DEBUG org.apache.http.wire - >> "Date: Tue, 04 Sep 2018 23:30:10 GMT[\r][\n]"
DEBUG org.apache.http.wire - >> "Content-Type:
application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
DEBUG org.apache.http.wire - >> "Connection: Keep-Alive[\r][\n]"
DEBUG org.apache.http.wire - >> "[\r][\n]"
DEBUG org.apache.http.headers - >> GET
/slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet HTTP/1.1
DEBUG org.apache.http.headers - >> Host: bukket.s3.us-east-1.amazonaws.com
DEBUG org.apache.http.headers - >> Authorization: AWS
XXXXXXXXXXXXXX9XX9XX:xxXxXxXxXx9xXXXXxxXXxX9xxXx=
DEBUG org.apache.http.headers - >> User-Agent: aws-sdk-java/1.7.4
Linux/4.17.19-1-MANJARO OpenJDK_64-Bit_Server_VM/25.181-b13/1.8.0_181
DEBUG org.apache.http.headers - >> Range: bytes=2102-98296
DEBUG org.apache.http.headers - >> Date: Tue, 04 Sep 2018 23:30:10 GMT
DEBUG org.apache.http.headers - >> Content-Type:
application/x-www-form-urlencoded; charset=utf-8
DEBUG org.apache.http.headers - >> Connection: Keep-Alive
DEBUG org.apache.http.wire - << "HTTP/1.1 206 Partial Content[\r][\n]"
DEBUG org.apache.http.wire - << "x-amz-id-2:
Xxx9xXXx99XxXXxXX9XxxxXxxXxx9X9X99XXXxXXxXxXxxXXxXxxXxXXXXxXxXXXXxxXX9X9Xx9=[\r][\n]"
DEBUG org.apache.http.wire - << "x-amz-request-id: 99X99XX999X9999X[\r][\n]"
DEBUG org.apache.http.wire - << "Date: Tue, 04 Sep 2018 23:30:11 GMT[\r][\n]"
DEBUG org.apache.http.wire - << "Last-Modified: Tue, 04 Sep 2018 21:44:29
GMT[\r][\n]"
DEBUG org.apache.http.wire - << "ETag:
"9x9xxxx999xxx999xx9999xxxx99xx99"[\r][\n]"
DEBUG org.apache.http.wire - << "Accept-Ranges: bytes[\r][\n]"
DEBUG org.apache.http.wire - << "Content-Range: bytes 2102-98296/98297[\r][\n]"
DEBUG org.apache.http.wire - << "Content-Type: application/octet-stream[\r][\n]"
DEBUG org.apache.http.wire - << "Content-Length: 96195[\r][\n]"
DEBUG org.apache.http.wire - << "Server: AmazonS3[\r][\n]"
DEBUG org.apache.http.wire - << "[\r][\n]"
DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 206
Partial Content
DEBUG org.apache.http.headers - << HTTP/1.1 206 Partial Content
DEBUG org.apache.http.headers - << x-amz-id-2:
Xxx9xXXx99XxXXxXX9XxxxXxxXxx9X9X99XXXxXXxXxXxxXXxXxxXxXXXXxXxXXXXxxXX9X9Xx9=
DEBUG org.apache.http.headers - << x-amz-request-id: 99X99XX999X9999X
DEBUG org.apache.http.headers - << Date: Tue, 04 Sep 2018 23:30:11 GMT
DEBUG org.apache.http.headers - << Last-Modified: Tue, 04 Sep 2018 21:44:29 GMT
DEBUG org.apache.http.headers - << ETag: "9x9xxxx999xxx999xx9999xxxx99xx99"
DEBUG org.apache.http.headers - << Accept-Ranges: bytes
DEBUG org.apache.http.headers - << Content-Range: bytes 2102-98296/98297
DEBUG org.apache.http.headers - << Content-Type: application/octet-stream
DEBUG org.apache.http.headers - << Content-Length: 96195
DEBUG org.apache.http.headers - << Server: AmazonS3
DEBUG c.a.http.impl.client.SdkHttpClient - Connection can be kept alive
indefinitely
DEBUG com.amazonaws.request - Received successful response: 206, AWS Request
ID: 99X99XX999X9999X
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem - Opening
'/slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet' for reading.
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem - Getting path status for
/slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet
(slice/9x99x99x-99x9-99x9-x99x-999x999xxxxx.parquet)}}
Each consequent request's range start is only a few bytes bigger, then previous
request (e.g. {{Range}} header from consequent requests is {{2315-98296}},
{{2351-98296}}, {{2387-98296}} and so on).
At some point of time, when only about 4K bytes of data is "read" (I guess that
by {{Range}} header start), Drill is all out of the connections:
{{DEBUG com.amazonaws.http.AmazonHttpClient - Retriable error detected, will
retry in 20000ms, attempt number: 8
DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection request: [route:
{s}->https://bukket.s3.us-east-1.amazonaws.com][total kept alive: 0; route
allocated: 100 of 100; total allocated: 100 of 100]
DEBUG org.eclipse.jetty.server.session - Scavenging sessions at 1536103994766
DEBUG c.a.h.c.ClientConnectionRequestFactory -
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
~[aws-java-sdk-1.7.4.jar:na]
at com.amazonaws.http.conn.$Proxy77.getConnection(Unknown Source)
[na:na]
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[httpclient-4.2.5.jar:4.2.5]
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
[aws-java-sdk-1.7.4.jar:na]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:892)
[hadoop-aws-2.7.1.jar:na]
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:373)
[hadoop-aws-2.7.1.jar:na]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
[hadoop-common-2.7.1.jar:na]
at
org.apache.drill.exec.store.dfs.DrillFileSystem.open(DrillFileSystem.java:134)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.parquet.hadoop.ColumnChunkIncReadStore.addColumn(ColumnChunkIncReadStore.java:246)
[drill-java-exec-1.14.0.jar:1.10.0]
at
org.apache.drill.exec.store.parquet2.DrillParquetReader.setup(DrillParquetReader.java:249)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas(ScanBatch.java:251)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:169)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281)
[drill-java-exec-1.14.0.jar:1.14.0]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[hadoop-common-2.7.1.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0.jar:1.14.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting
for connection from pool
at
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
~[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
~[httpclient-4.2.5.jar:4.2.5]
... 48 common frames omitted
INFO com.amazonaws.http.AmazonHttpClient - Unable to execute HTTP request:
Timeout waiting for connection from pool
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for
connection from pool
at
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
~[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
~[httpclient-4.2.5.jar:4.2.5]
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
~[aws-java-sdk-1.7.4.jar:na]
at com.amazonaws.http.conn.$Proxy77.getConnection(Unknown Source)
~[na:na]
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
~[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
~[httpclient-4.2.5.jar:4.2.5]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
~[httpclient-4.2.5.jar:4.2.5]
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
[aws-java-sdk-1.7.4.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
[aws-java-sdk-1.7.4.jar:na]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:892)
[hadoop-aws-2.7.1.jar:na]
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:373)
[hadoop-aws-2.7.1.jar:na]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
[hadoop-common-2.7.1.jar:na]
at
org.apache.drill.exec.store.dfs.DrillFileSystem.open(DrillFileSystem.java:134)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.parquet.hadoop.ColumnChunkIncReadStore.addColumn(ColumnChunkIncReadStore.java:246)
[drill-java-exec-1.14.0.jar:1.10.0]
at
org.apache.drill.exec.store.parquet2.DrillParquetReader.setup(DrillParquetReader.java:249)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas(ScanBatch.java:251)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:169)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281)
[drill-java-exec-1.14.0.jar:1.14.0]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[hadoop-common-2.7.1.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281)
[drill-java-exec-1.14.0.jar:1.14.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0.jar:1.14.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]}}
Drill makes 10 exponential retry attempts and finally fails.
I've tried different values for {{fs.s3a.connection.maximum}} (3, 10, 100) and
{{fs.s3a.connection.timeout}} (10000, 30000 60000) with the same luck (no luck).
Is that a connection leak? Can it be fixed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
