Nico Kruber created FLINK-11459:
-----------------------------------
Summary: Presto S3 does not show errors due to missing credentials
with minio
Key: FLINK-11459
URL: https://issues.apache.org/jira/browse/FLINK-11459
Project: Flink
Issue Type: Bug
Components: filesystem-connector
Affects Versions: 1.6.2
Reporter: Nico Kruber
It seems that when using minio for S3-like storage and with mis-configurations
such as missing (maybe also wrong) credentials gets into a failing state but
with no reason for it:
{code}
...
2019-01-29 15:43:27,676 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.heap.mb, 353
2019-01-29 15:43:27,738 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.heap.mb, 429
2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
[] - The job has 0 registered types and 0 default Kryo serializers
2019-01-29 15:43:29,943 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
2019-01-29 15:43:29,956 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2115.551],
2019-01-29 15:43:31,946 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
2019-01-29 15:43:31,958 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[3610.417],
2019-01-29 15:43:33,954 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
2019-01-29 15:43:33,963 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2917.786],
2019-01-29 15:43:36,133 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
2019-01-29 15:43:36,156 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2029.473],
2019-01-29 15:43:38,142 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
2019-01-29 15:43:38,164 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2092.878],
2019-01-29 15:43:42,181 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
2019-01-29 15:43:42,186 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2011.204],
2019-01-29 15:43:44,262 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
2019-01-29 15:43:44,276 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] -
ClientExecuteTime=[2024.312],
2019-01-29 15:43:44,585 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED
SIGNAL 15: SIGTERM. Shutting down as requested.
2019-01-29 15:43:44,628 INFO org.apache.flink.runtime.blob.TransientBlobCache
[] - Shutting down BLOB cache
2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
[] - Stopped BLOB server at 0.0.0.0:6124
{code}
With AWS S3, it is actually printing an exception instead:
{code}
2019-01-29 19:24:39,968 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.port, 8081
2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
- The job has 0 registered types and 0 default Kryo serializers
2019-01-29 19:24:43,117 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
2019-01-29 19:24:43,118 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2049.826],
2019-01-29 19:24:46,215 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
2019-01-29 19:24:46,216 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2004.182],
2019-01-29 19:24:50,384 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.15], CredentialsRequestTime=[2002.803],
2019-01-29 19:24:50,384 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2004.308],
2019-01-29 19:24:56,691 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2002.596], CredentialsRequestTime=[2002.45],
2019-01-29 19:24:56,691 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.177],
2019-01-29 19:25:07,058 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.26], CredentialsRequestTime=[2002.948],
2019-01-29 19:25:07,058 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2004.175],
2019-01-29 19:25:25,472 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2001.772], CredentialsRequestTime=[2001.611],
2019-01-29 19:25:25,473 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2002.873],
2019-01-29 19:25:25,475 ERROR
org.apache.flink.api.common.io.DelimitedInputFormat - Unexpected
problem while getting the file statistics for files '[s3://flink/LICENSE.gz]':
Unable to load credentials from service endpoint
org.apache.flink.fs.s3presto.shaded.com.amazonaws.SdkClientException: Unable to
load credentials from service endpoint
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:180)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:159)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307)
at
org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:85)
at
org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:526)
at
org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:505)
at
org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:356)
at
org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:47)
at
org.apache.flink.optimizer.dag.DataSourceNode.computeOperatorSpecificDefaultEstimates(DataSourceNode.java:166)
at
org.apache.flink.optimizer.dag.OptimizerNode.computeOutputEstimates(OptimizerNode.java:589)
at
org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:61)
at
org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:32)
at
org.apache.flink.optimizer.dag.DataSourceNode.accept(DataSourceNode.java:250)
at
org.apache.flink.optimizer.dag.SingleInputNode.accept(SingleInputNode.java:515)
at
org.apache.flink.optimizer.dag.DataSinkNode.accept(DataSinkNode.java:248)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:478)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:399)
at
org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:51)
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
at test.FlinkReadS3Test.main(FlinkReadS3Test.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$10(JarRunHandler.java:226)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:47)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:106)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121)
... 52 more
...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
