Hey Hao Sun, - Is this an intermittent failure or permanent? The logs indicate that some checkpoints completed before the error occurs (e.g. checkpoint numbers are greater than 1).
- Which Java versions are you using? And which Java image? I've Googled similar issues that seem to be related to the JVM, e.g. [1]. Best, Ufuk [1] https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972 On Thu, Nov 8, 2018 at 8:55 PM Hao Sun <[email protected]> wrote: > > Thanks, any insight/help here is appreciated. > > On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz <[email protected]> > wrote: >> >> Hi Hao, >> >> I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who were >> recently working with S3, maybe they will have some ideas. >> >> Best, >> >> Dawid >> >> On 03/11/2018 03:09, Hao Sun wrote: >> >> Same environment, new error. >> >> I can run the same docker image with my local Mac, but on K8S, this gives me >> this error. >> I can not think of any difference between local Docker and K8S Docker. >> >> Any hint will be helpful. Thanks >> >> ============ >> >> 2018-11-02 23:29:32,981 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> ConnectedStreams maxwell.accounts (00000000000000000000000000000000) >> switched from state RUNNING to FAILING. >> AsynchronousException{java.lang.Exception: Could not materialize checkpoint >> 235 for operator Source: KafkaSource(maxwell.accounts) -> >> MaxwellFilter->Maxwell(maxwell.accounts) -> >> FixedDelayWatermark(maxwell.accounts) -> >> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> influxdbSink(maxwell.accounts) (1/1).} >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> Caused by: java.lang.Exception: Could not materialize checkpoint 235 for >> operator Source: KafkaSource(maxwell.accounts) -> >> MaxwellFilter->Maxwell(maxwell.accounts) -> >> FixedDelayWatermark(maxwell.accounts) -> >> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> influxdbSink(maxwell.accounts) (1/1). >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) >> ... 6 more >> Caused by: java.util.concurrent.ExecutionException: >> java.lang.NoClassDefFoundError: Could not initialize class >> sun.security.ssl.SSLSessionImpl >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) >> >> at >> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(http://OperatorSnapshotFinalizer.java:53) >> >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) >> ... 5 more >> Caused by: java.lang.NoClassDefFoundError: Could not initialize class >> sun.security.ssl.SSLSessionImpl >> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) >> >> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572) >> >> at >> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown >> Source) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) >> at >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) >> at >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) >> at >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) >> at >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) >> at >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) >> at >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) >> at >> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) >> at >> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) >> at >> org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) >> at >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) >> at >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) >> at >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) >> at java.io.DataOutputStream.write(DataOutputStream.java:107) >> at >> org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) >> at >> org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) >> at >> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) >> at >> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) >> at >> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) >> at >> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) >> at >> org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) >> ... 7 more >> 2018-11-02 23:29:32,983 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to >> restart or fail the job ConnectedStreams maxwell.accounts >> (00000000000000000000000000000000) if no longer possible. >> >> ========================= >> >> On Thu, Nov 1, 2018 at 9:22 PM Hao Sun <[email protected]> wrote: >>> >>> I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and S3 >>> (presto) >>> I got this error when flink creating a checking point >>> >>> >>> =========== >>> 2018-11-02 04:00:55,011 INFO >>> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >>> ConnectedStreams maxwell.accounts (00000000000000000000000000000000) >>> switched from state RUNNING to FAILING. >>> AsynchronousException{java.lang.Exception: Could not materialize checkpoint >>> 130 for operator Source: KafkaSource(maxwell.accounts) -> >>> MaxwellFilter->Maxwell(maxwell.accounts) -> >>> FixedDelayWatermark(maxwell.accounts) -> >>> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >>> influxdbSink(maxwell.accounts) (1/1).} >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) >>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: java.lang.Exception: Could not materialize checkpoint 130 for >>> operator Source: KafkaSource(maxwell.accounts) -> >>> MaxwellFilter->Maxwell(maxwell.accounts) -> >>> FixedDelayWatermark(maxwell.accounts) -> >>> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >>> influxdbSink(maxwell.accounts) (1/1). >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) >>> ... 6 more >>> Caused by: java.util.concurrent.ExecutionException: >>> java.lang.ExceptionInInitializerError >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >>> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) >>> >>> at >>> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(http://OperatorSnapshotFinalizer.java:53) >>> >>> >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) >>> ... 5 more >>> Caused by: java.lang.ExceptionInInitializerError >>> >>> at sun.security.ssl.SSLSessionImpl.<init>(http://SSLSessionImpl.java:183) >>> at sun.security.ssl.SSLSessionImpl.<init>(http://SSLSessionImpl.java:148) >>> at sun.security.ssl.SSLSessionImpl.<clinit>(http://SSLSessionImpl.java:79) >>> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) >>> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572) >>> >>> >>> at >>> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown >>> Source) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) >>> at >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) >>> at >>> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) >>> at >>> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) >>> at >>> org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) >>> at >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) >>> at >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) >>> at >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) >>> at java.io.DataOutputStream.write(DataOutputStream.java:107) >>> at >>> org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) >>> at >>> org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) >>> at >>> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) >>> at >>> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) >>> at >>> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) >>> at >>> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) >>> at >>> org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) >>> ... 7 more >>> Caused by: java.security.ProviderException: Could not initialize NSS >>> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:223) >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:103) >>> >>> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224) >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206) >>> at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187) >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) >>> at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) >>> at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) >>> at java.security.Security.getProvider(Security.java:501) >>> >>> at >>> sun.security.ssl.SignatureAndHashAlgorithm.<clinit>(http://SignatureAndHashAlgorithm.java:415) >>> >>> >>> ... 72 more >>> Caused by: java.io.IOException: NSS is already initialized >>> at sun.security.pkcs11.Secmod.initialize(Secmod.java:173) >>> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:218) >>> ... 88 more >>> >>> ===========
