[
https://issues.apache.org/jira/browse/FLINK-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038494#comment-17038494
]
Arvid Heise edited comment on FLINK-16014 at 2/18/20 8:03 AM:
--------------------------------------------------------------
Root cause is a [bug in JDK8|https://bugs.openjdk.java.net/browse/JDK-8015099].
`XMLReaderFactory`
[caches|https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/jdk8u222-b10/jaxp/src/org/xml/sax/helpers/XMLReaderFactory.java#L146-L174]
the class name independent of the classloader. On EMR, xercesImpl is on
classpath (because of HDFS) and will be loaded at some point in time.
We have some workarounds, none of which would solve it in all cases. Since the
user may bundle any `SAXParser` in his code or put it in lib, the plugin might
always save a `XMLReader` that is not visible from the plugin.
Workarounds:
1) Add xercesImpl to s3 plugin. Would blow up file size but cover the most
common case of a `SAXParser`.
2) Add another smaller `SAXParser` to avoid having none.
For all options, we should eagerly initialize the `XMLReaderFactory` in
S3AFileSystem to win against user code and hopefully even `lib`.
was (Author: arvid heise):
Root cause is a bug in JDK8.
[`XMLReaderFactory`|https://bugs.openjdk.java.net/browse/JDK-8015099]
[caches|https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/jdk8u222-b10/jaxp/src/org/xml/sax/helpers/XMLReaderFactory.java#L146-L174]
the class name independent of the classloader. On EMR, xercesImpl is on
classpath (because of HDFS) and will be loaded at some point in time.
We have some workarounds, none of which would solve it in all cases. Since the
user may bundle any `SAXParser` in his code or put it in lib, the plugin might
always save a `XMLReader` that is not visible from the plugin.
Workarounds:
1) Add xercesImpl to s3 plugin. Would blow up file size but cover the most
common case of a `SAXParser`.
2) Add another smaller `SAXParser` to avoid having none.
For all options, we should eagerly initialize the `XMLReaderFactory` in
S3AFileSystem to win against user code and hopefully even `lib`.
> S3 plugin ClassNotFoundException SAXParser
> ------------------------------------------
>
> Key: FLINK-16014
> URL: https://issues.apache.org/jira/browse/FLINK-16014
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.10.0, 1.11.0
> Reporter: Arvid Heise
> Priority: Major
>
> While stress-testing s3 plugin on EMR.
>
> {noformat}
> org.apache.flink.util.FlinkRuntimeException: Could not perform checkpoint 2
> for operator Map (114/160).
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:839)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:104)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierUnaligner.notifyBarrierReceived(CheckpointBarrierUnaligner.java:149)
> at
> org.apache.flink.streaming.runtime.io.InputProcessorUtil$1.lambda$notifyBarrierReceived$0(InputProcessorUtil.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:87)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:255)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:508)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:492)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3://emr-unaligned-checkpoints-testing-eu-central-1/inflight/9ae223e41008b17568d7f63c12360268_output/part-file.-1:
> com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create
> an XMLReader: Couldn't initialize a SAX driver to create an XMLReader
> at
> org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.checkErroneousUnsafe(BufferPersisterImpl.java:262)
> at
> org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.add(BufferPersisterImpl.java:137)
> at
> org.apache.flink.runtime.io.network.BufferPersisterImpl.addBuffers(BufferPersisterImpl.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.prepareInflightDataSnapshot(StreamTask.java:935)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:898)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:870)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:826)
> ... 12 more
> Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3://emr-unaligned-checkpoints-testing-eu-central-1/inflight/9ae223e41008b17568d7f63c12360268_output/part-file.-1:
> com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create
> an XMLReader: Couldn't initialize a SAX driver to create an XMLReader
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2251)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2970)
> at
> org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.exists(HadoopFileSystem.java:152)
> at
> org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.exists(PluginFileSystemFactory.java:143)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.exists(SafetyNetWrapperFileSystem.java:102)
> at
> org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.get(BufferPersisterImpl.java:213)
> at
> org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.run(BufferPersisterImpl.java:167)
> Caused by: com.amazonaws.SdkClientException: Couldn't initialize a SAX driver
> to create an XMLReader
> at
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.<init>(XmlResponsesSaxParser.java:118)
> at
> com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:87)
> at
> com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:77)
> at
> com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
> at
> com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
> at
> com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4266)
> at
> com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:876)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$5(S3AFileSystem.java:1262)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1255)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2223)
> ... 9 more
> Caused by: org.xml.sax.SAXException: SAX2 driver class
> org.apache.xerces.parsers.SAXParser not found
> java.lang.ClassNotFoundException: org.apache.xerces.parsers.SAXParser
> at
> org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:230)
> at
> org.xml.sax.helpers.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:191)
> at
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.<init>(XmlResponsesSaxParser.java:115)
> ... 32 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.xerces.parsers.SAXParser
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> org.apache.flink.core.plugin.PluginLoader$PluginClassLoader.loadClass(PluginLoader.java:149)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.xml.sax.helpers.NewInstance.newInstance(NewInstance.java:82)
> at
> org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:228)
> ... 34 more{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
