Robert Metzger created FLINK-2568: ------------------------------------- Summary: Flink fat jar is missing org.w3c.dom.ElementTraversal in some environments. Key: FLINK-2568 URL: https://issues.apache.org/jira/browse/FLINK-2568 Project: Flink Issue Type: Bug Components: Build System Affects Versions: 0.9 Reporter: Robert Metzger
Error while writing into HDFS reported by a user. The user is using a Hadoop Configuration file using the XML XInclude feature. Maybe that is causing the issue? {code} 5:04:57,701 WARN org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception java.lang.NoClassDefFoundError: org/w3c/dom/ElementTraversal at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) [3:09:41 PM] Vyacheslav Zholudev: java.lang.NoClassDefFoundError: org/w3c/dom/ElementTraversal at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.xerces.parsers.AbstractDOMParser.startDocument(Unknown Source) at org.apache.xerces.xinclude.XIncludeHandler.startDocument(Unknown Source) at org.apache.xerces.impl.dtd.XMLDTDValidator.startDocument(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl.startEntity(Unknown Source) at org.apache.xerces.impl.XMLVersionDetector.startDocumentParsing(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1989) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1977) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2048) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2001) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1918) at org.apache.hadoop.conf.Configuration.get(Configuration.java:721) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:740) at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:912) at org.apache.hadoop.hdfs.protocol.HdfsConstants.<clinit>(HdfsConstants.java:67) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1140) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) {code} I could not find the class in the Flink fat jar. It seems like we have to include the xml-apis dependency into Flink: http://stackoverflow.com/questions/17515333/noclassdeffounderror-org-w3c-dom-elementtraversal because the class is not available on all JVMs: https://issues.apache.org/jira/browse/HADOOP-11831 {code} <dependency> <groupId>xml-apis</groupId> <artifactId>xml-apis</artifactId> <version>1.4.01</version> </dependency> {code} I'm opening this issue to track if more users are affected. The user who reported the issue to me was able to resolve the issue by manually providing the xml-apis dependency. If one one user is affected due to environment specific settings, I would not include the xml-apis into our fat jar (for now) -- This message was sent by Atlassian JIRA (v6.3.4#6332)