jackjlli opened a new pull request #6070: URL: https://github.com/apache/incubator-pinot/pull/6070
## Description This PR adds Hadoop related dependencies (only **runtime** scope) in pinot-tool module. These Hadoop jars are needed in some Pinot commands at runtime, such as `CreateSegment` for ORC files. In pinot-orc pom file, those Hadoop dependencies are in provided scope, which means the transitive dependencies won't get pulled in at runtime. https://www.baeldung.com/maven-dependency-scopes#:~:text=Transitive%20dependencies%2C%20on%20the%20other,%3A%20mvn%20dependency%3Atree%20command Current exception: ``` java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172] at org.apache.pinot.tools.admin.command.CreateSegmentCommand.execute(CreateSegmentCommand.java:264) ~[classes/:?] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:152) [classes/:?] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:164) [classes/:?] Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:1.8.0_172] at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) ~[?:1.8.0_172] at java.lang.Class.getConstructor0(Class.java:3075) ~[?:1.8.0_172] at java.lang.Class.getConstructor(Class.java:1825) ~[?:1.8.0_172] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:270) ~[classes/:?] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:239) ~[classes/:?] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:220) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReaderByClass(RecordReaderFactory.java:132) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReader(RecordReaderFactory.java:156) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReader(RecordReaderFactory.java:143) ~[classes/:?] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.getRecordReader(SegmentIndexCreationDriverImpl.java:129) ~[classes/:?] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.init(SegmentIndexCreationDriverImpl.java:96) ~[classes/:?] at org.apache.pinot.tools.admin.command.CreateSegmentCommand.lambda$execute$0(CreateSegmentCommand.java:238) ~[classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_172] Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_172] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_172] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_172] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_172] at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:1.8.0_172] at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) ~[?:1.8.0_172] at java.lang.Class.getConstructor0(Class.java:3075) ~[?:1.8.0_172] at java.lang.Class.getConstructor(Class.java:1825) ~[?:1.8.0_172] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:270) ~[classes/:?] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:239) ~[classes/:?] at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:220) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReaderByClass(RecordReaderFactory.java:132) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReader(RecordReaderFactory.java:156) ~[classes/:?] at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReader(RecordReaderFactory.java:143) ~[classes/:?] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.getRecordReader(SegmentIndexCreationDriverImpl.java:129) ~[classes/:?] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.init(SegmentIndexCreationDriverImpl.java:96) ~[classes/:?] at org.apache.pinot.tools.admin.command.CreateSegmentCommand.lambda$execute$0(CreateSegmentCommand.java:238) ~[classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_172] ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
