[ https://issues.apache.org/jira/browse/HADOOP-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641888#comment-16641888 ]
Tank Sui commented on HADOOP-15824: ----------------------------------- it should caused by sbt dist, looks the right FileSystem implements exclude from sbt dist. because the same error not happened in develop mode while it throws at production mode which run sbt dist. > RawLocalFileSystem initialized with Null Point Exception > -------------------------------------------------------- > > Key: HADOOP-15824 > URL: https://issues.apache.org/jira/browse/HADOOP-15824 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 2.8.3, 2.8.4, 3.1.1 > Environment: Hadoop 2.8.4 + Spark & yarn client launch > Reporter: Tank Sui > Priority: Minor > > {code:java} > [ERROR]09:33:13.143 [main] org.apache.spark.SparkContext - Error initializing > SparkContext. > 10/6/2018 5:33:13 PM java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > 10/6/2018 5:33:13 PM at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) > 10/6/2018 5:33:13 PM at > org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:351) > 10/6/2018 5:33:13 PM at > org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:649) > 10/6/2018 5:33:13 PM at > org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:863) > 10/6/2018 5:33:13 PM at > org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169) > 10/6/2018 5:33:13 PM at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57) > 10/6/2018 5:33:13 PM at > org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) > 10/6/2018 5:33:13 PM at > org.apache.spark.SparkContext.<init>(SparkContext.scala:500) > 10/6/2018 5:33:13 PM at > org.apache.spark.SparkContext.<init>(SparkContext.scala:126) > 10/6/2018 5:33:13 PM at services.SparkService.tryInit(SparkService.scala:49) > 10/6/2018 5:33:13 PM at > controllers.DataController.<init>(DataController.scala:38) > 10/6/2018 5:33:13 PM at > controllers.DataController$$FastClassByGuice$$9ed55d7d.newInstance(<generated>) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) > 10/6/2018 5:33:13 PM at > play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:409) > 10/6/2018 5:33:13 PM at > play.api.inject.ContextClassLoaderInjector$$anonfun$instanceOf$3.apply(Injector.scala:118) > 10/6/2018 5:33:13 PM at > play.api.inject.ContextClassLoaderInjector.withContext(Injector.scala:126) > 10/6/2018 5:33:13 PM at > play.api.inject.ContextClassLoaderInjector.instanceOf(Injector.scala:118) > 10/6/2018 5:33:13 PM at > play.api.inject.RoutesProvider$$anonfun$2.apply(BuiltinModule.scala:107) > 10/6/2018 5:33:13 PM at > play.api.inject.RoutesProvider$$anonfun$2.apply(BuiltinModule.scala:107) > 10/6/2018 5:33:13 PM at scala.Option.fold(Option.scala:158) > 10/6/2018 5:33:13 PM at > play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:107) > 10/6/2018 5:33:13 PM at > play.api.inject.RoutesProvider.get(BuiltinModule.scala:103) > 10/6/2018 5:33:13 PM at > play.api.inject.RoutesProvider.get(BuiltinModule.scala:102) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) > 10/6/2018 5:33:13 PM at > com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) > 10/6/2018 5:33:13 PM at com.google.inject.Guice.createInjector(Guice.java:99) > 10/6/2018 5:33:13 PM at com.google.inject.Guice.createInjector(Guice.java:84) > 10/6/2018 5:33:13 PM at > play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:185) > 10/6/2018 5:33:13 PM at > play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:137) > 10/6/2018 5:33:13 PM at > play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21) > 10/6/2018 5:33:13 PM at > play.core.server.ProdServerStart$.start(ProdServerStart.scala:51) > 10/6/2018 5:33:13 PM at > play.core.server.ProdServerStart$.main(ProdServerStart.scala:25) > 10/6/2018 5:33:13 PM at > play.core.server.ProdServerStart.main(ProdServerStart.scala) > 10/6/2018 5:33:13 PM Caused by: java.lang.reflect.InvocationTargetException: > null > 10/6/2018 5:33:13 PM at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > 10/6/2018 5:33:13 PM at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > 10/6/2018 5:33:13 PM at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > 10/6/2018 5:33:13 PM at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132) > 10/6/2018 5:33:13 PM ... 80 common frames omitted > 10/6/2018 5:33:13 PM Caused by: java.lang.NullPointerException: null > 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.Path.<init>(Path.java:141) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.Path.makeQualified(Path.java:535) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:485) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.RawLocalFileSystem.getInitialWorkingDirectory(RawLocalFileSystem.java:581) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.RawLocalFileSystem.<init>(RawLocalFileSystem.java:73) > 10/6/2018 5:33:13 PM at > org.apache.hadoop.fs.LocalFileSystem.<init>(LocalFileSystem.java:41) > 10/6/2018 5:33:13 PM ... 85 common frames omitted > {code} > This error happened in hadoop 2.8.4 > after investigation, i think below code in Path.java of apache common project > has some issue. > {code:java} > /** > * Create a new Path based on the child path resolved against the parent path. > * > * @param parent the parent path > * @param child the child path > */ > public Path(Path parent, Path child) { > // Add a slash to parent's path so resolution is compatible with URI's > URI parentUri = parent.uri; > String parentPath = parentUri.getPath(); > if (!(parentPath.equals("/") || parentPath.isEmpty())) { > try { > {code} > *parent* should be null while *parent.uri* called. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org