[
https://issues.apache.org/jira/browse/PIG-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dmitry d updated PIG-3450:
--------------------------
Description:
I try to run embedded python from java program using PigRunner. Script file
store in hdfs. And I get the following error:
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. null
java.lang.NullPointerException
at
org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
at
org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
at org.apache.pig.PigServer.registerJar(PigServer.java:458)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
at org.apache.pig.Main.run(Main.java:427)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
================================================================================
No error when I run python script from local path.
My Java code:
PigStats stats = PigRunner.run(new String[] {"-f",
"hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
My Python code:
#!/usr/bin/python
import sys
print('hello from python');
I saw source code where error is happened - FileLocalizer.java:
FileSystem localFs = FileSystem.getLocal(conf);
FileSystem srcFs;
if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
&& uri.getScheme() == null )||
uri.getScheme().equals("local") ) {
srcFs = localFs;
} else {
srcFs = path.getFileSystem(conf);
}
uri is local path to jython-standalone.jar. And uri.getScheme() return null.
And uri.getScheme().equals("local") is throw exception
Maybe I must redefine some property to get jython-standalone.jar from hdfs?
was:
I try to run embedded python from java program using PigRunner. Script file
store in hdfs. And I get the following error:
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. null
java.lang.NullPointerException
at
org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
at
org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
at org.apache.pig.PigServer.registerJar(PigServer.java:458)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
at org.apache.pig.Main.run(Main.java:427)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
================================================================================
No error when I run python script from local path.
My Java code:
PigStats stats = PigRunner.run(new String[] {"-f",
"hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
My Python code:
#!/usr/bin/python
import sys
print('hello from python');
I saw source code where error is happened - FileLocalizer.java:
FileSystem localFs = FileSystem.getLocal(conf);
FileSystem srcFs;
if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
&& uri.getScheme() == null )||
uri.getScheme().equals("local") ) {
srcFs = localFs;
} else {
srcFs = path.getFileSystem(conf);
}
uri is local path to jython-standalone.jar. And uri.getScheme() return null.
And uri.getScheme().equals("local") is throw exception
Maybe I must redefine some property to get jython-standalone.jar from hdfs?
> error when pigrunner.run embedded python stored in hdfs
> -------------------------------------------------------
>
> Key: PIG-3450
> URL: https://issues.apache.org/jira/browse/PIG-3450
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: dmitry d
>
> I try to run embedded python from java program using PigRunner. Script file
> store in hdfs. And I get the following error:
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. null
> java.lang.NullPointerException
> at
> org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
> at
> org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
> at org.apache.pig.PigServer.registerJar(PigServer.java:458)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
> at org.apache.pig.Main.run(Main.java:427)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
> at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> ================================================================================
> No error when I run python script from local path.
> My Java code:
> PigStats stats = PigRunner.run(new String[] {"-f",
> "hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
> My Python code:
> #!/usr/bin/python
> import sys
> print('hello from python');
> I saw source code where error is happened - FileLocalizer.java:
> FileSystem localFs = FileSystem.getLocal(conf);
> FileSystem srcFs;
> if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
> && uri.getScheme() == null )||
> uri.getScheme().equals("local") ) {
> srcFs = localFs;
> } else {
> srcFs = path.getFileSystem(conf);
> }
> uri is local path to jython-standalone.jar. And uri.getScheme() return null.
> And uri.getScheme().equals("local") is throw exception
> Maybe I must redefine some property to get jython-standalone.jar from hdfs?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
