[
https://issues.apache.org/jira/browse/PIG-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904017#comment-15904017
]
liyunzhang_intel commented on PIG-5177:
---------------------------------------
[~szita]:
quick look at this patch, several questions about it:
1. this case only fails in yarn-client mode, not in local mode?
2. why in mr mode, this does not fail?
ScriptEngine#getScriptAsStream
{code}
public static InputStream getScriptAsStream(String scriptPath) {
//protected static InputStream getScriptAsStream(String scriptPath) {
InputStream is = null;
File file = new File(scriptPath);
// In the frontend give preference to the local file.
// In the backend, try the jar first
if (UDFContext.getUDFContext().isFrontend() && file.exists()) {
try {
is = new FileInputStream(file);
} catch (FileNotFoundException e) {
throw new IllegalStateException("could not find existing file
"+scriptPath, e);
}
} else {
if (Shell.WINDOWS && scriptPath.charAt(1)==':') {
scriptPath = scriptPath.charAt(0) + scriptPath.substring(2);
}
// Try system, current and context classloader.
is = ScriptEngine.class.getResourceAsStream(scriptPath);
if (is == null) {
is = getResourceUsingClassLoader(scriptPath,
ScriptEngine.class.getClassLoader());
}
if (is == null) {
is = getResourceUsingClassLoader(scriptPath,
Thread.currentThread().getContextClassLoader());
}
if (is == null && !file.isAbsolute()) {
String path = "/" + scriptPath;
is = ScriptEngine.class.getResourceAsStream(path);
if (is == null) {
is = getResourceUsingClassLoader(path,
ScriptEngine.class.getClassLoader());
}
if (is == null) {
is = getResourceUsingClassLoader(path,
Thread.currentThread().getContextClassLoader());
}
}
}
if (is == null && file.exists()) {
try {
is = new FileInputStream(file);
} catch (FileNotFoundException e) {
throw new IllegalStateException("could not find existing file
"+scriptPath, e);
}
}
// TODO: discuss if we want to add logic here to load a script from HDFS
if (is == null) {
throw new IllegalStateException(
"Could not initialize interpreter (from file system or
classpath) with " + scriptPath);
}
return is;
}
{code}
> Scripting and StreamingPythonUDFs fail with Spark exec type
> -----------------------------------------------------------
>
> Key: PIG-5177
> URL: https://issues.apache.org/jira/browse/PIG-5177
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Adam Szita
> Assignee: Adam Szita
> Fix For: spark-branch
>
> Attachments: PIG-5177.0.patch, PIG-5177.1.patch, PIG-5177.2.patch
>
>
> We are thrown an exception because the Python script file is not found on the
> backend side (on spark executors).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
