[
https://issues.apache.org/jira/browse/PIG-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481026#comment-13481026
]
Rohini Palaniswamy commented on PIG-2973:
-----------------------------------------
Daniel,
bq. Rohini, did you apply the patch and test it works?
Tests pass. But wondering if we are missing a bigger problem because on
debugging found that UDFContext.getJobConf() is returning null in backend and
not frontend.
Cheolsoo,
bq. In fact, I am surprised that the exec() of ExecutableManager is called on
the front-end. But I am able to confirm that it's on the front-end by printing
out the output of UDFContext#isFrontend().
I put a breakpoint there and it never got executed in the frontend. It is
actually happening in the backend and the NPE is actually in syslog (under
build/test/logs/userlogs) of the job. The stack trace after adding the logging
statement is
{code}
java.lang.Throwable
at
org.apache.pig.impl.streaming.ExecutableManager.addJobConfToEnvironment(ExecutableManager.java:242)
at
org.apache.pig.impl.streaming.ExecutableManager.setupEnvironment(ExecutableManager.java:216)
at
org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:310)
at
org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:124)
at
org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:135)
at
org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:129)
at
org.apache.pig.impl.streaming.ExecutableManager$ProcessInputThread.run(ExecutableManager.java:407)
{code}
I am not that familiar with streaming. But from a cursory look, I am guessing
that the second command in the pipeline (which is launched from a separate
thread at the close of first one) might not be getting the env set correctly as
the conf is null (UDFContext is threadlocal and UDFContext.addJobConf() not
called for this thread). I think we need to investigate this more and
understand the effect of conf being null.
> TestStreaming test times out
> ----------------------------
>
> Key: PIG-2973
> URL: https://issues.apache.org/jira/browse/PIG-2973
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Rohini Palaniswamy
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2973.patch, test.tar.gz
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
