[
https://issues.apache.org/jira/browse/PIG-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park reopened PIG-4061:
--------------------------------
[~daijy], streaming udf is broken by your patch. I think you shouldn't add
these lines-
{code}
+ if ("StreamingUDF".equals(clazz.getSimpleName())) {
+ for (String fileName : StreamingUDF.getResourcesForJar()) {
+ jarLists.add(new File(fileName).toURI());
+ }
+ }
{code}
Here is the stack trace-
{code}
Caused by: java.io.FileNotFoundException: File /python/streaming/controller.py
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:516)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:729)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:506)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:402)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1951)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1919)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1884)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResource(TezResourceManager.java:82)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:105)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:108)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:89)
... 21 more
{code}
> Make Streaming UDF work in Tez
> ------------------------------
>
> Key: PIG-4061
> URL: https://issues.apache.org/jira/browse/PIG-4061
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Reporter: Cheolsoo Park
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4061-1.patch, PIG-4061-2.patch
>
>
> PIG-3478 fixed Streaming UDF in MR2. We should get it working in Tez.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
