[
https://issues.apache.org/jira/browse/HIVE-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joydeep Sen Sarma updated HIVE-1570:
------------------------------------
Attachment: 1570.1.patch
before running a map-reduce job in local mode we:
1. set a new working directory
2. symlink all added files from that working directory
this is pretty much identical to how hadoop sets up task execution environment.
all references to scripts and add files using their names only now resolve
correctly in local mode.
there was some hacky code in SemanticAnalyzer.java to deal with this that
doesn't work in all cases (when referenced file is not the first item in
command line or in automatic local mode). i have deleted it.
duplicated one of the tests so that we get coverage against a real cluster
(scriptfile1.q executed against minimr) and local mode (scriptfile2.q).
still running tests.
> referencing an added file by it's name in a transform script does not work in
> hive local mode
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-1570
> URL: https://issues.apache.org/jira/browse/HIVE-1570
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1570.1.patch
>
>
> Yongqiang tried this and it fails in local mode:
> add file ../data/scripts/dumpdata_script.py;
> select count(distinct subq.key) from
> (FROM src MAP src.key USING 'python dumpdata_script.py' AS key WHERE src.key
> = 10) subq;
> this needs to be fixed because it means we cannot choose local mode
> automatically in case of transform scripts (since different paths need to be
> used for cluster vs. local mode execution)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
