[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074878#comment-15074878
]
Rajat Khandelwal commented on HIVE-5672:
I see that the commit for this
jira(https://github.com/apache/hive/commit/f3a23dce083decc565447b7438527d7aec434b29#diff-f21e965d5c189c7d150fa1e8686eb294)
removes TOK_LOCAL_DIR altogether. Can someone elaborate why that is?
> Insert with custom separator not supported for non-local directory
> --
>
> Key: HIVE-5672
> URL: https://issues.apache.org/jira/browse/HIVE-5672
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0, 1.0.0
>Reporter: Romain Rigaux
>Assignee: Nemon Lou
> Fix For: 1.2.0
>
> Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
> HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
> HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
> HIVE-5672.7.patch.tar.gz, HIVE-5672.8.patch, HIVE-5672.8.patch.tar.gz
>
>
> https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
> directory don't seem to be supported:
> {code}
> insert overwrite directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select description FROM sample_07
> {code}
> {code}
> Error while compiling statement: FAILED: ParseException line 2:0 cannot
> recognize input near 'row' 'format' 'delimited' in select clause
> {code}
> This works (with 'local'):
> {code}
> insert overwrite local directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select code, description FROM sample_07
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521140#comment-14521140
]
Hive QA commented on HIVE-5672:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12729190/HIVE-5672.8.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8827 tests
executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3658/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3658/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3658/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12729190 - PreCommit-HIVE-TRUNK-Build
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
HIVE-5672.7.patch.tar.gz, HIVE-5672.8.patch, HIVE-5672.8.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522099#comment-14522099
]
Sushanth Sowmyan commented on HIVE-5672:
I see that the .8.patch has both the changes, and has passed all the tests. I'm
+1 on .8.patch.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
HIVE-5672.7.patch.tar.gz, HIVE-5672.8.patch, HIVE-5672.8.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522584#comment-14522584
]
Lefty Leverenz commented on HIVE-5672:
--
Doc note: Please review the updated note about custom separators in the DML
doc. (If it's okay, remove the TODOC1.2 label.)
* [DML -- Writing data into the filesystem from queries -- Notes |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Notes.2]
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Labels: TODOC1.2
Fix For: 1.2.0
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
HIVE-5672.7.patch.tar.gz, HIVE-5672.8.patch, HIVE-5672.8.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516983#comment-14516983
]
Nemon Lou commented on HIVE-5672:
-
The query plan for local directory is simple:
{quote}
explain insert overwrite local directory '/tmp/xxx' select * from src;
+---+
|Explain
|
+---+
| STAGE DEPENDENCIES:
|
| Stage-1 is a root stage
|
| Stage-0 depends on stages: Stage-1
|
|
|
| STAGE PLANS:
|
| Stage: Stage-1
|
| Map Reduce
|
| Map Operator Tree:
|
| TableScan
|
| alias: src
|
| Statistics: Num rows: 38 Data size: 11900 Basic stats: COMPLETE
Column stats: NONE|
| Select Operator
|
| expressions: id (type: string), starttime (type: bigint),
callerno (type: string), note (type: string) |
| outputColumnNames: _col0, _col1, _col2, _col3
|
| Statistics: Num rows: 38 Data size: 11900 Basic stats: COMPLETE
Column stats: NONE |
| File Output Operator
|
| compressed: false
|
| Statistics: Num rows: 38 Data size: 11900 Basic stats:
COMPLETE Column stats: NONE|
| table:
|
| input format: org.apache.hadoop.mapred.TextInputFormat
|
| output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
|
| serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
|
|
|
| Stage: Stage-0
|
| Move Operator
|
| files:
|
| hdfs directory: false
|
| destination: /tmp/xxx
|
|
|
+---+
29 rows selected (5.957 seconds)
{quote}
While the query plan for DFS directory is complicated：
{quote}
explain insert overwrite directory '/tmp/xxx' select * from src;
+-+
| Explain
|
+-+
| STAGE DEPENDENCIES:
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518637#comment-14518637
]
Nemon Lou commented on HIVE-5672:
-
Thanks to Xuefu and Sushanth for you review.
There is one more thing that i need to update :
removing occurrences of TOK_LOCAL_DIR.As you mentioned above.
Will update soon.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
HIVE-5672.7.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517851#comment-14517851
]
Xuefu Zhang commented on HIVE-5672:
---
+1 pending on test.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch,
HIVE-5672.7.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511006#comment-14511006
]
Nemon Lou commented on HIVE-5672:
-
This patch only adds the grammar support and does some code reflact.And it
works as expected.Hive can write to hdfs directory after applying this patch.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511084#comment-14511084
]
Xuefu Zhang commented on HIVE-5672:
---
Here is what I have about the combined grammar:
{code}
(local = KW_LOCAL)? KW_DIRECTORY StringLiteral tableRowFormat?
tableFileFormat?
- ^(TOK_DIR StringLiteral $local? tableRowFormat? tableFileFormat?)
{code}
With this, I'm sure SemanticAnalyzer has the information about whether the
directory is local.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz,
HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509385#comment-14509385
]
Hive QA commented on HIVE-5672:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12727622/HIVE-5672.5.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8729 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3545/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3545/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3545/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12727622 - PreCommit-HIVE-TRUNK-Build
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509927#comment-14509927
]
Sushanth Sowmyan commented on HIVE-5672:
Agree with Xuefu - the grammar might be simplified by simply making KW_LOCAL
optional, since there is no other place in hive code that seems to be making
use of TOK_LOCAL_DIR.
To wit, we could have :
{code}
KW_LOCAL? KW_DIRECTORY StringLiteral tableRowFormat? tableFileFormat? -
^(TOK_DIR StringLiteral tableRowFormat? tableFileFormat?)
{code}
This does mean that if at some point, we do still want to have a
differentiation between local and non-local writes, we have to go back to
Nemon's approach, and his approach is definitely the least damage done
approach of not trying to remove something that already exists, so his patch
makes sense from that point-of-view.
We have two approaches here, and I'm +1 on both:
a) Nemon's approach
b) Xuefu's suggestion : to make KW_LOCAL optional, and then emiting a TOK_DIR
instead of a TOK_LOCAL_DIR for that line, and removing any other occurrences of
TOK_LOCAL_DIR. We might eventually add it back because we want it, but it's
duplicate code pruning in the meanwhile.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509440#comment-14509440
]
Xuefu Zhang commented on HIVE-5672:
---
Looking at the patch, I'm not sure if I understand the changes correctly. I can
see that we modified the grammar to make local optional and the rest is about
refactoring. I'm not sure if this is sufficient. Did I miss anything?
Also, instead of adding a new grammar rule, we should combine it with the old
one. We just need to make KW_LOCAL optional.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch,
HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499023#comment-14499023
]
Lefty Leverenz commented on HIVE-5672:
--
How's this?
{quote}
As of Hive 0.11.0 the separator used can be specified, in earlier versions it
was always the ^A character (\001). However, custom separators are currently
only supported for LOCAL writes; this is a bug that will be fixed by HIVE-5672.
{quote}
* [DML -- Writing data into the filesystem from queries -- Notes |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Notes.2]
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496403#comment-14496403
]
Sushanth Sowmyan commented on HIVE-5672:
Adding additional doc note here :
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintothefilesystemfromqueries
needs to be updated to note that delimiters are not currently supported for
non-LOCAL writes, and once this patch goes in, we should note which version
fixed that in that doc.
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 1.0.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select description FROM sample_07
{code}
{code}
Error while compiling statement: FAILED: ParseException line 2:0 cannot
recognize input near 'row' 'format' 'delimited' in select clause
{code}
This works (with 'local'):
{code}
insert overwrite local directory '/tmp/test-02'
row format delimited
FIELDS TERMINATED BY ':'
select code, description FROM sample_07
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
[
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359232#comment-14359232
]
Hive QA commented on HIVE-5672:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704117/HIVE-5672.1.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3019/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3019/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3019/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig
itests/target itests/thirdparty itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-jmh/target itests/hive-unit/target
itests/custom-serde/target itests/util/target itests/qtest-spark/target
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target contrib/target service/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientpositive/alter_table_invalidate_column_stats.q.out
ql/src/test/queries/clientpositive/alter_table_invalidate_column_stats.q
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1666279.
At revision 1666279.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
patch: malformed patch at line 337:
patch: malformed patch at line 337:
patch: malformed patch at line 337:
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704117 - PreCommit-HIVE-TRUNK-Build
Insert with custom separator not supported for non-local directory
--
Key: HIVE-5672
URL: https://issues.apache.org/jira/browse/HIVE-5672
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Nemon Lou
Attachments: HIVE-5672.1.patch
https://issues.apache.org/jira/browse/HIVE-3682 is great but non local
directory don't seem to be supported:
{code}
insert overwrite
