[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484791#comment-14484791
]
Swarnim Kulkarni commented on HIVE-10226:
-
Few very minor comments on RB. +1 otherwise.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10189: Attachment: HIVE-10189.1.patch Update patch addressing [~spena] and [~chengxiang li]'s comments Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10120) Disallow create table with dot/colon in column name
[
https://issues.apache.org/jira/browse/HIVE-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484821#comment-14484821
]
Hive QA commented on HIVE-10120:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723678/HIVE-10120.02.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8665 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3321/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3321/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3321/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723678 - PreCommit-HIVE-TRUNK-Build
Disallow create table with dot/colon in column name
---
Key: HIVE-10120
URL: https://issues.apache.org/jira/browse/HIVE-10120
Project: Hive
Issue Type: Improvement
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-10120.01.patch, HIVE-10120.02.patch
Since we don't allow users to query column names with dot in the middle such
as emp.no, don't allow users to create tables with such columns that cannot
be queried. Fix the documentation to reflect this fix.
Here is an example. Consider this table:
{code}
CREATE TABLE a (`emp.no` string);
select `emp.no` from a; fails with this message:
FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp
from [0:emp.no]
{code}
The hive documentation needs to be fixed:
{code}
(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) seems
to indicate that any Unicode character can go between the backticks in the
select statement, but it doesn’t like the dot/colon or even select * when
there is a column that has a dot/colon.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10258) LLAP: orc_llap test fails again
[
https://issues.apache.org/jira/browse/HIVE-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-10258:
-
Attachment: HIVE-10258.patch
LLAP: orc_llap test fails again
---
Key: HIVE-10258
URL: https://issues.apache.org/jira/browse/HIVE-10258
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Prasanth Jayachandran
Attachments: HIVE-10258.patch
{noformat}
Caused by: java.io.IOException: java.io.IOException: java.io.IOException:
Corruption in ORC data encountered. To skip reading corrupted data, set
hive.exec.orc.skip.corrupt.data to true{noformat}
llap_partitioned passes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10158) LLAP: Varchar columns are not handled in encoded readers
[
https://issues.apache.org/jira/browse/HIVE-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran resolved HIVE-10158.
--
Resolution: Fixed
This is already committed to llap branch.
LLAP: Varchar columns are not handled in encoded readers
Key: HIVE-10158
URL: https://issues.apache.org/jira/browse/HIVE-10158
Project: Hive
Issue Type: Sub-task
Affects Versions: llap
Reporter: Gopal V
Assignee: Prasanth Jayachandran
Fix For: llap
Attachments: HIVE-10158.1.patch
ORC Varchar columns can't be read off the cache
{code}
Caused by: java.io.IOException: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$VarcharTreeReader cannot be
cast to
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:249)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:201)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:96)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$VarcharTreeReader cannot be
cast to
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader
at
org.apache.hadoop.hive.llap.io.decode.orc.stream.readers.CharacterStreamReader.seek(CharacterStreamReader.java:84)
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.createTreeReaders(OrcEncodedDataConsumer.java:332)
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.decodeBatch(OrcEncodedDataConsumer.java:98)
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:108)
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:35)
at
org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:278)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:48)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10259) LLAP: Explain plan diff in llap_partitioned.q test case
[ https://issues.apache.org/jira/browse/HIVE-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-10259. -- Resolution: Fixed Committed to llap branch LLAP: Explain plan diff in llap_partitioned.q test case --- Key: HIVE-10259 URL: https://issues.apache.org/jira/browse/HIVE-10259 Project: Hive Issue Type: Sub-task Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Attachments: HIVE-10259.patch Explain plan for llap_partitioned.q shows a diff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10259) LLAP: Explain plan diff in llap_partitioned.q test case
[ https://issues.apache.org/jira/browse/HIVE-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-10259: - Attachment: HIVE-10259.patch LLAP: Explain plan diff in llap_partitioned.q test case --- Key: HIVE-10259 URL: https://issues.apache.org/jira/browse/HIVE-10259 Project: Hive Issue Type: Sub-task Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Attachments: HIVE-10259.patch Explain plan for llap_partitioned.q shows a diff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks
[
https://issues.apache.org/jira/browse/HIVE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485132#comment-14485132
]
Hive QA commented on HIVE-8164:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723748/HIVE-8164.3.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8664 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3325/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3325/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3325/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723748 - PreCommit-HIVE-TRUNK-Build
Adding in a ReplicationTask that converts a Notification Event to actionable
tasks
--
Key: HIVE-8164
URL: https://issues.apache.org/jira/browse/HIVE-8164
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-8164.2.patch, HIVE-8164.3.patch, HIVE-8164.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9609) AddPartitionMessage.getPartitions() can return null
[
https://issues.apache.org/jira/browse/HIVE-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485034#comment-14485034
]
Hive QA commented on HIVE-9609:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723732/HIVE-9609.3.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3324/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3324/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3324/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723732 - PreCommit-HIVE-TRUNK-Build
AddPartitionMessage.getPartitions() can return null
---
Key: HIVE-9609
URL: https://issues.apache.org/jira/browse/HIVE-9609
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-9609.2.patch, HIVE-9609.3.patch, HIVE-9609.patch
DbNotificationListener and NotificationListener both depend on
AddPartitionEvent.getPartitions() to get their partitions to trigger a
message, but this can be null if an AddPartitionEvent was initialized on a
PartitionSpec rather than a ListPartition.
Also, AddPartitionEvent seems to have a duality, where getPartitions() works
only if instantiated on a ListPartition, and getPartitionIterator() works
only if instantiated on a PartitionSpec.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10237) create external table， location path contains space ，like '/user/hive/warehouse/custom.db/uigs_kmap '
[ https://issues.apache.org/jira/browse/HIVE-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu resolved HIVE-10237. - Resolution: Cannot Reproduce This issue can't be reproduced in the latest trunk. create external table， location path contains space ，like '/user/hive/warehouse/custom.db/uigs_kmap ' --- Key: HIVE-10237 URL: https://issues.apache.org/jira/browse/HIVE-10237 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.13.1 Environment: Hadoop 2.3.0-cdh5.0.0 hive 0.13.1 Reporter: xiaowei wang Assignee: Ferdinand Xu when i want to create a external table and give the table a location ，i write a wront location path， /user/hive/warehouse/custom.db/uigs_kmap ，which contains a space at the end of the path。 I think hive will trim the space of the location，but it does not。 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10189: Attachment: HIVE-10189.2.patch As discussed offline with [~chengxiang li], update the patch. Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.2.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10189: Attachment: (was: HIVE-10189.2.patch) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10260) hive 1.1.0 retun NULL value
[ https://issues.apache.org/jira/browse/HIVE-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-10260: -- Description: hive 0.10 correct result: 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 9150119100048 73263562015-01-19 10:44:44 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 121501191035580028 73263562015-01-19 10:35:58 UBDTK8D9XUZ9GRZU8NZNXDEG73D4PCZG2362223711289 161501191549050061 14837289 2015-01-19 15:49:05 Y49EY895ACABHS95DRQEE8DVFEB8JSE12360853052224 111501191426280023 115883224 2015-01-19 14:26:28 hive 1.1.0 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 9150119100048 7326356 NULL 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 121501191035580028 7326356 NULL UBDTK8D9XUZ9GRZU8NZNXDEG73D4PCZG2362223711289 161501191549050061 14837289 NULL Y49EY895ACABHS95DRQEE8DVFEB8JSE12360853052224 111501191426280023 115883224 NULL I found difference from log: In hive 1.1.0 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[date_id, chanl_id, sessn_id, gu_id, prov_id, city_id, landing_page_type_id, landing_track_time, landing_url, nav_refer_tracker_id, nav_refer_page_type_id, nav_refer_page_value, nav_refer_link_position, nav_tracker_id, nav_page_categ_id, nav_page_type_id, nav_page_value, nav_srce_type, internal_keyword, internal_result_sum, pltfm_id, app_vers, nav_link_position, nav_button_position, nav_track_time, nav_next_tracker_id, sessn_last_time, sessn_pv, detl_tracker_id, detl_page_type_id, detl_page_value, detl_pm_id, detl_link_position, detl_position_track_id, cart_tracker_id, cart_page_type_id, cart_page_value, cart_link_postion, cart_button_position, cart_position_track_id, cart_prod_id, ordr_tracker_id, ordr_page_type_id, ordr_code, updt_time, cart_pm_id, brand_code, categ_type, os, end_user_id, add_cart_flag, navgation_page_flag, nav_page_url, detl_button_position, manul_flag, manul_track_date, nav_refer_tpa, nav_refer_tpa_id, nav_refer_tpc, nav_refer_tpi, nav_refer_tcs, nav_refer_tcsa, nav_refer_tcdt, nav_refer_tcd, nav_refer_tci, nav_refer_postn_type, nav_tpa_id, nav_tpa, nav_tpc, nav_tpi, nav_tcs, nav_tcsa, nav_tcdt, nav_tcd, nav_tci, nav_postn_type, detl_tpa_id, detl_tpa, detl_tpc, detl_tpi, detl_tcs, detl_tcsa, detl_tcdt, detl_tcd, detl_tci, detl_postn_type, cart_tpa_id, cart_tpa, cart_tpc, cart_tpi, cart_tcs, cart_tcsa, cart_tcdt, cart_tcd, cart_tci, cart_postn_type] columnTypes=[string, bigint, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, int, string, string, string, string, string, string, int, string, string, string, bigint, string, string, string, string, string, string, string, string, bigint, string, string, string, string, bigint, string, int, string, string, string, int, string, string, int, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string] separator=[[B@e50bca4] nullstring=\N lastColumnTakesRest=false In hive 0.10 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[date_id, chanl_id, sessn_id, gu_id, prov_id, city_id, landing_page_type_id, landing_track_time, landing_url, nav_refer_tracker_id, nav_refer_page_type_id, nav_refer_page_value, nav_refer_link_position, nav_tracker_id, nav_page_categ_id, nav_page_type_id, nav_page_value, nav_srce_type, internal_keyword, internal_result_sum, pltfm_id, app_vers, nav_link_position, nav_button_position, nav_track_time, nav_next_tracker_id, sessn_last_time, sessn_pv, detl_tracker_id, detl_page_type_id, detl_page_value, detl_pm_id, detl_link_position, detl_position_track_id, cart_tracker_id, cart_page_type_id, cart_page_value, cart_link_postion, cart_button_position, cart_position_track_id, cart_prod_id, ordr_tracker_id, ordr_page_type_id, ordr_code, updt_time, cart_pm_id, brand_code, categ_type, os, end_user_id, add_cart_flag, navgation_page_flag, nav_page_url, detl_button_position, manul_flag, manul_track_date, nav_refer_tpa, nav_refer_tpa_id, nav_refer_tpc, nav_refer_tpi, nav_refer_tcs, nav_refer_tcsa, nav_refer_tcdt, nav_refer_tcd, nav_refer_tci, nav_refer_postn_type, nav_tpa_id, nav_tpa, nav_tpc, nav_tpi, nav_tcs, nav_tcsa, nav_tcdt, nav_tcd, nav_tci, nav_postn_type, detl_tpa_id, detl_tpa, detl_tpc, detl_tpi, detl_tcs, detl_tcsa, detl_tcdt, detl_tcd, detl_tci, detl_postn_type, cart_tpa_id, cart_tpa, cart_tpc, cart_tpi, cart_tcs,
[jira] [Updated] (HIVE-10251) HIVE-9664 makes hive depend on ivysettings.xml
[
https://issues.apache.org/jira/browse/HIVE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anant Nag updated HIVE-10251:
-
Attachment: HIVE-10251.1.patch
HIVE-9664 makes hive depend on ivysettings.xml
--
Key: HIVE-10251
URL: https://issues.apache.org/jira/browse/HIVE-10251
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Anant Nag
Attachments: HIVE-10251.1.patch
HIVE-9664 makes hive depend on the existence of ivysettings.xml, and if it is
not present, it makes hive NPE when instantiating a CLISessionState.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.session.DependencyResolver.init(DependencyResolver.java:61)
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:343)
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:334)
at org.apache.hadoop.hive.cli.CliSessionState.init(CliSessionState.java:60)
{noformat}
This happens because of the following bit:
{noformat}
// If HIVE_HOME is not defined or file is not found in HIVE_HOME/conf
then load default ivysettings.xml from class loader
if (ivysettingsPath == null || !(new File(ivysettingsPath).exists())) {
ivysettingsPath =
ClassLoader.getSystemResource(ivysettings.xml).getFile();
_console.printInfo(ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR, + ivysettingsPath + will be used);
}
{noformat}
This makes it so that an attempt to instantiate CliSessionState without an
ivysettings.xml file will cause hive to fail with an NPE. Hive should not
have a hard dependency on a ivysettings,xml being present, and this feature
should gracefully fail in that case instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9937) LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join
[ https://issues.apache.org/jira/browse/HIVE-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484929#comment-14484929 ] Gopal V commented on HIVE-9937: --- [~mmccline]: LGTM - +1. Good test coverage - this is just a new fast SerDe + test-cases, without any deviation for the main codepath until the new operators are introduced. Reading a Decimal from Key instead of Value might be a corner case perhaps. Before commit, can you verify the behavour of BinarySortableSerde on Decimal (trailing zeros/precision) - something like vector_decimal_round.q should do as a validity test. LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join -- Key: HIVE-9937 URL: https://issues.apache.org/jira/browse/HIVE-9937 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-9937.01.patch, HIVE-9937.02.patch, HIVE-9937.03.patch, HIVE-9937.04.patch, HIVE-9937.05.patch, HIVE-9937.06.patch, HIVE-9937.07.patch, HIVE-9937.08.patch, HIVE-9937.09.patch, HIVE-9937.91.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10253) Parquet PPD support DATE
[ https://issues.apache.org/jira/browse/HIVE-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-10253: - Attachment: HIVE-10253.patch This patch is for review and has been verified locally with the code of HIVE-10252 and PARQUET-247. Parquet PPD support DATE Key: HIVE-10253 URL: https://issues.apache.org/jira/browse/HIVE-10253 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Attachments: HIVE-10253.patch Hive should handle the DATE data type when generating and pushing the predicate to Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10260) hive 1.1.0 return NULL value
[ https://issues.apache.org/jira/browse/HIVE-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-10260: -- Summary: hive 1.1.0 return NULL value (was: hive 1.1.0 retun NULL value) hive 1.1.0 return NULL value Key: HIVE-10260 URL: https://issues.apache.org/jira/browse/HIVE-10260 Project: Hive Issue Type: Bug Affects Versions: 1.1.0 Environment: hive 1.1.0 hadoop 2.3.0 Reporter: tangjunjie hive 0.10 correct result: 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 9150119100048 73263562015-01-19 10:44:44 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 121501191035580028 73263562015-01-19 10:35:58 UBDTK8D9XUZ9GRZU8NZNXDEG73D4PCZG2362223711289 161501191549050061 14837289 2015-01-19 15:49:05 Y49EY895ACABHS95DRQEE8DVFEB8JSE12360853052224 111501191426280023 115883224 2015-01-19 14:26:28 hive 1.1.0 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 9150119100048 7326356 NULL 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 121501191035580028 7326356 NULL UBDTK8D9XUZ9GRZU8NZNXDEG73D4PCZG2362223711289 161501191549050061 14837289 NULL Y49EY895ACABHS95DRQEE8DVFEB8JSE12360853052224 111501191426280023 115883224 NULL I found difference from log: In hive 1.1.0 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[date_id, chanl_id, sessn_id, gu_id, prov_id, city_id, landing_page_type_id, landing_track_time, landing_url, nav_refer_tracker_id, nav_refer_page_type_id, nav_refer_page_value, nav_refer_link_position, nav_tracker_id, nav_page_categ_id, nav_page_type_id, nav_page_value, nav_srce_type, internal_keyword, internal_result_sum, pltfm_id, app_vers, nav_link_position, nav_button_position, nav_track_time, nav_next_tracker_id, sessn_last_time, sessn_pv, detl_tracker_id, detl_page_type_id, detl_page_value, detl_pm_id, detl_link_position, detl_position_track_id, cart_tracker_id, cart_page_type_id, cart_page_value, cart_link_postion, cart_button_position, cart_position_track_id, cart_prod_id, ordr_tracker_id, ordr_page_type_id, ordr_code, updt_time, cart_pm_id, brand_code, categ_type, os, end_user_id, add_cart_flag, navgation_page_flag, nav_page_url, detl_button_position, manul_flag, manul_track_date, nav_refer_tpa, nav_refer_tpa_id, nav_refer_tpc, nav_refer_tpi, nav_refer_tcs, nav_refer_tcsa, nav_refer_tcdt, nav_refer_tcd, nav_refer_tci, nav_refer_postn_type, nav_tpa_id, nav_tpa, nav_tpc, nav_tpi, nav_tcs, nav_tcsa, nav_tcdt, nav_tcd, nav_tci, nav_postn_type, detl_tpa_id, detl_tpa, detl_tpc, detl_tpi, detl_tcs, detl_tcsa, detl_tcdt, detl_tcd, detl_tci, detl_postn_type, cart_tpa_id, cart_tpa, cart_tpc, cart_tpi, cart_tcs, cart_tcsa, cart_tcdt, cart_tcd, cart_tci, cart_postn_type] columnTypes=[string, bigint, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, int, string, string, string, string, string, string, int, string, string, string, bigint, string, string, string, string, string, string, string, string, bigint, string, string, string, string, bigint, string, int, string, string, string, int, string, string, int, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string] separator=[[B@e50bca4] nullstring=\N lastColumnTakesRest=false In hive 0.10 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[date_id, chanl_id, sessn_id, gu_id, prov_id, city_id, landing_page_type_id, landing_track_time, landing_url, nav_refer_tracker_id, nav_refer_page_type_id, nav_refer_page_value, nav_refer_link_position, nav_tracker_id, nav_page_categ_id, nav_page_type_id, nav_page_value, nav_srce_type, internal_keyword, internal_result_sum, pltfm_id, app_vers, nav_link_position, nav_button_position, nav_track_time, nav_next_tracker_id, sessn_last_time, sessn_pv, detl_tracker_id, detl_page_type_id, detl_page_value, detl_pm_id, detl_link_position, detl_position_track_id, cart_tracker_id, cart_page_type_id, cart_page_value, cart_link_postion, cart_button_position, cart_position_track_id, cart_prod_id, ordr_tracker_id, ordr_page_type_id, ordr_code, updt_time, cart_pm_id, brand_code, categ_type, os, end_user_id, add_cart_flag, navgation_page_flag, nav_page_url, detl_button_position,
[jira] [Updated] (HIVE-10191) ORC: Cleanup writer per-row synchronization
[ https://issues.apache.org/jira/browse/HIVE-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10191: --- Attachment: HIVE-10191.2.patch Fix Preconditions check and rebase ORC: Cleanup writer per-row synchronization --- Key: HIVE-10191 URL: https://issues.apache.org/jira/browse/HIVE-10191 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gopal V Attachments: HIVE-10191.1.patch, HIVE-10191.2.patch ORC writers were originally meant to be thread-safe, but in the present day implementation each ORC writer is entirely share-nothing which converts most of the synchronized blocks in ORC as entirely uncontested locks. These uncontested locks prevent the JVM from inlining/optimizing these methods, while adding no extra thread-safety to the ORC writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10119) Allow Log verbosity to be set in hiveserver2 session
[
https://issues.apache.org/jira/browse/HIVE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484972#comment-14484972
]
Hive QA commented on HIVE-10119:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723736/HIVE-10119.4.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8667 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3323/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3323/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3323/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723736 - PreCommit-HIVE-TRUNK-Build
Allow Log verbosity to be set in hiveserver2 session
Key: HIVE-10119
URL: https://issues.apache.org/jira/browse/HIVE-10119
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-10119.1.patch, HIVE-10119.2.patch,
HIVE-10119.3.patch, HIVE-10119.4.patch
We need to be able to set logging per HS2 session.
The client often uses the map-reduce completion matrix (Execution) that shows
up in Beeline to debug performance. User might not want the verbose log view
all the time since it obfuscates the Execution information. Hence the client
should be able to change the verbosity level.
Also, there are 2 levels of verbosity at HS2 logging and not 3. The users
might want Execution + Performance counters only - so that level needs to be
added.
So for logs, the user should be able to set 3 levels of verbosity in the
session, that will override the default verbosity specified in the
hive-site.xml file.
0. None - IGNORE
1. Execution - Just shows the map-reduce tasks completing
2. Performance - Execution + Performance counters dumped at the end
3. Verbose - All logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10174) LLAP: ORC MemoryManager is singleton synchronized
[ https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-10174: -- Assignee: Gopal V LLAP: ORC MemoryManager is singleton synchronized - Key: HIVE-10174 URL: https://issues.apache.org/jira/browse/HIVE-10174 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: llap Reporter: Gopal V Assignee: Gopal V Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded performance. !orc-memorymanager-1.png! !orc-memorymanager-2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10256) Filter row groups based on the block statistics in Parquet
[
https://issues.apache.org/jira/browse/HIVE-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-10256:
Summary: Filter row groups based on the block statistics in Parquet (was:
Eliminate row groups based on the block statistics in Parquet)
Filter row groups based on the block statistics in Parquet
--
Key: HIVE-10256
URL: https://issues.apache.org/jira/browse/HIVE-10256
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In Parquet PPD, the not matched row groups should be eliminated. See
{{TestOrcSplitElimination}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10189: Attachment: HIVE-10189.2.patch Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.2.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9580) Server returns incorrect result from JOIN ON VARCHAR columns
[
https://issues.apache.org/jira/browse/HIVE-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-9580:
---
Attachment: HIVE-9580.patch
For join on varchar or char columns, currently we don't implicitly convert the
keys since they are of the same type with different lengths. While during the
query run-time, we assume all the keys are of the same type and will also
choose the first one as the type to be used to parse all the keys, which are
causing the issue.
The patch applies the the common type for all the keys even all the keys are of
the same type. Also change a type comparison to use equals() rather than ==.
Server returns incorrect result from JOIN ON VARCHAR columns
Key: HIVE-9580
URL: https://issues.apache.org/jira/browse/HIVE-9580
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0, 0.14.0
Reporter: Mike
Assignee: Aihua Xu
Attachments: HIVE-9580.patch
The database erroneously returns rows when joining two tables which each
contain a VARCHAR column and the join's ON condition uses the equality
operator on the VARCHAR columns.
**The following JDBC method exhibits the problem:
static void joinIssue()
throws SQLException {
String sql;
int rowsAffected;
ResultSet rs;
Statement stmt = con.createStatement();
String table1_Name = blahtab1;
String table1A_Name = blahtab1A;
String table1B_Name = blahtab1B;
String table2_Name = blahtab2;
try {
sql = drop table + table1_Name;
System.out.println(\nsql= + sql);
rowsAffected = stmt.executeUpdate(sql);
}
catch (SQLException se) {
println(Drop table error: + se.getMessage());
}
try {
sql = CREATE TABLE + table1_Name + ( +
VCHARCOL VARCHAR(10) +
,INTEGERCOL INT +
)
;
System.out.println(\nsql= + sql);
rowsAffected = stmt.executeUpdate(sql);
}
catch (SQLException se) {
println(create table error: + se.getMessage());
}
sql = insert into + table1_Name + values ('jklmnopqrs',
99);
System.out.println(\nsql= + sql);
stmt.executeUpdate(sql);
System.out.println(===);
try {
sql = drop table + table1A_Name;
System.out.println(\nsql= + sql);
rowsAffected = stmt.executeUpdate(sql);
}
catch (SQLException se) {
println(Drop table error: + se.getMessage());
}
try {
sql = CREATE TABLE + table1A_Name + ( +
VCHARCOL VARCHAR(10) +
)
;
System.out.println(\nsql= + sql);
rowsAffected = stmt.executeUpdate(sql);
}
catch (SQLException se) {
println(create table error: + se.getMessage());
}
sql = insert into + table1A_Name + values ('jklmnopqrs');
System.out.println(\nsql= + sql);
stmt.executeUpdate(sql);
System.out.println(===);
try {
sql = drop table + table1B_Name;
System.out.println(\nsql= + sql);
rowsAffected = stmt.executeUpdate(sql);
}
catch (SQLException se) {
println(Drop table error: + se.getMessage());
}
try {
sql = CREATE TABLE + table1B_Name + ( +
VCHARCOL VARCHAR(11) +
,INTEGERCOL INT +
)
;
[jira] [Commented] (HIVE-9633) Add HCatClient.dropPartitions() overload to skip deletion of partition-directories.
[
https://issues.apache.org/jira/browse/HIVE-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485424#comment-14485424
]
Hive QA commented on HIVE-9633:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697657/HIVE-9633.1.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3328/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3328/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3328/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697657 - PreCommit-HIVE-TRUNK-Build
Add HCatClient.dropPartitions() overload to skip deletion of
partition-directories.
---
Key: HIVE-9633
URL: https://issues.apache.org/jira/browse/HIVE-9633
Project: Hive
Issue Type: Bug
Components: API, HCatalog, Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9633.1.patch
{{HCatClient.dropPartitions()}} doesn't provide a way to explicitly skip the
deletion of partition-directory, as {{HiveMetaStoreClient.dropPartitions()}}
does.
This'll come in handy when using HCatClient to drop partitions, but not
delete data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10261) Data size can be underestimated when computed with partial column stats
[
https://issues.apache.org/jira/browse/HIVE-10261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485439#comment-14485439
]
Mostafa Mokhtar commented on HIVE-10261:
[~lirui]
Can you please attach an explain plan along with query and actual number of
rows for the operator with underestimation?
Data size can be underestimated when computed with partial column stats
---
Key: HIVE-10261
URL: https://issues.apache.org/jira/browse/HIVE-10261
Project: Hive
Issue Type: Bug
Reporter: Rui Li
With {{hive.stats.fetch.column.stats=true}}, we'll estimate data size with
column stats when annotating operators with statistics. However, when column
stats is partial, we're likely to underestimate data size, which may hurt
performance, e.g. picking an inappropriate small table for map join.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9609) AddPartitionMessage.getPartitions() can return null
[ https://issues.apache.org/jira/browse/HIVE-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485558#comment-14485558 ] Sushanth Sowmyan commented on HIVE-9609: Tests failures noted here are unconnected with this fix. AddPartitionMessage.getPartitions() can return null --- Key: HIVE-9609 URL: https://issues.apache.org/jira/browse/HIVE-9609 Project: Hive Issue Type: Bug Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-9609.2.patch, HIVE-9609.3.patch, HIVE-9609.patch DbNotificationListener and NotificationListener both depend on AddPartitionEvent.getPartitions() to get their partitions to trigger a message, but this can be null if an AddPartitionEvent was initialized on a PartitionSpec rather than a ListPartition. Also, AddPartitionEvent seems to have a duality, where getPartitions() works only if instantiated on a ListPartition, and getPartitionIterator() works only if instantiated on a PartitionSpec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks
[ https://issues.apache.org/jira/browse/HIVE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485564#comment-14485564 ] Sushanth Sowmyan commented on HIVE-8164: Tests failures noted here are unconnected with this fix. With Alan's +1 of the .2.patch, and .3.patch being a trivial update on .2.patch to cover Alan's warning, I'm going to go ahead and commit it to trunk. Adding in a ReplicationTask that converts a Notification Event to actionable tasks -- Key: HIVE-8164 URL: https://issues.apache.org/jira/browse/HIVE-8164 Project: Hive Issue Type: Sub-task Components: Import/Export Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-8164.2.patch, HIVE-8164.3.patch, HIVE-8164.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9679) Remove redundant null-checks from DbNotificationListener.
[
https://issues.apache.org/jira/browse/HIVE-9679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485580#comment-14485580
]
Hive QA commented on HIVE-9679:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12698587/HIVE-9679.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3329/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3329/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3329/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12698587 - PreCommit-HIVE-TRUNK-Build
Remove redundant null-checks from DbNotificationListener.
-
Key: HIVE-9679
URL: https://issues.apache.org/jira/browse/HIVE-9679
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Priority: Minor
Attachments: HIVE-9679.patch
There's a couple of unnecessary null-checks in {{DbNotificationListener}}.
There's no way they'd fire. Shall we remove these?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10119) Allow Log verbosity to be set in hiveserver2 session
[ https://issues.apache.org/jira/browse/HIVE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485587#comment-14485587 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-10119: -- The test failures look unrelated to my change. Thanks Hari Allow Log verbosity to be set in hiveserver2 session Key: HIVE-10119 URL: https://issues.apache.org/jira/browse/HIVE-10119 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-10119.1.patch, HIVE-10119.2.patch, HIVE-10119.3.patch, HIVE-10119.4.patch We need to be able to set logging per HS2 session. The client often uses the map-reduce completion matrix (Execution) that shows up in Beeline to debug performance. User might not want the verbose log view all the time since it obfuscates the Execution information. Hence the client should be able to change the verbosity level. Also, there are 2 levels of verbosity at HS2 logging and not 3. The users might want Execution + Performance counters only - so that level needs to be added. So for logs, the user should be able to set 3 levels of verbosity in the session, that will override the default verbosity specified in the hive-site.xml file. 0. None - IGNORE 1. Execution - Just shows the map-reduce tasks completing 2. Performance - Execution + Performance counters dumped at the end 3. Verbose - All logs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9633) Add HCatClient.dropPartitions() overload to skip deletion of partition-directories.
[
https://issues.apache.org/jira/browse/HIVE-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485589#comment-14485589
]
Sushanth Sowmyan commented on HIVE-9633:
The test failures noted here are not related to this patch, +1.
Will go ahead and commit.
Add HCatClient.dropPartitions() overload to skip deletion of
partition-directories.
---
Key: HIVE-9633
URL: https://issues.apache.org/jira/browse/HIVE-9633
Project: Hive
Issue Type: Bug
Components: API, HCatalog, Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9633.1.patch
{{HCatClient.dropPartitions()}} doesn't provide a way to explicitly skip the
deletion of partition-directory, as {{HiveMetaStoreClient.dropPartitions()}}
does.
This'll come in handy when using HCatClient to drop partitions, but not
delete data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3378) UDF to obtain the numeric day of an year from date or timestamp in HIVE.
[ https://issues.apache.org/jira/browse/HIVE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485606#comment-14485606 ] Alexander Pivovarov commented on HIVE-3378: --- Actually, the latest test result shows 0 failures (±0) , 75 skipped (±0) UDF to obtain the numeric day of an year from date or timestamp in HIVE. Key: HIVE-3378 URL: https://issues.apache.org/jira/browse/HIVE-3378 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0 Reporter: Deepti Antony Assignee: Alexander Pivovarov Attachments: HIVE-3378.02.patch, HIVE-3378.02.patch, HIVE-3378.1.patch.txt Hive current releases lacks a function which returns the numeric day of an year if a date or timestamp is given .The function DAYOFYEAR(date) would return the numeric day from a date / timestamp or which would be useful while using HiveQL.DAYOFYEAR can be used to compare data with respect to number of days till the given date.It can be used in different domains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9752) Documentation for HBase metastore
[ https://issues.apache.org/jira/browse/HIVE-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485608#comment-14485608 ] Alan Gates commented on HIVE-9752: -- Need to document how to set the connection manager and what pointers to Tephra docs on how to set up Tephra. Documentation for HBase metastore - Key: HIVE-9752 URL: https://issues.apache.org/jira/browse/HIVE-9752 Project: Hive Issue Type: Sub-task Components: Documentation Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates All of the documentation we will need to write for the HBase metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10090) Add connection manager for Tephra
[ https://issues.apache.org/jira/browse/HIVE-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485610#comment-14485610 ] Alan Gates commented on HIVE-10090: --- Thanks for catching the docs requirement Lefty. I linked this to HIVE-9752. Add connection manager for Tephra - Key: HIVE-10090 URL: https://issues.apache.org/jira/browse/HIVE-10090 Project: Hive Issue Type: Sub-task Components: Metastore Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates Fix For: hbase-metastore-branch Attachments: HIVE-10090.2.patch, HIVE-10090.patch The task is to create an implementation of HBaseConnection that will use Tephra for transaction management. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9472) Implement 7 simple UDFs added to Hive
[
https://issues.apache.org/jira/browse/HIVE-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov resolved HIVE-9472.
---
Resolution: Fixed
MONTHS_BETWEEN was committed to trunk yesterday.
all other functions are already available
Implement 7 simple UDFs added to Hive
-
Key: HIVE-9472
URL: https://issues.apache.org/jira/browse/HIVE-9472
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
PROBLEM: Hive doesn't not have the following built-in functions.
1) MONTHS_BETWEEN(d1,d2)
2) LAST_DAY(d)
3) FIRST_DAY(d)
4) ADD_MONTHS (date, integer)
5) CHR()
6) REPLACE(char, search_string, replacement_string)
7) TRANSLATE(expr, from_string, to_string)
Without these, not able to do certain analysis.
USE CASE:
1) MONTHS_BETWEEN(d1,d2)
2) LAST_DAY(d)
3) FIRST_DAY(d)
4) ADD_MONTHS (date, integer)
ADD_MONTHS returns the date date plus integer months. If date is the last day
of the month or if the resulting month has fewer days than the day component
of date, then the result is the last day of the resulting month. Otherwise,
the result has the same day component as date.
5) CHR()
CHR returns the character having the binary equivalent to n (where n is
between 0 and 255)as a VARCHAR2 value
6) REPLACE(char, search_string, replacement_string)
REPLACE returns char with every occurrence of search_string replaced with
replacement_string.
SELECT REPLACE('JACK and JUE','J','BL') Changes;
Changes
--
BLACK and BLUE
7) TRANSLATE(expr, from_string, to_string)
TRANSLATE returns expr with all occurrences of each character in from_string
replaced by its corresponding character in to_string.
SELECT TRANSLATE('2KR 229', '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ ',
'99XX-') (CHAR(26)) License;
License
---
9XX-999
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9472) Implement 7 simple UDFs added to Hive
[
https://issues.apache.org/jira/browse/HIVE-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9472:
--
Fix Version/s: 1.2.0
Implement 7 simple UDFs added to Hive
-
Key: HIVE-9472
URL: https://issues.apache.org/jira/browse/HIVE-9472
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
Fix For: 1.2.0
PROBLEM: Hive doesn't not have the following built-in functions.
1) MONTHS_BETWEEN(d1,d2)
2) LAST_DAY(d)
3) FIRST_DAY(d)
4) ADD_MONTHS (date, integer)
5) CHR()
6) REPLACE(char, search_string, replacement_string)
7) TRANSLATE(expr, from_string, to_string)
Without these, not able to do certain analysis.
USE CASE:
1) MONTHS_BETWEEN(d1,d2)
2) LAST_DAY(d)
3) FIRST_DAY(d)
4) ADD_MONTHS (date, integer)
ADD_MONTHS returns the date date plus integer months. If date is the last day
of the month or if the resulting month has fewer days than the day component
of date, then the result is the last day of the resulting month. Otherwise,
the result has the same day component as date.
5) CHR()
CHR returns the character having the binary equivalent to n (where n is
between 0 and 255)as a VARCHAR2 value
6) REPLACE(char, search_string, replacement_string)
REPLACE returns char with every occurrence of search_string replaced with
replacement_string.
SELECT REPLACE('JACK and JUE','J','BL') Changes;
Changes
--
BLACK and BLUE
7) TRANSLATE(expr, from_string, to_string)
TRANSLATE returns expr with all occurrences of each character in from_string
replaced by its corresponding character in to_string.
SELECT TRANSLATE('2KR 229', '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ ',
'99XX-') (CHAR(26)) License;
License
---
9XX-999
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10222) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10222: --- Attachment: HIVE-10222.01.patch Triggering another QA run. Upgrade Calcite dependency to newest version Key: HIVE-10222 URL: https://issues.apache.org/jira/browse/HIVE-10222 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-10222.01.patch, HIVE-10222.patch Upgrade Calcite version to 1.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485243#comment-14485243 ] Chengxiang Li commented on HIVE-10189: -- +1 Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.2.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. Current score is as follows: BenchmarkMode Samples Score Error Units o.a.h.b.v.VectorizationBench.DoubleAddDoubleExpr.bench avgt2 20719.882 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleAddLongExpr.bench avgt2 22216.747 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideDoubleExpr.benchavgt2 54319.682 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideLongExpr.bench avgt2 34774.870 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddDoubleExpr.bench avgt2 47144.954 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddLongExpr.bench avgt2 21483.787 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideDoubleExpr.bench avgt2 49765.990 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideLongExpr.benchavgt2 34117.538 ? NaN ns/op -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10231) Compute partition column stats fails if partition col type is date
[
https://issues.apache.org/jira/browse/HIVE-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485244#comment-14485244
]
Chaoyu Tang commented on HIVE-10231:
Thanks, [~ashutoshc].
I looked into the report:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3319/testReport
Actually there are no test failures. I wonder if above 13 TestMinimrCliDriver
failures might be caused by some issue from build or report infra?
Compute partition column stats fails if partition col type is date
--
Key: HIVE-10231
URL: https://issues.apache.org/jira/browse/HIVE-10231
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 1.0.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-10231.1.patch, HIVE-10231.1.patch, HIVE-10231.patch
Currently the command analyze table .. partition .. compute statistics for
columns may only work for partition column type of string, numeric types,
but not others like date. See following case using date as partition coltype:
{code}
create table colstatspartdate (key int, value string) partitioned by (ds
date, hr int);
insert into colstatspartdate partition (ds=date '2015-04-02', hr=2) select
key, value from src limit 20;
analyze table colstatspartdate partition (ds=date '2015-04-02', hr=2) compute
statistics for columns;
{code}
you will get RuntimeException:
{code}
FAILED: RuntimeException Cannot convert to Date from: int
15/04/06 17:30:01 ERROR ql.Driver: FAILED: RuntimeException Cannot convert to
Date from: int
java.lang.RuntimeException: Cannot convert to Date from: int
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDate(PrimitiveObjectInspectorUtils.java:1048)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DateConverter.convert(PrimitiveObjectInspectorConverter.java:264)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.typeCast(ConstantPropagateProcFactory.java:163)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.propagate(ConstantPropagateProcFactory.java:333)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:242)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9917) After HIVE-3454 is done, make int to timestamp conversion configurable
[ https://issues.apache.org/jira/browse/HIVE-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485302#comment-14485302 ] Aihua Xu commented on HIVE-9917: Internally when we convert timestamp to numerical type, it converts to double so it should be fine. The latest patch fixed the unit tests and the unit tests failed above seem to be unrelated. [~jdere] Anything else I should check ? After HIVE-3454 is done, make int to timestamp conversion configurable -- Key: HIVE-9917 URL: https://issues.apache.org/jira/browse/HIVE-9917 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-9917.patch After HIVE-3454 is fixed, we will have correct behavior of converting int to timestamp. While the customers are using such incorrect behavior for so long, better to make it configurable so that in one release, it will default to old/inconsistent way and the next release will default to new/consistent way. And then we will deprecate it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485298#comment-14485298
]
Hive QA commented on HIVE-10122:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723749/HIVE-10122.patch
{color:red}ERROR:{color} -1 due to 171 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
[jira] [Commented] (HIVE-9558) [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485361#comment-14485361
]
Sergio Peña commented on HIVE-9558:
---
Thanks [~dongc]
The patch looks good.
+1
[Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable
in vectorized mode
---
Key: HIVE-9558
URL: https://issues.apache.org/jira/browse/HIVE-9558
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Attachments: HIVE-9558.1.patch, HIVE-9558.2.patch, HIVE-9558.patch
When using Parquet in vectorized mode,
{{VectorColumnAssignFactory.buildAssigners(..)}} does not handle
HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable.
We need fix this and add test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-3299) UDF DAYNAME(date) to HIVE
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-3299:
--
Description:
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
was:
Current releases of Hive lacks a function which would return the day name
corresponding to a date / timestamp value which might be a part of a column.
The function -DAYNAME (date) would return the day name from a date / timestamp
or column which would be useful while using HiveQL. This would find its use in
various business sectors like retail, which would help in identifying the
trends and sales datails for a particular weekday for entire year,month or week.
Functionality :-
Function Name: DAYNAME (date)
Returns the name of the weekday for date.
Example: hive SELECT DAYNAME('2012-07-25');
- 'Wednesday'
Usage :-
Case 1 : To find DAY NAME corresponding to a particular date
hive SELECT DAYNAME('2012-07-25');
- 'Wednesday'
Case 2 : To query a table to find details based on a particular day name
Table :-
date |item id|store id|value|unit|price
01/07/2012|110001|00003|0.99|1.00|0.99
02/07/2012|110001|00008|0.99|0.00|0.00
03/07/2012|110001|00009|0.99|0.00|0.00
04/07/2012|110001|001112002|0.99|0.00|0.00
05/07/2012|110001|001112003|0.99|0.00|0.00
06/07/2012|110001|001112006|0.99|1.00|0.99
07/07/2012|110001|001112007|0.99|0.00|0.00
08/07/2012|110001|001112008|0.99|0.00|0.00
09/07/2012|110001|001112009|0.99|0.00|0.00
10/07/2012|110001|001112010|0.99|0.00|0.00
11/07/2012|110001|001113003|0.99|0.00|0.00
12/07/2012|110001|001113006|0.99|0.00|0.00
13/07/2012|110001|001113008|0.99|0.00|0.00
14/07/2012|110001|001113010|0.99|0.00|0.00
15/07/2012|110001|001114002|0.99|0.00|0.00
16/07/2012|110001|001114004|0.99|1.00|0.99
17/07/2012|110001|001114005|0.99|0.00|0.00
18/07/2012|110001|001121004|0.99|0.00|0.00
Query : select * from sales where dayname(date)='wednesday';
Result :-
04/07/2012|110001|001112002|0.99|0.00|0.00
11/07/2012|110001|001113003|0.99|0.00|0.00
18/07/2012|110001|001121004|0.99|0.00|0.00
UDF DAYNAME(date) to HIVE
---
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Labels: patch
Attachments: HIVE-3299.1.patch.txt, HIVE-3299.patch.txt,
Hive-3299_Testcase.doc, udf_dayname.q, udf_dayname.q.out
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-3299) UDF DAYNAME(date) to HIVE
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov reassigned HIVE-3299:
-
Assignee: Alexander Pivovarov
UDF DAYNAME(date) to HIVE
---
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Assignee: Alexander Pivovarov
Labels: patch
Attachments: HIVE-3299.1.patch.txt, HIVE-3299.patch.txt,
Hive-3299_Testcase.doc, udf_dayname.q, udf_dayname.q.out
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks
[ https://issues.apache.org/jira/browse/HIVE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485796#comment-14485796 ] Lefty Leverenz commented on HIVE-8164: -- No doc needed? (Other than javadoc, of course.) Adding in a ReplicationTask that converts a Notification Event to actionable tasks -- Key: HIVE-8164 URL: https://issues.apache.org/jira/browse/HIVE-8164 Project: Hive Issue Type: Sub-task Components: Import/Export Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Fix For: 1.2.0 Attachments: HIVE-8164.2.patch, HIVE-8164.3.patch, HIVE-8164.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10262) CBO (Calcite Return Path): Temporarily disable Aggregate check input for bucketing
[ https://issues.apache.org/jira/browse/HIVE-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran resolved HIVE-10262. --- Resolution: Fixed CBO (Calcite Return Path): Temporarily disable Aggregate check input for bucketing -- Key: HIVE-10262 URL: https://issues.apache.org/jira/browse/HIVE-10262 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10160) Give a warning when grouping or ordering by a constant column
[
https://issues.apache.org/jira/browse/HIVE-10160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485799#comment-14485799
]
Hive QA commented on HIVE-10160:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723776/HIVE-10160.5.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8665 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3331/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3331/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3331/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723776 - PreCommit-HIVE-TRUNK-Build
Give a warning when grouping or ordering by a constant column
-
Key: HIVE-10160
URL: https://issues.apache.org/jira/browse/HIVE-10160
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Lefty Leverenz
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-10160.1.patch, HIVE-10160.3.patch,
HIVE-10160.4.patch, HIVE-10160.5.patch
To avoid confusion, a warning should be issued when users specify column
positions instead of names in a GROUP BY or ORDER BY clause (unless
hive.groupby.orderby.position.alias is set to true in Hive 0.11.0 or later).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10206) Improve Alter Table to not initialize Serde unnecessarily
[
https://issues.apache.org/jira/browse/HIVE-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-10206:
-
Attachment: HIVE-10206.2.patch
Attaching again, not sure why it did not trigger.
Improve Alter Table to not initialize Serde unnecessarily
-
Key: HIVE-10206
URL: https://issues.apache.org/jira/browse/HIVE-10206
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Szehon Ho
Priority: Minor
Attachments: HIVE-10206.2.patch, HIVE-10206.2.patch, HIVE-10206.patch
Create an avro table with an external avsc file like:
{noformat}
CREATE TABLE test(...)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='file:///Users/szehon/Temp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Delete test.avsc file.
Try to modify the table properties:
{noformat}
alter table test set tblproperties
('avro.schema.url'='file:///Users/szehon/Temp/test2.avsc');
{noformat}
Will throw an exception like AvroSerdeException:
{noformat}
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:119)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:163)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:101)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:78)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:520)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:377)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:274)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:256)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:595)
at
org.apache.hadoop.hive.ql.exec.DDLTask.alterTableOrSinglePartition(DDLTask.java:3383)
at
org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3340)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:332)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1181)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks
[ https://issues.apache.org/jira/browse/HIVE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485819#comment-14485819 ] Sushanth Sowmyan commented on HIVE-8164: Not yet, at least - I should create a full wiki page on replication support - I've now created a wiki page for that : HIVE-10264 Adding in a ReplicationTask that converts a Notification Event to actionable tasks -- Key: HIVE-8164 URL: https://issues.apache.org/jira/browse/HIVE-8164 Project: Hive Issue Type: Sub-task Components: Import/Export Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Fix For: 1.2.0 Attachments: HIVE-8164.2.patch, HIVE-8164.3.patch, HIVE-8164.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3299) Create UDF DAYNAME(date)
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-3299:
--
Summary: Create UDF DAYNAME(date) (was: UDF DAYNAME(date) to HIVE )
Create UDF DAYNAME(date)
-
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Assignee: Alexander Pivovarov
Labels: patch
Attachments: HIVE-3299.1.patch.txt, HIVE-3299.patch.txt,
Hive-3299_Testcase.doc, udf_dayname.q, udf_dayname.q.out
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10264) Document Replication support on wiki
[ https://issues.apache.org/jira/browse/HIVE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10264: Attachment: apache_hivedr.0.pdf Attaching content presentation pdf describing some of the why/what/how of this replication development. I'll update this as we go along to become a full wiki page. Document Replication support on wiki Key: HIVE-10264 URL: https://issues.apache.org/jira/browse/HIVE-10264 Project: Hive Issue Type: Sub-task Components: Import/Export Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: apache_hivedr.0.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3299) Create UDF DAYNAME(date)
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-3299:
--
Attachment: HIVE-3299.2.patch
patch #2
- the function extends GenericUDF and uses GenericUDF methods.
- added JUnit tests
- added q test for string/date/timestamps arguments
Create UDF DAYNAME(date)
-
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Assignee: Alexander Pivovarov
Labels: patch
Attachments: HIVE-3299.1.patch.txt, HIVE-3299.2.patch,
HIVE-3299.patch.txt, Hive-3299_Testcase.doc, udf_dayname.q, udf_dayname.q.out
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-3200) Results of the assert_true UDF are not retrieving through ResultSet
[
https://issues.apache.org/jira/browse/HIVE-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov resolved HIVE-3200.
---
Resolution: Won't Fix
Looks like assert_trueUDF was created by huawei. So, it is not Hive built-in
UDF. Close Jira.
{code}
at
com.huawei.isap.test.smartump.hive.Hive_UDF_FT.testassert_trueUDF(Hive_UDF_FT.java:2100)
{code}
Results of the assert_true UDF are not retrieving through ResultSet
---
Key: HIVE-3200
URL: https://issues.apache.org/jira/browse/HIVE-3200
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.9.0
Reporter: Kristam Subba Swathi
Please find the exception trace for more details
{noformat}
ava.lang.RuntimeException: Internal error: no LazyObject for VOID
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyPrimitiveClass(LazyFactory.java:116)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObject(LazyFactory.java:152)
at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.parse(LazyStruct.java:108)
at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.getField(LazyStruct.java:190)
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:188)
at
org.apache.hadoop.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:169)
at
com.huawei.isap.test.smartump.hive.Hive_UDF_FT.testassert_trueUDF(Hive_UDF_FT.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10231) Compute partition column stats fails if partition col type is date
[
https://issues.apache.org/jira/browse/HIVE-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485431#comment-14485431
]
Aihua Xu commented on HIVE-10231:
-
I didn't look into what causes the failures but those are not related to your
change. I also got the same list of failures from my test.
Compute partition column stats fails if partition col type is date
--
Key: HIVE-10231
URL: https://issues.apache.org/jira/browse/HIVE-10231
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 1.0.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-10231.1.patch, HIVE-10231.1.patch, HIVE-10231.patch
Currently the command analyze table .. partition .. compute statistics for
columns may only work for partition column type of string, numeric types,
but not others like date. See following case using date as partition coltype:
{code}
create table colstatspartdate (key int, value string) partitioned by (ds
date, hr int);
insert into colstatspartdate partition (ds=date '2015-04-02', hr=2) select
key, value from src limit 20;
analyze table colstatspartdate partition (ds=date '2015-04-02', hr=2) compute
statistics for columns;
{code}
you will get RuntimeException:
{code}
FAILED: RuntimeException Cannot convert to Date from: int
15/04/06 17:30:01 ERROR ql.Driver: FAILED: RuntimeException Cannot convert to
Date from: int
java.lang.RuntimeException: Cannot convert to Date from: int
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDate(PrimitiveObjectInspectorUtils.java:1048)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DateConverter.convert(PrimitiveObjectInspectorConverter.java:264)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.typeCast(ConstantPropagateProcFactory.java:163)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.propagate(ConstantPropagateProcFactory.java:333)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:242)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10231) Compute partition column stats fails if partition col type is date
[
https://issues.apache.org/jira/browse/HIVE-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10231:
Affects Version/s: 1.1.0
0.14.0
Compute partition column stats fails if partition col type is date
--
Key: HIVE-10231
URL: https://issues.apache.org/jira/browse/HIVE-10231
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-10231.1.patch, HIVE-10231.1.patch, HIVE-10231.patch
Currently the command analyze table .. partition .. compute statistics for
columns may only work for partition column type of string, numeric types,
but not others like date. See following case using date as partition coltype:
{code}
create table colstatspartdate (key int, value string) partitioned by (ds
date, hr int);
insert into colstatspartdate partition (ds=date '2015-04-02', hr=2) select
key, value from src limit 20;
analyze table colstatspartdate partition (ds=date '2015-04-02', hr=2) compute
statistics for columns;
{code}
you will get RuntimeException:
{code}
FAILED: RuntimeException Cannot convert to Date from: int
15/04/06 17:30:01 ERROR ql.Driver: FAILED: RuntimeException Cannot convert to
Date from: int
java.lang.RuntimeException: Cannot convert to Date from: int
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDate(PrimitiveObjectInspectorUtils.java:1048)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DateConverter.convert(PrimitiveObjectInspectorConverter.java:264)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.typeCast(ConstantPropagateProcFactory.java:163)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.propagate(ConstantPropagateProcFactory.java:333)
at
org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:242)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10250) Optimize AuthorizationPreEventListener to reuse TableWrapper objects
[
https://issues.apache.org/jira/browse/HIVE-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan reassigned HIVE-10250:
---
Assignee: Mithun Radhakrishnan
Optimize AuthorizationPreEventListener to reuse TableWrapper objects
Key: HIVE-10250
URL: https://issues.apache.org/jira/browse/HIVE-10250
Project: Hive
Issue Type: Bug
Components: Authorization
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-10250.1.patch
Here's the {{PartitionWrapper}} class in {{AuthorizationPreEventListener}}:
{code:java|title=AuthorizationPreEventListener.java}
public static class PartitionWrapper extends
org.apache.hadoop.hive.ql.metadata.Partition {
...
public PartitionWrapper(org.apache.hadoop.hive.metastore.api.Partition
mapiPart, PreEventContext context) throws ... {
Partition wrapperApiPart = mapiPart.deepCopy();
Table t = context.getHandler().get_table_core(
mapiPart.getDbName(),
mapiPart.getTableName());
...
}
{code}
{{PreAddPartitionEvent}} (and soon, {{PreDropPartitionEvent}}) correspond not
just to a single partition, but an entire set of partitions added atomically.
When the event is authorized, {{HMSHandler.get_table_core()}} will be called
once for every partition in the Event instance.
Since we already make the assumption that the partition-sets correspond to a
single table, we might as well make a single call.
I'll have a patch for this, shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10226:
--
Attachment: HIVE-10226.4.patch
minor changes per comments from [~swarnim]
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10226:
--
Attachment: HIVE-10226.4.patch
Previously uploaded wrong version of v4 patch, uploading correct patch.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9609) AddPartitionMessage.getPartitions() can return null
[ https://issues.apache.org/jira/browse/HIVE-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485663#comment-14485663 ] Alan Gates commented on HIVE-9609: -- +1 AddPartitionMessage.getPartitions() can return null --- Key: HIVE-9609 URL: https://issues.apache.org/jira/browse/HIVE-9609 Project: Hive Issue Type: Bug Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-9609.2.patch, HIVE-9609.3.patch, HIVE-9609.patch DbNotificationListener and NotificationListener both depend on AddPartitionEvent.getPartitions() to get their partitions to trigger a message, but this can be null if an AddPartitionEvent was initialized on a PartitionSpec rather than a ListPartition. Also, AddPartitionEvent seems to have a duality, where getPartitions() works only if instantiated on a ListPartition, and getPartitionIterator() works only if instantiated on a PartitionSpec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9633) Add HCatClient.dropPartitions() overload to skip deletion of partition-directories.
[
https://issues.apache.org/jira/browse/HIVE-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485622#comment-14485622
]
Sushanth Sowmyan commented on HIVE-9633:
Committed to trunk. Thanks, Mithun!
Add HCatClient.dropPartitions() overload to skip deletion of
partition-directories.
---
Key: HIVE-9633
URL: https://issues.apache.org/jira/browse/HIVE-9633
Project: Hive
Issue Type: Bug
Components: API, HCatalog, Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Fix For: 1.2.0
Attachments: HIVE-9633.1.patch
{{HCatClient.dropPartitions()}} doesn't provide a way to explicitly skip the
deletion of partition-directory, as {{HiveMetaStoreClient.dropPartitions()}}
does.
This'll come in handy when using HCatClient to drop partitions, but not
delete data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485640#comment-14485640
]
Jason Dere commented on HIVE-10226:
---
test failures do not look related
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10226:
--
Attachment: (was: HIVE-10226.4.patch)
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-3510) to_utc_timestamp() does not properly handle UNIX timestamp conversions without nanoseconds
[
https://issues.apache.org/jira/browse/HIVE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov resolved HIVE-3510.
---
Resolution: Cannot Reproduce
Assignee: Alexander Pivovarov
checked this on 1.2.0 - function output is correct
{code}
hive SELECT
to_utc_timestamp(unix_timestamp('27/Sep/2012:09:53:35','dd/MMM/:HH:mm:ss'),
'America/Denver');
OK
2012-09-27 15:53:35
{code}
to_utc_timestamp() does not properly handle UNIX timestamp conversions
without nanoseconds
--
Key: HIVE-3510
URL: https://issues.apache.org/jira/browse/HIVE-3510
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.9.0
Environment: Ubuntu LTS 10.04
Reporter: Richard Nadeau
Assignee: Alexander Pivovarov
When attempting to convert a UNIX timestamp to UTC the conversion fails if
the input does not include micro or nano seconds:
{code}SELECT
to_utc_timestamp(unix_timestamp('27/Sep/2012:09:53:35','dd/MMM/:HH:mm:ss'),'America/Denver')
FROM source_table LIMIT 1;{code}
*1970-01-16 14:39:21.215*
Converting to a float provides a workaround:
{code}SELECT
to_utc_timestamp(unix_timestamp('27/Sep/2012:09:53:35','dd/MMM/:HH:mm:ss')
* 1.0,'America/Denver') FROM source_table LIMIT 1;{code}
*2012-09-27 15:53:35.0*
As does adding nano seconds:
{code}SELECT
to_utc_timestamp(unix_timestamp('27/Sep/2012:09:53:35','dd/MMM/:HH:mm:ss')
* 1000,'America/Denver') FROM source_table LIMIT 1;{code}
*2012-09-27 15:53:35.0*
This problem may be at the heart of HIVE-3454 as well.
Documentation here:
https://cwiki.apache.org/Hive/languagemanual-types.html#LanguageManualTypes-Timestamps
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3604) Casting strings to booleans gives unexpected result
[
https://issues.apache.org/jira/browse/HIVE-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485688#comment-14485688
]
Alexander Pivovarov commented on HIVE-3604:
---
Added If cast(exp as boolean) hive returns true for non-empty string. to
LanguageManual UDF
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-TypeConversionFunctions
Casting strings to booleans gives unexpected result
---
Key: HIVE-3604
URL: https://issues.apache.org/jira/browse/HIVE-3604
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.8.1
Environment: Amazon EMR
Reporter: Ariel M
I would expect the following query to return false in both columns:
select 'false', cast('false' as boolean) from example_table limit 1;
but here are the actual results:
Total MapReduce CPU Time Spent: X seconds X msec
OK
false true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-3604) Casting strings to booleans gives unexpected result
[
https://issues.apache.org/jira/browse/HIVE-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov resolved HIVE-3604.
---
Resolution: Won't Fix
Casting strings to booleans gives unexpected result
---
Key: HIVE-3604
URL: https://issues.apache.org/jira/browse/HIVE-3604
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.8.1
Environment: Amazon EMR
Reporter: Ariel M
I would expect the following query to return false in both columns:
select 'false', cast('false' as boolean) from example_table limit 1;
but here are the actual results:
Total MapReduce CPU Time Spent: X seconds X msec
OK
false true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-5162) Add mod(a, b) semantic for modular function
[
https://issues.apache.org/jira/browse/HIVE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov reopened HIVE-5162:
---
Assignee: Alexander Pivovarov (was: Anandha L Ranganathan)
pmod is not the same as mod
e.g.
{code}
select mod(-35, 6), pmod(-35, 6)
-51
{code}
Add mod(a, b) semantic for modular function
---
Key: HIVE-5162
URL: https://issues.apache.org/jira/browse/HIVE-5162
Project: Hive
Issue Type: Improvement
Components: UDF
Affects Versions: 0.11.0
Reporter: Xiu
Assignee: Alexander Pivovarov
Priority: Minor
Currently Hive is supporting 'a % b' as modular function, not mod(a, b).
Unlike MySQL which supports both.
It would be a good idea to support both.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10239) Create scripts to do metastore upgrade tests on jenkins for Derby, Oracle and PostgreSQL
[
https://issues.apache.org/jira/browse/HIVE-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485893#comment-14485893
]
Naveen Gangam commented on HIVE-10239:
--
Appears the pre-commit build failed because of the way the patch file was
named.
{code}
Logs are located:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3318
Exception in thread main java.lang.RuntimeException: Status
[name=ILLEGAL_ARGUMENT, message=Profile DONOTCOMMIT-mr2 not found]
at org.apache.hive.ptest.api.Status.assertOKOrFailed(Status.java:69)
at
org.apache.hive.ptest.api.client.PTestClient.testTailLog(PTestClient.java:170)
at
org.apache.hive.ptest.api.client.PTestClient.testStart(PTestClient.java:127)
at
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:312)
{code}
I will rename and re-attach the patch.
Create scripts to do metastore upgrade tests on jenkins for Derby, Oracle and
PostgreSQL
Key: HIVE-10239
URL: https://issues.apache.org/jira/browse/HIVE-10239
Project: Hive
Issue Type: Improvement
Affects Versions: 1.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Attachments: HIVE-10239-DONOTCOMMIT.patch, HIVE-10239.patch
Need to create DB-implementation specific scripts to use the framework
introduced in HIVE-9800 to have any metastore schema changes tested across
all supported databases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10239) Create scripts to do metastore upgrade tests on jenkins for Derby, Oracle and PostgreSQL
[ https://issues.apache.org/jira/browse/HIVE-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-10239: - Attachment: HIVE-10239.DONOTCOMMIT.patch Create scripts to do metastore upgrade tests on jenkins for Derby, Oracle and PostgreSQL Key: HIVE-10239 URL: https://issues.apache.org/jira/browse/HIVE-10239 Project: Hive Issue Type: Improvement Affects Versions: 1.1.0 Reporter: Naveen Gangam Assignee: Naveen Gangam Attachments: HIVE-10239-DONOTCOMMIT.patch, HIVE-10239.DONOTCOMMIT.patch, HIVE-10239.patch Need to create DB-implementation specific scripts to use the framework introduced in HIVE-9800 to have any metastore schema changes tested across all supported databases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5021) Add Interval data type
[ https://issues.apache.org/jira/browse/HIVE-5021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485926#comment-14485926 ] Mostafa Mokhtar commented on HIVE-5021: --- [~jdere] Should this be resolved? Add Interval data type -- Key: HIVE-5021 URL: https://issues.apache.org/jira/browse/HIVE-5021 Project: Hive Issue Type: Bug Components: Types Reporter: Jason Dere Assignee: Jason Dere If we want to support arithmetic operations involving date/timestamp types, we should add support for the interval data type where the result is an offset of years/months/days/hours/minutes/seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485993#comment-14485993 ] Chaoyu Tang commented on HIVE-5376: --- [~sershe] [~hsubramaniyan] Could you elaborate the comment on https://issues.apache.org/jira/browse/HIVE-5376?focusedCommentId=13960805page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13960805 I do not quite understand what the difficulty is to implement this JIRA at metastore site and why metastore needs QL. Thanks. Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.14.0 Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10243) Introduce JoinAlgorithm Interface
[ https://issues.apache.org/jira/browse/HIVE-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10243: --- Attachment: HIVE-10243.cbo.patch Introduce JoinAlgorithm Interface - Key: HIVE-10243 URL: https://issues.apache.org/jira/browse/HIVE-10243 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-10243.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7155) WebHCat controller job exceeds container memory limit
[
https://issues.apache.org/jira/browse/HIVE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486727#comment-14486727
]
Lefty Leverenz commented on HIVE-7155:
--
Doc note: *templeton.mapper.memory.mb* is documented in the WebHCat
Configuration wiki, at the end of the table of configuration variables.
(Better late than never.) I took the liberty of changing Templeton to
WebHCat in the description -- should have thought of that before the commit.
* [WebHCat Configuration -- Configuration Variables |
https://cwiki.apache.org/confluence/display/Hive/WebHCat+Configure#WebHCatConfigure-ConfigurationVariables]
WebHCat controller job exceeds container memory limit
-
Key: HIVE-7155
URL: https://issues.apache.org/jira/browse/HIVE-7155
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: shanyu zhao
Assignee: shanyu zhao
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7155.1.patch, HIVE-7155.2.patch, HIVE-7155.patch
Submit a Hive query on a large table via WebHCat results in failure because
the WebHCat controller job is killed by Yarn since it exceeds the memory
limit (set by mapreduce.map.memory.mb, defaults to 1GB):
{code}
INSERT OVERWRITE TABLE Temp_InjusticeEvents_2014_03_01_00_00 SELECT * from
Stage_InjusticeEvents where LogTimestamp '2014-03-01 00:00:00' and
LogTimestamp = '2014-03-01 01:00:00';
{code}
We could increase mapreduce.map.memory.mb to solve this problem, but this way
we are changing this setting system wise.
We need to provide a WebHCat configuration to overwrite
mapreduce.map.memory.mb when submitting the controller job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9710) HiveServer2 should support cookie based authentication, when using HTTP transport.
[ https://issues.apache.org/jira/browse/HIVE-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-9710: Attachment: HIVE-9710.6.patch Including : 1. Secure flag for http Cookie 2. isHttpOnly flag 3. fixing the unit test failures 4. Integrating [~vgumashta]'s comments from previous review. Thanks Hari HiveServer2 should support cookie based authentication, when using HTTP transport. -- Key: HIVE-9710 URL: https://issues.apache.org/jira/browse/HIVE-9710 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-9710.1.patch, HIVE-9710.2.patch, HIVE-9710.3.patch, HIVE-9710.4.patch, HIVE-9710.5.patch, HIVE-9710.6.patch HiveServer2 should generate cookies and validate the client cookie send to it so that it need not perform User/Password or a Kerberos based authentication on each HTTP request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10273) Union with partition tables which have no data fails with NPE
[ https://issues.apache.org/jira/browse/HIVE-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-10273: - Assignee: Vikram Dixit K Union with partition tables which have no data fails with NPE - Key: HIVE-10273 URL: https://issues.apache.org/jira/browse/HIVE-10273 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 1.2.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486145#comment-14486145
]
Mithun Radhakrishnan commented on HIVE-9674:
Actually, [~sushanth], let's hold off for right now, on this one. I'll rebase
this under the assumption that HIVE-9609 is good to go.
*DropPartitionEvent should handle partition-sets.
-
Key: HIVE-9674
URL: https://issues.apache.org/jira/browse/HIVE-9674
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9674.2.patch
Dropping a set of N partitions from a table currently results in N
DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
is wasteful, especially so for large N. It also makes it impossible to even
try to run authorization-checks on all partitions in a batch.
Taking the cue from HIVE-9609, we should compose an {{IterablePartition}}
in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10271) remove hive.server2.thrift.http.min/max.worker.threads properties
[ https://issues.apache.org/jira/browse/HIVE-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486517#comment-14486517 ] Thejas M Nair commented on HIVE-10271: -- +1 remove hive.server2.thrift.http.min/max.worker.threads properties - Key: HIVE-10271 URL: https://issues.apache.org/jira/browse/HIVE-10271 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-10271.1.patch PROBLEM: Those properties are not used even when hiveserver2 in http mode. The properties used are hive.server2.thrift.min/max.worker.threads. Remove those 2 properties as they are causing confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10275) GenericUDF getTimestampValue should return Timestamp instead of Date
[ https://issues.apache.org/jira/browse/HIVE-10275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-10275: --- Component/s: UDF GenericUDF getTimestampValue should return Timestamp instead of Date Key: HIVE-10275 URL: https://issues.apache.org/jira/browse/HIVE-10275 Project: Hive Issue Type: Bug Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Currently getTimestampValue casts Timestamp to Date and returns Date. Hive Timestamp types stores time with nanosecond precision. Timestamp class has getNanos method to extract nanoseconds. Date class has getTime method which retuns unix time in milliseconds. So, it order to be able to get nanoseconds from Timestamp fields GenericUDF getTimestampValue should return Timestamp instead of Date. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3241) GenericUDAF when packaged as a JAR raises IllegalAccessException unless the JAR is placed in lib dir
[ https://issues.apache.org/jira/browse/HIVE-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3241: -- Attachment: HIVE-3241.1.patch patch #1 GenericUDAF when packaged as a JAR raises IllegalAccessException unless the JAR is placed in lib dir Key: HIVE-3241 URL: https://issues.apache.org/jira/browse/HIVE-3241 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.9.0 Reporter: Shreepadma Venugopalan Attachments: HIVE-3241.1.patch, compute-stats.jar GenericUDAFEvaluator has a member variable Mode which is protected. Accessing mode in a GenericUDAF class raises IllegalAccessException when packaged as a jar. This looks like a class loader issue because the error doesn't occur when the jar is placed in HIVE_LIB directory. The fix for this issue is to provide public getters for member variable mode in GenericUDAFEvaluator. Steps to reproduce, Copy the attached jar to a dir other than HIVE_LIB On a hive shell, add jar path-to-jar create temporary function compute_stats as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats'; select compute_stats(some_col) from some_table; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10274) Send context and description to tez via dag info
[ https://issues.apache.org/jira/browse/HIVE-10274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-10274: -- Attachment: HIVE-10274.1.patch Send context and description to tez via dag info Key: HIVE-10274 URL: https://issues.apache.org/jira/browse/HIVE-10274 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-10274.1.patch tez has a way to specify context and description (which is shown in the ui) for each dag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3299) Create UDF DAYNAME(date)
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-3299:
--
Attachment: HIVE-3299.3.patch
patch #3
- fixed GenericUDFDayName class comments
Create UDF DAYNAME(date)
-
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Assignee: Alexander Pivovarov
Labels: patch
Attachments: HIVE-3299.1.patch.txt, HIVE-3299.2.patch,
HIVE-3299.3.patch, HIVE-3299.patch.txt, Hive-3299_Testcase.doc,
udf_dayname.q, udf_dayname.q.out
dayname(date/timestamp/string)
Returns the name of the weekday for date. The language used for the name is
English.
select dayname('2015-04-08');
Wednesday
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10270) Cannot use Decimal constant 0.09765625BD
[
https://issues.apache.org/jira/browse/HIVE-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486353#comment-14486353
]
Sergey Shelukhin commented on HIVE-10270:
-
I think this should be a blocker, number 0.09765625 is very important in my use
of Hive
Cannot use Decimal constant 0.09765625BD
Key: HIVE-10270
URL: https://issues.apache.org/jira/browse/HIVE-10270
Project: Hive
Issue Type: Bug
Components: Types
Reporter: Jason Dere
Assignee: Jason Dere
{noformat}
hive select 0.09765625BD;
FAILED: IllegalArgumentException Decimal scale must be less than or equal to
precision
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9115) Hive build failure on hadoop-2.7 due to HADOOP-11356
[
https://issues.apache.org/jira/browse/HIVE-9115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486392#comment-14486392
]
Hive QA commented on HIVE-9115:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723799/HIVE-9115.3.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3334/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3334/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3334/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723799 - PreCommit-HIVE-TRUNK-Build
Hive build failure on hadoop-2.7 due to HADOOP-11356
Key: HIVE-9115
URL: https://issues.apache.org/jira/browse/HIVE-9115
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-9115.1.patch, HIVE-9115.2.patch, HIVE-9115.3.patch
HADOOP-11356 removes org.apache.hadoop.fs.permission.AccessControlException,
causing build break on Hive when compiling against hadoop-2.7:
{noformat}
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[808,63]
cannot find symbol
symbol: class AccessControlException
location: package org.apache.hadoop.fs.permission
[INFO] 1 error
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10190) CBO: AST mode checks for TABLESAMPLE with AST.toString().contains(TOK_TABLESPLITSAMPLE)
[
https://issues.apache.org/jira/browse/HIVE-10190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laljo John Pullokkaran updated HIVE-10190:
--
Assignee: Pengcheng Xiong (was: Laljo John Pullokkaran)
CBO: AST mode checks for TABLESAMPLE with
AST.toString().contains(TOK_TABLESPLITSAMPLE)
-
Key: HIVE-10190
URL: https://issues.apache.org/jira/browse/HIVE-10190
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 1.2.0
Reporter: Gopal V
Assignee: Pengcheng Xiong
Priority: Trivial
Labels: perfomance
{code}
public static boolean validateASTForUnsupportedTokens(ASTNode ast) {
String astTree = ast.toStringTree();
// if any of following tokens are present in AST, bail out
String[] tokens = { TOK_CHARSETLITERAL, TOK_TABLESPLITSAMPLE };
for (String token : tokens) {
if (astTree.contains(token)) {
return false;
}
}
return true;
}
{code}
This is an issue for a SQL query which is bigger in AST form than in text
(~700kb).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10272) Some HCat tests fail under windows
[
https://issues.apache.org/jira/browse/HIVE-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486463#comment-14486463
]
Sushanth Sowmyan commented on HIVE-10272:
-
[~hsubramaniyan], could you please review? Thanks!
Some HCat tests fail under windows
--
Key: HIVE-10272
URL: https://issues.apache.org/jira/browse/HIVE-10272
Project: Hive
Issue Type: Bug
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10272.patch
Some HCat tests fail under windows with errors like this:
{noformat}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Pathname
/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/scratchdir from
D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/scratchdir is not a valid
DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:197)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at
org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:594)
at
org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:552)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:185)
{noformat}
We need to sanitize HiveConf objects with
WindowsPathUtil.convertPathsFromWindowsToHdfs if running under windows before
we use them to instantiate a SessionState/Driver
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10190) CBO: AST mode checks for TABLESAMPLE with AST.toString().contains(TOK_TABLESPLITSAMPLE)
[
https://issues.apache.org/jira/browse/HIVE-10190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486058#comment-14486058
]
Pengcheng Xiong commented on HIVE-10190:
I would prefer a BFS rather than a DFS. And I think it will solve the problem.
If it is still an issue, i can try to solve it in the parser level.
CBO: AST mode checks for TABLESAMPLE with
AST.toString().contains(TOK_TABLESPLITSAMPLE)
-
Key: HIVE-10190
URL: https://issues.apache.org/jira/browse/HIVE-10190
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 1.2.0
Reporter: Gopal V
Assignee: Pengcheng Xiong
Priority: Trivial
Labels: perfomance
{code}
public static boolean validateASTForUnsupportedTokens(ASTNode ast) {
String astTree = ast.toStringTree();
// if any of following tokens are present in AST, bail out
String[] tokens = { TOK_CHARSETLITERAL, TOK_TABLESPLITSAMPLE };
for (String token : tokens) {
if (astTree.contains(token)) {
return false;
}
}
return true;
}
{code}
This is an issue for a SQL query which is bigger in AST form than in text
(~700kb).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10266) Boolean expression True and True returns False
[
https://issues.apache.org/jira/browse/HIVE-10266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486286#comment-14486286
]
Jason Dere commented on HIVE-10266:
---
I think this is the result of HIVE-9278, fixed in Hive 1.0
As a workaround in 0.14, you can set hive.cache.expr.evaluation=false and see
if that fixes the issue.
Boolean expression True and True returns False
--
Key: HIVE-10266
URL: https://issues.apache.org/jira/browse/HIVE-10266
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.14.0
Reporter: ckran
Fix For: 0.13.0
A Hive query with a Boolean expression with day and month calculations that
each evaluate to TRUE with use of AND evaluates to FALSE.
create table datest (cntr int, date date ) row format delimited fields
terminated by ',' stored as textfile ;
insert into datest values (1,'2015-04-8') ;
select
((DAY('2015-05-25') - DAY(DATE)) 25),
((MONTH('2015-05-25') - MONTH(DATE)) = 1) ,
((DAY('2015-05-25') - DAY(DATE)) 25) AND ((MONTH('2015-05-25') -
MONTH(DATE)) = 1)
from datest
Returns values
True | True | False
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-5162) Add mod(a, b) semantic for modular function
[ https://issues.apache.org/jira/browse/HIVE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-5162: -- Attachment: HIVE-5162.1.patch patch #1 - added GenericUDFMod which extends GenericUDFOPMod - fixed function name and signature Add mod(a, b) semantic for modular function --- Key: HIVE-5162 URL: https://issues.apache.org/jira/browse/HIVE-5162 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.11.0 Reporter: Xiu Assignee: Alexander Pivovarov Priority: Minor Attachments: HIVE-5162.1.patch Currently Hive is supporting 'a % b' as modular function, not mod(a, b). Unlike MySQL which supports both. It would be a good idea to support both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10267) HIVE-9664 makes hive depend on ivysettings.xml : trivial breakage fix
[
https://issues.apache.org/jira/browse/HIVE-10267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10267:
Summary: HIVE-9664 makes hive depend on ivysettings.xml : trivial breakage
fix (was: HIVE-9664 makes hive depend on ivysettings.xml)
HIVE-9664 makes hive depend on ivysettings.xml : trivial breakage fix
-
Key: HIVE-10267
URL: https://issues.apache.org/jira/browse/HIVE-10267
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Anant Nag
Labels: patch
HIVE-9664 makes hive depend on the existence of ivysettings.xml, and if it is
not present, it makes hive NPE when instantiating a CLISessionState.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.session.DependencyResolver.init(DependencyResolver.java:61)
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:343)
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:334)
at org.apache.hadoop.hive.cli.CliSessionState.init(CliSessionState.java:60)
{noformat}
This happens because of the following bit:
{noformat}
// If HIVE_HOME is not defined or file is not found in HIVE_HOME/conf
then load default ivysettings.xml from class loader
if (ivysettingsPath == null || !(new File(ivysettingsPath).exists())) {
ivysettingsPath =
ClassLoader.getSystemResource(ivysettings.xml).getFile();
_console.printInfo(ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR, + ivysettingsPath + will be used);
}
{noformat}
This makes it so that an attempt to instantiate CliSessionState without an
ivysettings.xml file will cause hive to fail with an NPE. Hive should not
have a hard dependency on a ivysettings,xml being present, and this feature
should gracefully fail in that case instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486078#comment-14486078 ] Sergey Shelukhin commented on HIVE-5376: I think whatever that JIRA was doing needs some code from ql module. If it doesn't then the below is irrelevant. If it does, the problem is that ql module depends on metastore module, so the reverse dependency cannot be added at compile time. So, if you look at Expr stuff, it has a proxy class where interface is in metastore, impl is in ql, and it's instantiated in metastore once via reflection. Or someone could unf*** Hive module structure, but that seems like too big of a task :) Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.14.0 Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10269) HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10269: Attachment: HIVE-10269.patch HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor - Key: HIVE-10269 URL: https://issues.apache.org/jira/browse/HIVE-10269 Project: Hive Issue Type: Bug Components: Metastore Reporter: Gabor Liptak Assignee: Ferdinand Xu Attachments: HIVE-10269.patch Compiling trunk fails when building based on instructions in https://cwiki.apache.org/confluence/display/Hive/HowToContribute $ git status On branch trunk Your branch is up-to-date with 'origin/trunk'. nothing to commit, working directory clean $ mvn clean install -DskipTests -Phadoop-1 ...[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-metastore: Compilation failure: Compilation failure: [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6089,29] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6090,35] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hive-metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10243) Introduce JoinAlgorithm Interface
[ https://issues.apache.org/jira/browse/HIVE-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10243: --- Attachment: HIVE-10243.cbo.patch Introduce JoinAlgorithm Interface - Key: HIVE-10243 URL: https://issues.apache.org/jira/browse/HIVE-10243 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-10243.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9647) Discrepancy in cardinality estimates between partitioned and un-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486266#comment-14486266
]
Hive QA commented on HIVE-9647:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723797/HIVE-9647.03.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8666 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723797 - PreCommit-HIVE-TRUNK-Build
Discrepancy in cardinality estimates between partitioned and un-partitioned
tables
---
Key: HIVE-9647
URL: https://issues.apache.org/jira/browse/HIVE-9647
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 0.14.0
Reporter: Mostafa Mokhtar
Assignee: Pengcheng Xiong
Fix For: 1.2.0
Attachments: HIVE-9647.01.patch, HIVE-9647.02.patch,
HIVE-9647.03.patch
High-level summary
HiveRelMdSelectivity.computeInnerJoinSelectivity relies on per column number
of distinct value to estimate join selectivity.
The way statistics are aggregated for partitioned tables results in
discrepancy in number of distinct values which results in different plans
between partitioned and un-partitioned schemas.
The table below summarizes the NDVs in computeInnerJoinSelectivity which are
used to estimate selectivity of joins.
||Column ||Partitioned count distincts|| Un-Partitioned count
distincts
|sr_customer_sk |71,245 |1,415,625|
|sr_item_sk |38,846|62,562|
|sr_ticket_number |71,245 |34,931,085|
|ss_customer_sk |88,476|1,415,625|
|ss_item_sk |38,846|62,562|
|ss_ticket_number|100,756 |56,256,175|
The discrepancy is because NDV calculation for a partitioned table assumes
that the NDV range is contained within each partition and is calculates as
select max(NUM_DISTINCTS) from PART_COL_STATS” .
This is problematic for columns like ticket number which are naturally
increasing with the partitioned date column ss_sold_date_sk.
Suggestions
Use Hyper Log Log as suggested by Gopal, there is an HLL implementation for
HBASE co-porccessors which we can use as a reference here
Using the global stats from TAB_COL_STATS and the per
[jira] [Commented] (HIVE-10263) CBO (Calcite Return Path): Aggregate checking input for bucketing should be conditional
[ https://issues.apache.org/jira/browse/HIVE-10263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486413#comment-14486413 ] Jesus Camacho Rodriguez commented on HIVE-10263: Btw, run CBO tests and passed. CBO (Calcite Return Path): Aggregate checking input for bucketing should be conditional --- Key: HIVE-10263 URL: https://issues.apache.org/jira/browse/HIVE-10263 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-10263.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10222) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10222: --- Attachment: HIVE-10222.01.patch Upgrade Calcite dependency to newest version Key: HIVE-10222 URL: https://issues.apache.org/jira/browse/HIVE-10222 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-10222.01.patch, HIVE-10222.patch Upgrade Calcite version to 1.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10222) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10222: --- Attachment: (was: HIVE-10222.01.patch) Upgrade Calcite dependency to newest version Key: HIVE-10222 URL: https://issues.apache.org/jira/browse/HIVE-10222 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-10222.01.patch, HIVE-10222.patch Upgrade Calcite version to 1.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9736) StorageBasedAuthProvider should batch namenode-calls where possible.
[
https://issues.apache.org/jira/browse/HIVE-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486151#comment-14486151
]
Sushanth Sowmyan commented on HIVE-9736:
I've not looked at this patch in detail yet. But I'd also like to point to some
refactoring [~cnauroth] did recently in HIVE-10223 for you to look at to see
that we gel with some of Chris's work.
StorageBasedAuthProvider should batch namenode-calls where possible.
Key: HIVE-9736
URL: https://issues.apache.org/jira/browse/HIVE-9736
Project: Hive
Issue Type: Bug
Components: Metastore, Security
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9736.1.patch
Consider a table partitioned by 2 keys (dt, region). Say a dt partition could
have 1 associated regions. Consider that the user does:
{code:sql}
ALTER TABLE my_table DROP PARTITION (dt='20150101');
{code}
As things stand now, {{StorageBasedAuthProvider}} will make individual
{{DistributedFileSystem.listStatus()}} calls for each partition-directory,
and authorize each one separately. It'd be faster to batch the calls, and
examine multiple FileStatus objects at once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10268) Merge cbo branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-10268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10268: Attachment: HIVE-10268.patch Merge cbo branch into trunk --- Key: HIVE-10268 URL: https://issues.apache.org/jira/browse/HIVE-10268 Project: Hive Issue Type: Task Components: CBO Affects Versions: cbo-branch Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-10268.patch Merge patch generated on basis of diffs of trunk with cbo-branch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10122:
Attachment: HIVE-10122.01.patch
Fixing a small NPE. Looks like some outputs may change, let me get another run
to see which ones to update after the bug fix
Hive metastore filter-by-expression is broken for non-partition expressions
---
Key: HIVE-10122
URL: https://issues.apache.org/jira/browse/HIVE-10122
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10122.01.patch, HIVE-10122.patch
See
https://issues.apache.org/jira/browse/HIVE-10091?focusedCommentId=14382413page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14382413
These two lines of code
{noformat}
// Replace virtual columns with nulls. See javadoc for details.
prunerExpr = removeNonPartCols(prunerExpr, extractPartColNames(tab),
partColsUsedInFilter);
// Remove all parts that are not partition columns. See javadoc for
details.
ExprNodeDesc compactExpr = compactExpr(prunerExpr.clone());
{noformat}
are supposed to take care of this; I see there were bunch of changes to this
code over some time, and now it appears to be broken.
Thanks to [~thejas] for info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10269) HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486613#comment-14486613 ] Ferdinand Xu commented on HIVE-10269: - Hi [~qwertymaniac] [~vgumashta], can you help me review this jira? Thank you! HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor - Key: HIVE-10269 URL: https://issues.apache.org/jira/browse/HIVE-10269 Project: Hive Issue Type: Bug Components: Metastore Reporter: Gabor Liptak Assignee: Ferdinand Xu Attachments: HIVE-10269.patch Compiling trunk fails when building based on instructions in https://cwiki.apache.org/confluence/display/Hive/HowToContribute $ git status On branch trunk Your branch is up-to-date with 'origin/trunk'. nothing to commit, working directory clean $ mvn clean install -DskipTests -Phadoop-1 ...[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-metastore: Compilation failure: Compilation failure: [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6089,29] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6090,35] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hive-metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10269) HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu reassigned HIVE-10269: --- Assignee: Ferdinand Xu HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor - Key: HIVE-10269 URL: https://issues.apache.org/jira/browse/HIVE-10269 Project: Hive Issue Type: Bug Components: Metastore Reporter: Gabor Liptak Assignee: Ferdinand Xu Compiling trunk fails when building based on instructions in https://cwiki.apache.org/confluence/display/Hive/HowToContribute $ git status On branch trunk Your branch is up-to-date with 'origin/trunk'. nothing to commit, working directory clean $ mvn clean install -DskipTests -Phadoop-1 ...[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-metastore: Compilation failure: Compilation failure: [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6089,29] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6090,35] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hive-metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10016) Remove duplicated Hive table schema parsing in DataWritableReadSupport
[
https://issues.apache.org/jira/browse/HIVE-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486617#comment-14486617
]
Hive QA commented on HIVE-10016:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723827/HIVE-10016.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8665 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3336/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3336/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3336/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723827 - PreCommit-HIVE-TRUNK-Build
Remove duplicated Hive table schema parsing in DataWritableReadSupport
--
Key: HIVE-10016
URL: https://issues.apache.org/jira/browse/HIVE-10016
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Attachments: HIVE-10016-parquet.patch, HIVE-10016.1-parquet.patch,
HIVE-10016.patch
In {{DataWritableReadSupport.init()}}, the table schema is created and its
string format is set in conf. When construct the
{{ParquetRecordReaderWrapper}} , the schema is fetched from conf and parsed
several times.
We could remove these schema parsing, and improve the speed of
getRecordReader a bit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
