[jira] [Updated] (HIVE-10372) Update parquet version to 1.6.0
[ https://issues.apache.org/jira/browse/HIVE-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10372: Summary: Update parquet version to 1.6.0 (was: Bump parquet version to 1.6.0) Update parquet version to 1.6.0 --- Key: HIVE-10372 URL: https://issues.apache.org/jira/browse/HIVE-10372 Project: Hive Issue Type: Bug Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10372.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10350) CBO: Use total size instead of bucket count to determine number of splits parallelism
[
https://issues.apache.org/jira/browse/HIVE-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laljo John Pullokkaran resolved HIVE-10350.
---
Resolution: Fixed
CBO: Use total size instead of bucket count to determine number of splits
parallelism
Key: HIVE-10350
URL: https://issues.apache.org/jira/browse/HIVE-10350
Project: Hive
Issue Type: Sub-task
Components: CBO
Affects Versions: 1.2.0
Reporter: Mostafa Mokhtar
Assignee: Mostafa Mokhtar
Fix For: 1.2.0
Attachments: HIVE-10331.01.patch, HIVE-10350.2.patch
Not an overflow but parallelism ends up being -1 as it uses number of buckets
{code}
final int parallelism = RelMetadataQuery.splitCount(join) == null
? 1 : RelMetadataQuery.splitCount(join);
{code}
{code}
2015-04-13 18:19:09,154 DEBUG [main]: cost.HiveCostModel
(HiveCostModel.java:getJoinCost(62)) - COMMON_JOIN cost: {1600892.857142857
rows, 2.4463782008994658E7 cpu, 8.54445445875E10 io}
2015-04-13 18:19:09,155 DEBUG [main]: cost.HiveCostModel
(HiveCostModel.java:getJoinCost(62)) - MAP_JOIN cost: {1600892.857142857
rows, 1601785.714285714 cpu, -1698787.48 io}
2015-04-13 18:19:09,155 DEBUG [main]: cost.HiveCostModel
(HiveCostModel.java:getJoinCost(72)) - MAP_JOIN selected
2015-04-13 18:19:09,157 DEBUG [main]: parse.CalcitePlanner
(CalcitePlanner.java:apply(862)) - Plan After Join Reordering:
HiveSort(fetch=[100]): rowcount = 6006.726049749041, cumulative cost =
{1.1468867492063493E8 rows, 1.166177684126984E8 cpu, -1.1757664816220238E9
io}, id = 3000
HiveSort(sort0=[$0], dir0=[ASC]): rowcount = 6006.726049749041, cumulative
cost = {1.1468867492063493E8 rows, 1.166177684126984E8 cpu,
-1.1757664816220238E9 io}, id = 2998
HiveProject(customer_id=[$4], customername=[concat($9, ', ', $8)]):
rowcount = 6006.726049749041, cumulative cost = {1.1468867492063493E8 rows,
1.166177684126984E8 cpu, -1.1757664816220238E9 io}, id = 3136
HiveJoin(condition=[=($1, $5)], joinType=[inner],
joinAlgorithm=[map_join], cost=[{5.557820341269841E7 rows,
5.557840182539682E7 cpu, -4299694.122023809 io}]): rowcount =
6006.726049749041, cumulative cost = {1.1468867492063493E8 rows,
1.166177684126984E8 cpu, -1.1757664816220238E9 io}, id = 3132
HiveJoin(condition=[=($0, $1)], joinType=[inner],
joinAlgorithm=[map_join], cost=[{5.7498805E7 rows, 5.9419605E7 cpu,
-1.15248E9 io}]): rowcount = 5.5578005E7, cumulative cost = {5.7498805E7
rows, 5.9419605E7 cpu, -1.15248E9 io}, id = 3100
HiveProject(sr_cdemo_sk=[$4]): rowcount = 5.5578005E7, cumulative
cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 2992
HiveTableScan(table=[[tpcds_bin_orc_200.store_returns]]):
rowcount = 5.5578005E7, cumulative cost = {0}, id = 2878
HiveProject(cd_demo_sk=[$0]): rowcount = 1920800.0, cumulative cost
= {0.0 rows, 0.0 cpu, 0.0 io}, id = 2978
HiveTableScan(table=[[tpcds_bin_orc_200.customer_demographics]]):
rowcount = 1920800.0, cumulative cost = {0}, id = 2868
HiveJoin(condition=[=($10, $1)], joinType=[inner],
joinAlgorithm=[map_join], cost=[{1787.9365079365077 rows, 1790.15873015873
cpu, -8000.0 io}]): rowcount = 198.4126984126984, cumulative cost =
{1611666.507936508 rows, 1619761.5873015872 cpu, -1.89867875E7 io}, id = 3130
HiveJoin(condition=[=($0, $4)], joinType=[inner],
joinAlgorithm=[map_join], cost=[{8985.714285714286 rows, 16185.714285714286
cpu, -1.728E7 io}]): rowcount = 1785.7142857142856, cumulative cost =
{1609878.5714285714 rows, 1617971.4285714284 cpu, -1.89787875E7 io}, id = 3128
HiveProject(hd_demo_sk=[$0], hd_income_band_sk=[$1]): rowcount =
7200.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 2982
HiveTableScan(table=[[tpcds_bin_orc_200.household_demographics]]): rowcount =
7200.0, cumulative cost = {0}, id = 2871
HiveJoin(condition=[=($3, $6)], joinType=[inner],
joinAlgorithm=[map_join], cost=[{1600892.857142857 rows, 1601785.714285714
cpu, -1698787.48 io}]): rowcount = 1785.7142857142856, cumulative
cost = {1600892.857142857 rows, 1601785.714285714 cpu, -1698787.48
io}, id = 3105
HiveProject(c_customer_id=[$1], c_current_cdemo_sk=[$2],
c_current_hdemo_sk=[$3], c_current_addr_sk=[$4], c_first_name=[$8],
c_last_name=[$9]): rowcount = 160.0, cumulative cost = {0.0 rows, 0.0
cpu, 0.0 io}, id = 2970
HiveTableScan(table=[[tpcds_bin_orc_200.customer]]): rowcount
= 160.0, cumulative cost = {0}, id = 2862
HiveProject(ca_address_sk=[$0], ca_city=[$6]): rowcount =
892.8571428571428, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 2974
[jira] [Commented] (HIVE-9923) No clear message when from is missing
[
https://issues.apache.org/jira/browse/HIVE-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499203#comment-14499203
]
Hive QA commented on HIVE-9923:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725916/HIVE-9923.2.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8712 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3468/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3468/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3468/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725916 - PreCommit-HIVE-TRUNK-Build
No clear message when from is missing
---
Key: HIVE-9923
URL: https://issues.apache.org/jira/browse/HIVE-9923
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jeff Zhang
Assignee: Yongzhi Chen
Attachments: HIVE-9923.1.patch, HIVE-9923.2.patch
For the following sql, from is missing but it throw NPE which is not clear
for user.
{code}
hive insert overwrite directory '/tmp/hive-3' select sb1.name, sb2.age
student_bucketed sb1 join student_bucketed sb2 on sb1.name=sb2.name;
FAILED: NullPointerException null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics
[
https://issues.apache.org/jira/browse/HIVE-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499370#comment-14499370
]
Alan Gates commented on HIVE-10228:
---
Ok, so once the comments are added on why DROP TABLE FOR REPLICATION is
different than IF EXISTS and the new JIRAs referenced in review board I filed
I'm +1 on this patch.
Changes to Hive Export/Import/DropTable/DropPartition to support replication
semantics
--
Key: HIVE-10228
URL: https://issues.apache.org/jira/browse/HIVE-10228
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10228.2.patch, HIVE-10228.3.patch,
HIVE-10228.4.patch, HIVE-10228.patch
We need to update a couple of hive commands to support replication semantics.
To wit, we need the following:
EXPORT ... [FOR [METADATA] REPLICATION(“comment”)]
Export will now support an extra optional clause to tell it that this export
is being prepared for the purpose of replication. There is also an additional
optional clause here, that allows for the export to be a metadata-only
export, to handle cases of capturing the diff for alter statements, for
example.
Also, if done for replication, the non-presence of a table, or a table being
a view/offline table/non-native table is not considered an error, and
instead, will result in a successful no-op.
IMPORT ... (as normal) – but handles new semantics
No syntax changes for import, but import will have to change to be able to
handle all the permutations of export dumps possible. Also, import will have
to ensure that it should update the object only if the update being imported
is not older than the state of the object. Also, import currently does not
work with dbname.tablename kind of specification, this should be fixed to
work.
DROP TABLE ... FOR REPLICATION('eventid')
Drop Table now has an additional clause, to specify that this drop table is
being done for replication purposes, and that the dop should not actually
drop the table if the table is newer than that event id specified.
ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid')
Similarly, Drop Partition also has an equivalent change to Drop Table.
=
In addition, we introduce a new property repl.last.id, which when tagged on
to table properties or partition properties on a replication-destination,
holds the effective state identifier of the object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics
[
https://issues.apache.org/jira/browse/HIVE-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10228:
Attachment: HIVE-10228.4.patch
Attached updated patch.
Changes to Hive Export/Import/DropTable/DropPartition to support replication
semantics
--
Key: HIVE-10228
URL: https://issues.apache.org/jira/browse/HIVE-10228
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10228.2.patch, HIVE-10228.3.patch,
HIVE-10228.4.patch, HIVE-10228.patch
We need to update a couple of hive commands to support replication semantics.
To wit, we need the following:
EXPORT ... [FOR [METADATA] REPLICATION(“comment”)]
Export will now support an extra optional clause to tell it that this export
is being prepared for the purpose of replication. There is also an additional
optional clause here, that allows for the export to be a metadata-only
export, to handle cases of capturing the diff for alter statements, for
example.
Also, if done for replication, the non-presence of a table, or a table being
a view/offline table/non-native table is not considered an error, and
instead, will result in a successful no-op.
IMPORT ... (as normal) – but handles new semantics
No syntax changes for import, but import will have to change to be able to
handle all the permutations of export dumps possible. Also, import will have
to ensure that it should update the object only if the update being imported
is not older than the state of the object. Also, import currently does not
work with dbname.tablename kind of specification, this should be fixed to
work.
DROP TABLE ... FOR REPLICATION('eventid')
Drop Table now has an additional clause, to specify that this drop table is
being done for replication purposes, and that the dop should not actually
drop the table if the table is newer than that event id specified.
ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid')
Similarly, Drop Partition also has an equivalent change to Drop Table.
=
In addition, we introduce a new property repl.last.id, which when tagged on
to table properties or partition properties on a replication-destination,
holds the effective state identifier of the object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10377) CBO (Calcite Return Path): move qbid from RelOptHiveTable to HiveTableScan
[ https://issues.apache.org/jira/browse/HIVE-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-10377: --- Attachment: HIVE-10377.01.patch passed the cbo tests. [~ashutoshc], could you please take a look? CBO (Calcite Return Path): move qbid from RelOptHiveTable to HiveTableScan -- Key: HIVE-10377 URL: https://issues.apache.org/jira/browse/HIVE-10377 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Fix For: 1.2.0 Attachments: HIVE-10377.01.patch Similar to tableAlias, move qbid from RelOptHiveTable to HiveTableScan so that each table scan can have its own qbid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10364) The HMS upgrade script test does not publish results when prepare.sh fails.
[ https://issues.apache.org/jira/browse/HIVE-10364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499290#comment-14499290 ] Szehon Ho commented on HIVE-10364: -- +1 The HMS upgrade script test does not publish results when prepare.sh fails. --- Key: HIVE-10364 URL: https://issues.apache.org/jira/browse/HIVE-10364 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-10364.1.patch The HMS upgrade script must publish succeed or failure results to JIRA. This bug is not publishing any results on JIRA is the prepare.sh script fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10347) Merge spark to trunk 4/15/2015
[ https://issues.apache.org/jira/browse/HIVE-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10347: - Attachment: HIVE-10347.2.patch Merge spark to trunk 4/15/2015 -- Key: HIVE-10347 URL: https://issues.apache.org/jira/browse/HIVE-10347 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10347.2.patch, HIVE-10347.2.patch, HIVE-10347.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10374) Revert deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499351#comment-14499351 ] Alan Gates commented on HIVE-10374: --- +1 Revert deprecation message to HiveCLI - Key: HIVE-10374 URL: https://issues.apache.org/jira/browse/HIVE-10374 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10374.patch Reverting HIVE-10304 while there is a discussion about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10375) CBO (Calcite Return Path): disable the identity project remover for some union operators
[ https://issues.apache.org/jira/browse/HIVE-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-10375: -- Assignee: Pengcheng Xiong CBO (Calcite Return Path): disable the identity project remover for some union operators Key: HIVE-10375 URL: https://issues.apache.org/jira/browse/HIVE-10375 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Fix For: 1.2.0 The identity project remover will remove the Project operator that is required by some Union Operators. A temporary solution is to disable it for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10347) Merge spark to trunk 4/15/2015
[
https://issues.apache.org/jira/browse/HIVE-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499282#comment-14499282
]
Hive QA commented on HIVE-10347:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725931/HIVE-10347.2.patch
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 8745 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3469/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3469/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3469/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725931 - PreCommit-HIVE-TRUNK-Build
Merge spark to trunk 4/15/2015
--
Key: HIVE-10347
URL: https://issues.apache.org/jira/browse/HIVE-10347
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-10347.2.patch, HIVE-10347.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4625) HS2 should not attempt to get delegation token from metastore if using embedded metastore
[
https://issues.apache.org/jira/browse/HIVE-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499358#comment-14499358
]
Hive QA commented on HIVE-4625:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725948/HIVE-4625.2.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8711 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3470/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3470/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3470/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725948 - PreCommit-HIVE-TRUNK-Build
HS2 should not attempt to get delegation token from metastore if using
embedded metastore
-
Key: HIVE-4625
URL: https://issues.apache.org/jira/browse/HIVE-4625
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.11.0
Reporter: Thejas M Nair
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-4625.1.patch, HIVE-4625.2.patch
In kerberos secure mode, with doas enabled, Hive server2 tries to get
delegation token from metastore even if the metastore is being used in
embedded mode.
To avoid failure in that case, it uses catch block for
UnsupportedOperationException thrown that does nothing. But this leads to an
error being logged by lower levels and can mislead users into thinking that
there is a problem.
It should check if delegation token mode is supported with current
configuration before calling the function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10222) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499249#comment-14499249 ] Lefty Leverenz commented on HIVE-10222: --- Commit r1674185 gives the correct Summary text but the wrong jira number: HIVE-10122 : Upgrade Calcite dependency to newest version -- can that be fixed? Upgrade Calcite dependency to newest version Key: HIVE-10222 URL: https://issues.apache.org/jira/browse/HIVE-10222 Project: Hive Issue Type: Task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-10222.01.patch, HIVE-10222.02.patch, HIVE-10222.03.patch, HIVE-10222.04.patch, HIVE-10222.patch Upgrade Calcite version to 1.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10371) CBO (Calcite Return Path): Tag not set up correctly for Join operator in HiveOpConverter
[ https://issues.apache.org/jira/browse/HIVE-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499188#comment-14499188 ] Ashutosh Chauhan commented on HIVE-10371: - +1 CBO (Calcite Return Path): Tag not set up correctly for Join operator in HiveOpConverter Key: HIVE-10371 URL: https://issues.apache.org/jira/browse/HIVE-10371 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10371.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10306) We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
[ https://issues.apache.org/jira/browse/HIVE-10306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499207#comment-14499207 ] Thejas M Nair commented on HIVE-10306: -- +1 We need to print tez summary when hive.server2.logging.level = PERFORMANCE. - Key: HIVE-10306 URL: https://issues.apache.org/jira/browse/HIVE-10306 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-10306.1.patch, HIVE-10306.2.patch, HIVE-10306.3.patch, HIVE-10306.4.patch, HIVE-10306.5.patch We need to print tez summary when hive.server2.logging.level = PERFORMANCE. We introduced this parameter via HIVE-10119. The logging param for levels is only relevant to HS2, so for hive-cli users the hive.tez.exec.print.summary still makes sense. We can check for log-level param as well, in places we are checking value of hive.tez.exec.print.summary. Ie, consider hive.tez.exec.print.summary=true if log.level = PERFORMANCE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10367) org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows
[
https://issues.apache.org/jira/browse/HIVE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499450#comment-14499450
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-10367:
--
Failures are unrelated.
Thanks
Hari
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows
Key: HIVE-10367
URL: https://issues.apache.org/jira/browse/HIVE-10367
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-10367.1.patch
{code}
java.io.IOException: Failed to execute create table encrypted_table(a int, b
string) location
'D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable'.
Driver returned 1 Error: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.IllegalArgumentException: Pathname
/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable
from
hdfs://127.0.0.1:64735/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable
is not a valid DFS filename.)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.executeStatementOnDriver(TestHCatLoaderEncryption.java:163)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.createTableInSpecifiedPath(TestHCatLoaderEncryption.java:423)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:197)
{code}
It looks like the error is happening because of the below query:
create table encrypted_table(a int, b string) location
'D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429056939184/warehouse/encryptedTable'
This gets called from createTableInSpecifiedPath() in
TestHCatLoaderEncryption. I think an invalid HDFS path is passed here when
we are running the tests on Windows OS, so in line 197, changing
{code}
createTableInSpecifiedPath(ENCRYPTED_TABLE, a int, b string,
encryptedTablePath, driver);
{code}
to the below line might resolve the issue.
{code}
createTableInSpecifiedPath(ENCRYPTED_TABLE, a int, b string,
WindowsPathUtil.getHdfsUriString(encryptedTablePath), driver);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10284) enable container reuse for grace hash join
[
https://issues.apache.org/jira/browse/HIVE-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499536#comment-14499536
]
Hive QA commented on HIVE-10284:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725994/HIVE-10284.9.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8711 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3472/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3472/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3472/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725994 - PreCommit-HIVE-TRUNK-Build
enable container reuse for grace hash join
---
Key: HIVE-10284
URL: https://issues.apache.org/jira/browse/HIVE-10284
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Wei Zheng
Attachments: HIVE-10284.1.patch, HIVE-10284.2.patch,
HIVE-10284.3.patch, HIVE-10284.4.patch, HIVE-10284.5.patch,
HIVE-10284.6.patch, HIVE-10284.7.patch, HIVE-10284.8.patch, HIVE-10284.9.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4625) HS2 should not attempt to get delegation token from metastore if using embedded metastore
[ https://issues.apache.org/jira/browse/HIVE-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-4625: Attachment: HIVE-4625.3.patch Test failures with TestHadoop20SAuthBridge look like HIVE-10353 and HIVE-10354. Made sure TestHadoop20SAuthBridge works with patch 3 locally. cc-ing [~thejas] for reviewing patch 3. Thanks Hari HS2 should not attempt to get delegation token from metastore if using embedded metastore - Key: HIVE-4625 URL: https://issues.apache.org/jira/browse/HIVE-4625 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-4625.1.patch, HIVE-4625.2.patch, HIVE-4625.3.patch In kerberos secure mode, with doas enabled, Hive server2 tries to get delegation token from metastore even if the metastore is being used in embedded mode. To avoid failure in that case, it uses catch block for UnsupportedOperationException thrown that does nothing. But this leads to an error being logged by lower levels and can mislead users into thinking that there is a problem. It should check if delegation token mode is supported with current configuration before calling the function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10367) org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows
[
https://issues.apache.org/jira/browse/HIVE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499444#comment-14499444
]
Hive QA commented on HIVE-10367:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725954/HIVE-10367.1.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8711 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3471/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3471/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3471/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725954 - PreCommit-HIVE-TRUNK-Build
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows
Key: HIVE-10367
URL: https://issues.apache.org/jira/browse/HIVE-10367
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-10367.1.patch
{code}
java.io.IOException: Failed to execute create table encrypted_table(a int, b
string) location
'D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable'.
Driver returned 1 Error: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.IllegalArgumentException: Pathname
/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable
from
hdfs://127.0.0.1:64735/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable
is not a valid DFS filename.)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.executeStatementOnDriver(TestHCatLoaderEncryption.java:163)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.createTableInSpecifiedPath(TestHCatLoaderEncryption.java:423)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:197)
{code}
It looks like the error is happening because of the below query:
create table encrypted_table(a int, b string) location
'D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429056939184/warehouse/encryptedTable'
This gets
[jira] [Updated] (HIVE-10347) Merge spark to trunk 4/15/2015
[ https://issues.apache.org/jira/browse/HIVE-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10347: - Description: CLEAR LIBRARY CACHE Merge spark to trunk 4/15/2015 -- Key: HIVE-10347 URL: https://issues.apache.org/jira/browse/HIVE-10347 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10347.2.patch, HIVE-10347.2.patch, HIVE-10347.patch CLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[ https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-9824: --- Attachment: HIVE-9824.02.patch LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;) -- Key: HIVE-9824 URL: https://issues.apache.org/jira/browse/HIVE-9824 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch Today's VectorMapJoinOperator is a pass-through that converts each row from a vectorized row batch in a Java Object[] row and passes it to the MapJoinOperator superclass. This enhancement creates specialized vectorized map join operator classes that are optimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[ https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499497#comment-14499497 ] Matt McCline commented on HIVE-9824: Patch 02: Removed trailing whitespace, unneeded imports, changed ReduceRecordSource.processVectorGroup to copy key by value instead of by reference, removed accidentally left in debugDisplayRow calls that affect performance. LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;) -- Key: HIVE-9824 URL: https://issues.apache.org/jira/browse/HIVE-9824 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch Today's VectorMapJoinOperator is a pass-through that converts each row from a vectorized row batch in a Java Object[] row and passes it to the MapJoinOperator superclass. This enhancement creates specialized vectorized map join operator classes that are optimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9365) The Metastore should take port configuration from hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuben Kuhnert reassigned HIVE-9365: Assignee: Reuben Kuhnert The Metastore should take port configuration from hive-site.xml --- Key: HIVE-9365 URL: https://issues.apache.org/jira/browse/HIVE-9365 Project: Hive Issue Type: Improvement Reporter: Nicolas Thiébaud Assignee: Reuben Kuhnert Priority: Minor Labels: metastore Original Estimate: 3h Remaining Estimate: 3h As opposed to the cli. Having this configuration in the launcher script create fragmentation and does is not consistent with the way the hive stack is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[
https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499765#comment-14499765
]
Matt McCline commented on HIVE-9824:
Well, add to the mystery, when I run that query with vectorization on and on MR
(i.e. no native vector map join since we only do Tez), I get the following
exception!
{noformat}
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.ql.exec.OperatorFactory.getVectorOperator(OperatorFactory.java:159)
... 58 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.createVectorExpression(VectorizationContext.java:1037)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressionForUdf(VectorizationContext.java:995)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1162)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:440)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.init(VectorFilterOperator.java:54)
... 63 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.createVectorExpression(VectorizationContext.java:1037)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressionForUdf(VectorizationContext.java:995)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1162)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:440)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.createVectorExpression(VectorizationContext.java:1013)
... 67 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getInputColumnIndex(VectorizationContext.java:290)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.createVectorExpression(VectorizationContext.java:1017)
... 71 more
{noformat}
LLAP: Native Vectorization of Map Join so previously CPU bound queries shift
their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
--
Key: HIVE-9824
URL: https://issues.apache.org/jira/browse/HIVE-9824
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch
Today's VectorMapJoinOperator is a pass-through that converts each row from a
vectorized row batch in a Java Object[] row and passes it to the
MapJoinOperator superclass.
This enhancement creates specialized vectorized map join operator classes
that are optimized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9923) No clear message when from is missing
[
https://issues.apache.org/jira/browse/HIVE-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499781#comment-14499781
]
Yongzhi Chen commented on HIVE-9923:
[~szehon] or [~csun], could you review the code? Thanks
No clear message when from is missing
---
Key: HIVE-9923
URL: https://issues.apache.org/jira/browse/HIVE-9923
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jeff Zhang
Assignee: Yongzhi Chen
Attachments: HIVE-9923.1.patch, HIVE-9923.2.patch
For the following sql, from is missing but it throw NPE which is not clear
for user.
{code}
hive insert overwrite directory '/tmp/hive-3' select sb1.name, sb2.age
student_bucketed sb1 join student_bucketed sb2 on sb1.name=sb2.name;
FAILED: NullPointerException null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10306) We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
[
https://issues.apache.org/jira/browse/HIVE-10306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499744#comment-14499744
]
Hive QA commented on HIVE-10306:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726017/HIVE-10306.5.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8716 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3474/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3474/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3474/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726017 - PreCommit-HIVE-TRUNK-Build
We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
-
Key: HIVE-10306
URL: https://issues.apache.org/jira/browse/HIVE-10306
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-10306.1.patch, HIVE-10306.2.patch,
HIVE-10306.3.patch, HIVE-10306.4.patch, HIVE-10306.5.patch
We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
We introduced this parameter via HIVE-10119.
The logging param for levels is only relevant to HS2, so for hive-cli users
the hive.tez.exec.print.summary still makes sense. We can check for log-level
param as well, in places we are checking value of
hive.tez.exec.print.summary. Ie, consider hive.tez.exec.print.summary=true if
log.level = PERFORMANCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9923) No clear message when from is missing
[
https://issues.apache.org/jira/browse/HIVE-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499779#comment-14499779
]
Yongzhi Chen commented on HIVE-9923:
The 16 test failures are not related to the change.
1. 13 minimr failures appear in many other precommit builds.
2. org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view failed 23
times in precommit builds.
3. two failure:
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
does not call parser related code.
No clear message when from is missing
---
Key: HIVE-9923
URL: https://issues.apache.org/jira/browse/HIVE-9923
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jeff Zhang
Assignee: Yongzhi Chen
Attachments: HIVE-9923.1.patch, HIVE-9923.2.patch
For the following sql, from is missing but it throw NPE which is not clear
for user.
{code}
hive insert overwrite directory '/tmp/hive-3' select sb1.name, sb2.age
student_bucketed sb1 join student_bucketed sb2 on sb1.name=sb2.name;
FAILED: NullPointerException null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9923) No clear message when from is missing
[
https://issues.apache.org/jira/browse/HIVE-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499795#comment-14499795
]
Yongzhi Chen commented on HIVE-9923:
And the 3 non minimr failures appear in this build too:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3470/testReport/
No clear message when from is missing
---
Key: HIVE-9923
URL: https://issues.apache.org/jira/browse/HIVE-9923
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jeff Zhang
Assignee: Yongzhi Chen
Attachments: HIVE-9923.1.patch, HIVE-9923.2.patch
For the following sql, from is missing but it throw NPE which is not clear
for user.
{code}
hive insert overwrite directory '/tmp/hive-3' select sb1.name, sb2.age
student_bucketed sb1 join student_bucketed sb2 on sb1.name=sb2.name;
FAILED: NullPointerException null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[ https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499599#comment-14499599 ] Gopal V commented on HIVE-9824: --- A quick benchmark says that this makes simple map-joins ~5x faster - 280s - 54secs. LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;) -- Key: HIVE-9824 URL: https://issues.apache.org/jira/browse/HIVE-9824 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch Today's VectorMapJoinOperator is a pass-through that converts each row from a vectorized row batch in a Java Object[] row and passes it to the MapJoinOperator superclass. This enhancement creates specialized vectorized map join operator classes that are optimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10376) Move code to create jar for ivydownload.q to a separate id in maven ant-run-plugin in itests/pom.xml
[ https://issues.apache.org/jira/browse/HIVE-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Nag updated HIVE-10376: - Attachment: HIVE-10376.patch Move code to create jar for ivydownload.q to a separate id in maven ant-run-plugin in itests/pom.xml Key: HIVE-10376 URL: https://issues.apache.org/jira/browse/HIVE-10376 Project: Hive Issue Type: Improvement Reporter: Anant Nag Assignee: Anant Nag Attachments: HIVE-10376.patch Currently the code to create an example jar for ivyDownload.q is piggybanked on the download-spark ant-run-plugin id. This code should be moved to a separate execution id called something like create-ivytest-jar or more generally itests-setup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[
https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499712#comment-14499712
]
Gopal V commented on HIVE-9824:
---
[~mmccline]: Here's a simplified test-case
{code}
explain select
s_state, count(1)
from store_sales,
store,
date_dim
where store_sales.ss_sold_date_sk = date_dim.d_date_sk and
store_sales.ss_store_sk = store.s_store_sk and
store.s_state in ('KS','AL', 'MN', 'AL', 'SC', 'VT')
group by s_state
order by s_state
limit 100;
{code}
LLAP: Native Vectorization of Map Join so previously CPU bound queries shift
their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
--
Key: HIVE-9824
URL: https://issues.apache.org/jira/browse/HIVE-9824
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch
Today's VectorMapJoinOperator is a pass-through that converts each row from a
vectorized row batch in a Java Object[] row and passes it to the
MapJoinOperator superclass.
This enhancement creates specialized vectorized map join operator classes
that are optimized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499916#comment-14499916 ] Yongzhi Chen commented on HIVE-10312: - I got exact the same 16 failures when I did other precommit tests, so the test failures are not related to the patch. SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10312: Attachment: HIVE-10312.1.patch SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch, HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10312: Attachment: (was: HIVE-10312.1.patch) SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10312: Assignee: Aihua Xu SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Assignee: Aihua Xu Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10312: Assignee: (was: Aihua Xu) SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10312: Assignee: (was: Aihua Xu) SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10217) LLAP: Support caching of uncompressed ORC data
[
https://issues.apache.org/jira/browse/HIVE-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499598#comment-14499598
]
Gopal V commented on HIVE-10217:
I guess we can use an arbitrary size to read this data in, since the layout
after decompression will match reading this in 256kb chunks (or whatever's
ideal for the allocator).
LLAP: Support caching of uncompressed ORC data
--
Key: HIVE-10217
URL: https://issues.apache.org/jira/browse/HIVE-10217
Project: Hive
Issue Type: Sub-task
Affects Versions: llap
Reporter: Gopal V
Assignee: Sergey Shelukhin
Fix For: llap
{code}
Caused by: java.io.IOException: ORC compression buffer size (0) is smaller
than LLAP low-level cache minimum allocation size (131072). Decrease the
value for hive.llap.io.cache.orc.alloc.min
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:137)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:48)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[
https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499637#comment-14499637
]
Hive QA commented on HIVE-9824:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726114/HIVE-9824.02.patch
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 8730 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_aggregate_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_char_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_context
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_mapjoin
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3473/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3473/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3473/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726114 - PreCommit-HIVE-TRUNK-Build
LLAP: Native Vectorization of Map Join so previously CPU bound queries shift
their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
--
Key: HIVE-9824
URL: https://issues.apache.org/jira/browse/HIVE-9824
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch
Today's VectorMapJoinOperator is a pass-through that converts each row from a
vectorized row batch in a Java Object[] row and passes it to the
MapJoinOperator superclass.
This enhancement creates specialized vectorized map join operator classes
that are optimized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[ https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499707#comment-14499707 ] Matt McCline commented on HIVE-9824: [~gopalv] thanks for running quick performance tests. As for Duplicate column 3 in ordered column map the code assumes the big table retain output column mapping is unique. Maybe this is a wrong assumption. LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;) -- Key: HIVE-9824 URL: https://issues.apache.org/jira/browse/HIVE-9824 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch Today's VectorMapJoinOperator is a pass-through that converts each row from a vectorized row batch in a Java Object[] row and passes it to the MapJoinOperator superclass. This enhancement creates specialized vectorized map join operator classes that are optimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-633) ADD FILE command does not accept quoted filenames
[ https://issues.apache.org/jira/browse/HIVE-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499883#comment-14499883 ] Hari Sekhon commented on HIVE-633: -- This has been bugging me for such a long time with add jar 'blah.jar', add file blah.py, source file.sql etc. Seems like it would be a minor improvement to have the parser to do a string.replaceAll() or similar method to replace single and double quotes in these file tokens? ADD FILE command does not accept quoted filenames - Key: HIVE-633 URL: https://issues.apache.org/jira/browse/HIVE-633 Project: Hive Issue Type: Bug Affects Versions: 0.3.0 Environment: Ubuntu Linux (intrepid) Reporter: Saurabh Nanda Priority: Minor The following command says file does not exist. Removing the quotes around the filename makes it work. hive add files '/tmp/testing.jar'; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9791) insert into table throws NPE
[
https://issues.apache.org/jira/browse/HIVE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen reassigned HIVE-9791:
--
Assignee: Yongzhi Chen
insert into table throws NPE
Key: HIVE-9791
URL: https://issues.apache.org/jira/browse/HIVE-9791
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Alexander Pivovarov
Assignee: Yongzhi Chen
to reproduce NPE run the following
{code}
create table a as select 'A' letter;
OK
insert into table a select 'B' letter;
FAILED: NullPointerException null
-- works fine if add from table to select statement
insert into table a select 'B' letter from dual;
OK
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10307) Support to use number literals in partition column
[
https://issues.apache.org/jira/browse/HIVE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500010#comment-14500010
]
Hive QA commented on HIVE-10307:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726020/HIVE-10307.2.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8697 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-ptf_general_queries.q-avro_joins.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3475/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3475/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3475/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726020 - PreCommit-HIVE-TRUNK-Build
Support to use number literals in partition column
--
Key: HIVE-10307
URL: https://issues.apache.org/jira/browse/HIVE-10307
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 1.0.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Attachments: HIVE-10307.1.patch, HIVE-10307.2.patch, HIVE-10307.patch
Data types like TinyInt, SmallInt, BigInt or Decimal can be expressed as
literals with postfix like Y, S, L, or BD appended to the number. These
literals work in most Hive queries, but do not when they are used as
partition column value. For a partitioned table like:
create table partcoltypenum (key int, value string) partitioned by (tint
tinyint, sint smallint, bint bigint);
insert into partcoltypenum partition (tint=100Y, sint=1S,
bint=1000L) select key, value from src limit 30;
Queries like select, describe and drop partition do not work. For an example
select * from partcoltypenum where tint=100Y and sint=1S and
bint=1000L;
does not return any rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[ https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500379#comment-14500379 ] Sergey Shelukhin commented on HIVE-9824: nm, I see it https://reviews.apache.org/r/33281/ LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;) -- Key: HIVE-9824 URL: https://issues.apache.org/jira/browse/HIVE-9824 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch Today's VectorMapJoinOperator is a pass-through that converts each row from a vectorized row batch in a Java Object[] row and passes it to the MapJoinOperator superclass. This enhancement creates specialized vectorized map join operator classes that are optimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10233) Hive on LLAP: Memory manager
[ https://issues.apache.org/jira/browse/HIVE-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-10233: -- Attachment: (was: HIVE-10233-WIP.patch) Hive on LLAP: Memory manager Key: HIVE-10233 URL: https://issues.apache.org/jira/browse/HIVE-10233 Project: Hive Issue Type: Bug Components: Tez Affects Versions: llap Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-10233-WIP-2.patch We need a memory manager in llap/tez to manage the usage of memory across threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10233) Hive on LLAP: Memory manager
[ https://issues.apache.org/jira/browse/HIVE-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-10233: -- Attachment: HIVE-10233-WIP.3.patch Address Sid's comments. Hive on LLAP: Memory manager Key: HIVE-10233 URL: https://issues.apache.org/jira/browse/HIVE-10233 Project: Hive Issue Type: Bug Components: Tez Affects Versions: llap Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-10233-WIP-2.patch We need a memory manager in llap/tez to manage the usage of memory across threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10233) Hive on LLAP: Memory manager
[ https://issues.apache.org/jira/browse/HIVE-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-10233: -- Attachment: (was: HIVE-10233-WIP.3.patch) Hive on LLAP: Memory manager Key: HIVE-10233 URL: https://issues.apache.org/jira/browse/HIVE-10233 Project: Hive Issue Type: Bug Components: Tez Affects Versions: llap Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-10233-WIP-2.patch We need a memory manager in llap/tez to manage the usage of memory across threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10202) Beeline outputs prompt+query on standard output when used in non-interactive mode
[
https://issues.apache.org/jira/browse/HIVE-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500315#comment-14500315
]
Sergio Peña commented on HIVE-10202:
It's good to mention this on that section [~leftylev].
Maybe something like this?:
Reduce the amount of informational messages displayed (true) or not (false). It
also stops displaying the log messages for the query from HiveServer2 (Hive
0.14 and later) and the HQL commands.
Beeline outputs prompt+query on standard output when used in non-interactive
mode
-
Key: HIVE-10202
URL: https://issues.apache.org/jira/browse/HIVE-10202
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Naveen Gangam
Fix For: 1.2.0
Attachments: HIVE-10202.patch
When passing a SQL script file to Hive CLI, the prompt+query is not sent to
the standard output nor standard error. This is totally fine because users
might want to send only the query results to the standard output, and parse
the results from it.
In the case of BeeLine, the promp+query is sent to the standard output
causing extra parsing on the user scripts to avoid reading the prompt+query.
Another drawback is in the security side. Sensitive queries are logged
directly to the files where the standard output is redirected.
How to reproduce:
{noformat}
$ cat /tmp/query.sql
select *
from test
limit 1;
$ beeline --showheader=false --outputformat=tsv2 -u
jdbc:hive2://localhost:1 -f /tmp/query.sql /tmp/output.log 2
/tmp/error.log
$ cat /tmp/output.log
0: jdbc:hive2://localhost:1 select *
. . . . . . . . . . . . . . . . from test
. . . . . . . . . . . . . . . . limit 1;
451 451.713 false y2dh7 [866,528,936]
0: jdbc:hive2://localhost:1
{noformat}
We should avoid sending the prompt+query to the standard output/error
whenever a script file is passed to BeeLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4625) HS2 should not attempt to get delegation token from metastore if using embedded metastore
[ https://issues.apache.org/jira/browse/HIVE-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-4625: Attachment: (was: HIVE-4625.3.patch) HS2 should not attempt to get delegation token from metastore if using embedded metastore - Key: HIVE-4625 URL: https://issues.apache.org/jira/browse/HIVE-4625 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-4625.1.patch, HIVE-4625.2.patch In kerberos secure mode, with doas enabled, Hive server2 tries to get delegation token from metastore even if the metastore is being used in embedded mode. To avoid failure in that case, it uses catch block for UnsupportedOperationException thrown that does nothing. But this leads to an error being logged by lower levels and can mislead users into thinking that there is a problem. It should check if delegation token mode is supported with current configuration before calling the function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10202) Beeline outputs prompt+query on standard output when used in non-interactive mode
[
https://issues.apache.org/jira/browse/HIVE-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500344#comment-14500344
]
Naveen Gangam commented on HIVE-10202:
--
[~spena] I dont think thats desirable. My take is that --silent should supress
additional info but not the prompt in interactive mode. I will take another
look. Perhaps a modification to the fix for HIVE-10087. We atleast have
workaround to not use --silent for now. Thanks for pointing this out.
Beeline outputs prompt+query on standard output when used in non-interactive
mode
-
Key: HIVE-10202
URL: https://issues.apache.org/jira/browse/HIVE-10202
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Naveen Gangam
Fix For: 1.2.0
Attachments: HIVE-10202.patch
When passing a SQL script file to Hive CLI, the prompt+query is not sent to
the standard output nor standard error. This is totally fine because users
might want to send only the query results to the standard output, and parse
the results from it.
In the case of BeeLine, the promp+query is sent to the standard output
causing extra parsing on the user scripts to avoid reading the prompt+query.
Another drawback is in the security side. Sensitive queries are logged
directly to the files where the standard output is redirected.
How to reproduce:
{noformat}
$ cat /tmp/query.sql
select *
from test
limit 1;
$ beeline --showheader=false --outputformat=tsv2 -u
jdbc:hive2://localhost:1 -f /tmp/query.sql /tmp/output.log 2
/tmp/error.log
$ cat /tmp/output.log
0: jdbc:hive2://localhost:1 select *
. . . . . . . . . . . . . . . . from test
. . . . . . . . . . . . . . . . limit 1;
451 451.713 false y2dh7 [866,528,936]
0: jdbc:hive2://localhost:1
{noformat}
We should avoid sending the prompt+query to the standard output/error
whenever a script file is passed to BeeLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4625) HS2 should not attempt to get delegation token from metastore if using embedded metastore
[ https://issues.apache.org/jira/browse/HIVE-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-4625: Attachment: HIVE-4625.3.patch HS2 should not attempt to get delegation token from metastore if using embedded metastore - Key: HIVE-4625 URL: https://issues.apache.org/jira/browse/HIVE-4625 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-4625.1.patch, HIVE-4625.2.patch, HIVE-4625.3.patch In kerberos secure mode, with doas enabled, Hive server2 tries to get delegation token from metastore even if the metastore is being used in embedded mode. To avoid failure in that case, it uses catch block for UnsupportedOperationException thrown that does nothing. But this leads to an error being logged by lower levels and can mislead users into thinking that there is a problem. It should check if delegation token mode is supported with current configuration before calling the function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10306) We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
[ https://issues.apache.org/jira/browse/HIVE-10306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500350#comment-14500350 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-10306: -- The test failures look unrelated to the fix. Thanks Hari We need to print tez summary when hive.server2.logging.level = PERFORMANCE. - Key: HIVE-10306 URL: https://issues.apache.org/jira/browse/HIVE-10306 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-10306.1.patch, HIVE-10306.2.patch, HIVE-10306.3.patch, HIVE-10306.4.patch, HIVE-10306.5.patch We need to print tez summary when hive.server2.logging.level = PERFORMANCE. We introduced this parameter via HIVE-10119. The logging param for levels is only relevant to HS2, so for hive-cli users the hive.tez.exec.print.summary still makes sense. We can check for log-level param as well, in places we are checking value of hive.tez.exec.print.summary. Ie, consider hive.tez.exec.print.summary=true if log.level = PERFORMANCE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10374) Revert HIVE-10304 deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10374: - Summary: Revert HIVE-10304 deprecation message to HiveCLI (was: Revert deprecation message to HiveCLI) Revert HIVE-10304 deprecation message to HiveCLI Key: HIVE-10374 URL: https://issues.apache.org/jira/browse/HIVE-10374 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Fix For: 1.2.0 Attachments: HIVE-10374.patch Reverting HIVE-10304 while there is a discussion about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10374) Revert deprecation message to HiveCLI
[
https://issues.apache.org/jira/browse/HIVE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500299#comment-14500299
]
Hive QA commented on HIVE-10374:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726043/HIVE-10374.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8711 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3477/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3477/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3477/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726043 - PreCommit-HIVE-TRUNK-Build
Revert deprecation message to HiveCLI
-
Key: HIVE-10374
URL: https://issues.apache.org/jira/browse/HIVE-10374
Project: Hive
Issue Type: Bug
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-10374.patch
Reverting HIVE-10304 while there is a discussion about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10380: --- Attachment: HIVE-10380.cbo.patch [~pxiong], [~ashutoshc], [~jpullokkaran], this patch will let us continue debugging other issues. CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10233) Hive on LLAP: Memory manager
[ https://issues.apache.org/jira/browse/HIVE-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-10233: -- Attachment: HIVE-10233-WIP-3.patch Hive on LLAP: Memory manager Key: HIVE-10233 URL: https://issues.apache.org/jira/browse/HIVE-10233 Project: Hive Issue Type: Bug Components: Tez Affects Versions: llap Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-10233-WIP-2.patch, HIVE-10233-WIP-3.patch We need a memory manager in llap/tez to manage the usage of memory across threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10373) CBO (Calcite Return Path): Complete translation for all kinds of RexNode in Windowing
[ https://issues.apache.org/jira/browse/HIVE-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-10373. Resolution: Fixed CBO (Calcite Return Path): Complete translation for all kinds of RexNode in Windowing - Key: HIVE-10373 URL: https://issues.apache.org/jira/browse/HIVE-10373 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10373.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10331) ORC : Is null SARG filters out all row groups written in old ORC format
[
https://issues.apache.org/jira/browse/HIVE-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500142#comment-14500142
]
Hive QA commented on HIVE-10331:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726042/HIVE-10331.05.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8711 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3476/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3476/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3476/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726042 - PreCommit-HIVE-TRUNK-Build
ORC : Is null SARG filters out all row groups written in old ORC format
---
Key: HIVE-10331
URL: https://issues.apache.org/jira/browse/HIVE-10331
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.1.0
Reporter: Mostafa Mokhtar
Assignee: Mostafa Mokhtar
Fix For: 1.2.0
Attachments: HIVE-10331.01.patch, HIVE-10331.02.patch,
HIVE-10331.03.patch, HIVE-10331.03.patch, HIVE-10331.04.patch,
HIVE-10331.05.patch
Queries are returning wrong results as all row groups gets filtered out and
no rows get scanned.
{code}
SELECT
count(*)
FROM
store_sales
WHERE
ss_addr_sk IS NULL
{code}
With hive.optimize.index.filter disabled we get the correct results
In pickRowGroups stats show that hasNull_ is fales, while the rowgroup
actually has null.
Same query runs fine for newly loaded ORC tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10222) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500219#comment-14500219 ] Ashutosh Chauhan commented on HIVE-10222: - Thanks, Lefty for pointing that out. Updated svn commit message. Upgrade Calcite dependency to newest version Key: HIVE-10222 URL: https://issues.apache.org/jira/browse/HIVE-10222 Project: Hive Issue Type: Task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-10222.01.patch, HIVE-10222.02.patch, HIVE-10222.03.patch, HIVE-10222.04.patch, HIVE-10222.patch Upgrade Calcite version to 1.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10377) CBO (Calcite Return Path): move qbid from RelOptHiveTable to HiveTableScan
[ https://issues.apache.org/jira/browse/HIVE-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-10377. - Resolution: Fixed Fix Version/s: (was: 1.2.0) cbo-branch Committed to branch. Thanks, Pengcheng! CBO (Calcite Return Path): move qbid from RelOptHiveTable to HiveTableScan -- Key: HIVE-10377 URL: https://issues.apache.org/jira/browse/HIVE-10377 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Fix For: cbo-branch Attachments: HIVE-10377.01.patch Similar to tableAlias, move qbid from RelOptHiveTable to HiveTableScan so that each table scan can have its own qbid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-10304) Add deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reopened HIVE-10304: -- Reopening as the change was reverted. Add deprecation message to HiveCLI -- Key: HIVE-10304 URL: https://issues.apache.org/jira/browse/HIVE-10304 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 1.1.0 Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC1.2 Fix For: 1.2.0 Attachments: HIVE-10304.2.patch, HIVE-10304.3.patch, HIVE-10304.patch As Beeline is now the recommended command line tool to Hive, we should add a message to HiveCLI to indicate that it is deprecated and redirect them to Beeline. This is not suggesting to remove HiveCLI for now, but just a helpful direction for user to know the direction to focus attention in Beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500426#comment-14500426 ] Laljo John Pullokkaran commented on HIVE-10380: --- [~jcamachorodriguez] Could you refine the patch to use: Fully qualified name (db.table name from relopthivetable) + Table Alias + QBID CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6090) Audit logs for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500179#comment-14500179 ] Thiruvel Thirumoolan commented on HIVE-6090: Thanks Thejas, will rebase and upload. Audit logs for HiveServer2 -- Key: HIVE-6090 URL: https://issues.apache.org/jira/browse/HIVE-6090 Project: Hive Issue Type: Improvement Components: Diagnosability, HiveServer2 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: audit, hiveserver Fix For: 1.2.0 Attachments: HIVE-6090.1.WIP.patch, HIVE-6090.1.patch, HIVE-6090.patch HiveMetastore has audit logs and would like to audit all queries or requests to HiveServer2 also. This will help in understanding how the APIs were used, queries submitted, users etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500079#comment-14500079 ] Mubashir Kazia commented on HIVE-10312: --- [~ychena] or [~aihuaxu] So if that is the case can you please review the patch and let me know if any changes are needed? SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10313) Literal Decimal ExprNodeConstantDesc should contain value of HiveDecimal instead of String
[
https://issues.apache.org/jira/browse/HIVE-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500057#comment-14500057
]
Xuefu Zhang commented on HIVE-10313:
Thanks for fixing this. Just curious, what problems had this bug caused? I'm
wondering why it didn't surface until now.
Literal Decimal ExprNodeConstantDesc should contain value of HiveDecimal
instead of String
--
Key: HIVE-10313
URL: https://issues.apache.org/jira/browse/HIVE-10313
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.0.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-10313.1.patch, HIVE-10313.patch
In TyepCheckProcFactory.NumExprProcessor, the ExprNodeConstantDesc is created
from strVal:
{code}
else if (expr.getText().endsWith(BD)) {
// Literal decimal
String strVal = expr.getText().substring(0, expr.getText().length()
- 2);
HiveDecimal hd = HiveDecimal.create(strVal);
int prec = 1;
int scale = 0;
if (hd != null) {
prec = hd.precision();
scale = hd.scale();
}
DecimalTypeInfo typeInfo = TypeInfoFactory.getDecimalTypeInfo(prec,
scale);
return new ExprNodeConstantDesc(typeInfo, strVal);
}
{code}
It should use HiveDecmal:
return new ExprNodeConstantDesc(typeInfo, hd);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10313) Literal Decimal ExprNodeConstantDesc should contain value of HiveDecimal instead of String
[
https://issues.apache.org/jira/browse/HIVE-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500118#comment-14500118
]
Chaoyu Tang commented on HIVE-10313:
[~xuefuz] I ran into this problem in coding when I tried to do a conversion
between decimals. For example, if I convert the value of ExprNodeConstantDesc
parsed from a decimal .22BD (typeinfo decimal(6,2)) to a decimal (10,4). I
saw the ClassCastException error:
{code}
ERROR ql.Driver: FAILED: ClassCastException java.lang.String cannot be cast to
org.apache.hadoop.hive.common.type.HiveDecimal
java.lang.ClassCastException: java.lang.String cannot be cast to
org.apache.hadoop.hive.common.type.HiveDecimal
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getHiveDecimal(PrimitiveObjectInspectorUtils.java:1001)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$HiveDecimalConverter.convert(PrimitiveObjectInspectorConverter.java:349I
...
{code}
Tracing down to TypeCheckProcFactory.NumExprProcessor, I saw the
ExprNodeConstantDesc was created with typeInfo (of decimal) but value is
actually of type String. But as I understand the API
ExprNodeConstantDesc(TypeInfo typeInfo, Object value), the typeInfo should
reflect and interpret the type of value object.
Literal Decimal ExprNodeConstantDesc should contain value of HiveDecimal
instead of String
--
Key: HIVE-10313
URL: https://issues.apache.org/jira/browse/HIVE-10313
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.0.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-10313.1.patch, HIVE-10313.patch
In TyepCheckProcFactory.NumExprProcessor, the ExprNodeConstantDesc is created
from strVal:
{code}
else if (expr.getText().endsWith(BD)) {
// Literal decimal
String strVal = expr.getText().substring(0, expr.getText().length()
- 2);
HiveDecimal hd = HiveDecimal.create(strVal);
int prec = 1;
int scale = 0;
if (hd != null) {
prec = hd.precision();
scale = hd.scale();
}
DecimalTypeInfo typeInfo = TypeInfoFactory.getDecimalTypeInfo(prec,
scale);
return new ExprNodeConstantDesc(typeInfo, strVal);
}
{code}
It should use HiveDecmal:
return new ExprNodeConstantDesc(typeInfo, hd);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10354) Investigate the test failure of TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
[ https://issues.apache.org/jira/browse/HIVE-10354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500093#comment-14500093 ] Thejas M Nair commented on HIVE-10354: -- This test seems to pass - http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3475/testReport/org.apache.hadoop.hive.thrift/TestHadoop20SAuthBridge/ Investigate the test failure of TestHadoop20SAuthBridge.testSaslWithHiveMetaStore - Key: HIVE-10354 URL: https://issues.apache.org/jira/browse/HIVE-10354 Project: Hive Issue Type: Bug Reporter: Aihua Xu It failed with: java.lang.NullPointerException: null at org.apache.hadoop.hive.metastore.HiveMetaStore.getDelegationToken(HiveMetaStore.java:5752) at org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.getDelegationTokenStr(TestHadoop20SAuthBridge.java:318) at org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.obtainTokenAndAddIntoUGI(TestHadoop20SAuthBridge.java:339) at org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore(TestHadoop20SAuthBridge.java:231) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10264) Document Replication support on wiki
[ https://issues.apache.org/jira/browse/HIVE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10264: Affects Version/s: 1.2.0 Document Replication support on wiki Key: HIVE-10264 URL: https://issues.apache.org/jira/browse/HIVE-10264 Project: Hive Issue Type: Sub-task Components: Import/Export Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Labels: TODOC1.2 Attachments: EXIMReplicationReplayProtocol.pdf, apache_hivedr.0.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10264) Document Replication support on wiki
[ https://issues.apache.org/jira/browse/HIVE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10264: Labels: TODOC1.2 (was: ) Document Replication support on wiki Key: HIVE-10264 URL: https://issues.apache.org/jira/browse/HIVE-10264 Project: Hive Issue Type: Sub-task Components: Import/Export Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Labels: TODOC1.2 Attachments: EXIMReplicationReplayProtocol.pdf, apache_hivedr.0.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7973) Hive Replication Support
[ https://issues.apache.org/jira/browse/HIVE-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-7973: --- Description: A need for replication is a common one in many database management systems, and it's important for hive to evolve support for such a tool as part of its ecosystem. Hive already supports an EXPORT and IMPORT command, which can be used to dump out tables, distcp them to another cluster, and and import/create from that. If we had a mechanism by which exports and imports could be automated, it establishes the base with which replication can be developed. One place where this kind of automation can be developed is with aid of the HiveMetaStoreEventHandler mechanisms, to generate notifications when certain changes are committed to the metastore, and then translate those notifications to export actions, distcp actions and import actions on another import action. Part of that already exists is with the Notification system that is part of hcatalog-server-extensions. Initially, this was developed to be able to trigger a JMS notification, which an Oozie workflow can use to can start off actions keyed on the finishing of a job that used HCatalog to write to a table. While this currently lives under hcatalog, the primary reason for its existence has a scope well past hcatalog alone, and can be used as-is without the use of HCatalog IF/OF. This can be extended, with the help of a library which does that aforementioned translation. I also think that these sections should live in a core hive module, rather than being tucked away inside hcatalog. Once we have rudimentary support for table partition replication, we can then move on to further requirements of replication, such as metadata replications (such as replication of changes to roles/etc), and/or optimize away the requirement to distcp and use webhdfs instead, etc. This Story tracks all the bits that go into development of such a system - I'll create multiple smaller tasks inside this as we go on. Please also see HIVE-10264 for documentation-related links for this, and https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment for associated wiki (currently in progress) was: A need for replication is a common one in many database management systems, and it's important for hive to evolve support for such a tool as part of its ecosystem. Hive already supports an EXPORT and IMPORT command, which can be used to dump out tables, distcp them to another cluster, and and import/create from that. If we had a mechanism by which exports and imports could be automated, it establishes the base with which replication can be developed. One place where this kind of automation can be developed is with aid of the HiveMetaStoreEventHandler mechanisms, to generate notifications when certain changes are committed to the metastore, and then translate those notifications to export actions, distcp actions and import actions on another import action. Part of that already exists is with the Notification system that is part of hcatalog-server-extensions. Initially, this was developed to be able to trigger a JMS notification, which an Oozie workflow can use to can start off actions keyed on the finishing of a job that used HCatalog to write to a table. While this currently lives under hcatalog, the primary reason for its existence has a scope well past hcatalog alone, and can be used as-is without the use of HCatalog IF/OF. This can be extended, with the help of a library which does that aforementioned translation. I also think that these sections should live in a core hive module, rather than being tucked away inside hcatalog. Once we have rudimentary support for table partition replication, we can then move on to further requirements of replication, such as metadata replications (such as replication of changes to roles/etc), and/or optimize away the requirement to distcp and use webhdfs instead, etc. This Story tracks all the bits that go into development of such a system - I'll create multiple smaller tasks inside this as we go on. Hive Replication Support Key: HIVE-7973 URL: https://issues.apache.org/jira/browse/HIVE-7973 Project: Hive Issue Type: Bug Components: Import/Export Reporter: Sushanth Sowmyan A need for replication is a common one in many database management systems, and it's important for hive to evolve support for such a tool as part of its ecosystem. Hive already supports an EXPORT and IMPORT command, which can be used to dump out tables, distcp them to another cluster, and and import/create from that. If we had a mechanism by which exports and imports could be automated, it establishes the base with which replication can be developed. One place where this kind of
[jira] [Updated] (HIVE-10383) Add tests for partition level statistics + refactor stats tests of TestHBaseStore [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10383: Issue Type: Sub-task (was: Bug) Parent: HIVE-9452 Add tests for partition level statistics + refactor stats tests of TestHBaseStore [hbase-metastore branch] -- Key: HIVE-10383 URL: https://issues.apache.org/jira/browse/HIVE-10383 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9824) LLAP: Native Vectorization of Map Join so previously CPU bound queries shift their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
[
https://issues.apache.org/jira/browse/HIVE-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500535#comment-14500535
]
Mostafa Mokhtar commented on HIVE-9824:
---
[~mmccline]
These failures are un-related
{code}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{code}
LLAP: Native Vectorization of Map Join so previously CPU bound queries shift
their bottleneck to I/O and make it possible for the rest of LLAP to shine ;)
--
Key: HIVE-9824
URL: https://issues.apache.org/jira/browse/HIVE-9824
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-9824.01.patch, HIVE-9824.02.patch
Today's VectorMapJoinOperator is a pass-through that converts each row from a
vectorized row batch in a Java Object[] row and passes it to the
MapJoinOperator superclass.
This enhancement creates specialized vectorized map join operator classes
that are optimized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics
[
https://issues.apache.org/jira/browse/HIVE-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10228:
Attachment: HIVE-10228.5.patch
One more minor update to patch from 10228.4.patch - Clarified some comments.
Changes to Hive Export/Import/DropTable/DropPartition to support replication
semantics
--
Key: HIVE-10228
URL: https://issues.apache.org/jira/browse/HIVE-10228
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10228.2.patch, HIVE-10228.3.patch,
HIVE-10228.4.patch, HIVE-10228.5.patch, HIVE-10228.patch
We need to update a couple of hive commands to support replication semantics.
To wit, we need the following:
EXPORT ... [FOR [METADATA] REPLICATION(“comment”)]
Export will now support an extra optional clause to tell it that this export
is being prepared for the purpose of replication. There is also an additional
optional clause here, that allows for the export to be a metadata-only
export, to handle cases of capturing the diff for alter statements, for
example.
Also, if done for replication, the non-presence of a table, or a table being
a view/offline table/non-native table is not considered an error, and
instead, will result in a successful no-op.
IMPORT ... (as normal) – but handles new semantics
No syntax changes for import, but import will have to change to be able to
handle all the permutations of export dumps possible. Also, import will have
to ensure that it should update the object only if the update being imported
is not older than the state of the object. Also, import currently does not
work with dbname.tablename kind of specification, this should be fixed to
work.
DROP TABLE ... FOR REPLICATION('eventid')
Drop Table now has an additional clause, to specify that this drop table is
being done for replication purposes, and that the dop should not actually
drop the table if the table is newer than that event id specified.
ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid')
Similarly, Drop Partition also has an equivalent change to Drop Table.
=
In addition, we introduce a new property repl.last.id, which when tagged on
to table properties or partition properties on a replication-destination,
holds the effective state identifier of the object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[
https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500434#comment-14500434
]
Jesus Camacho Rodriguez commented on HIVE-10380:
[~jpullokkaran], QBID already contains the table alias. It is generated by the
following method:
{noformat}
protected String getAliasId(String alias, QB qb) {
return (qb.getId() == null ? alias : qb.getId() + : +
alias).toLowerCase();
}
{noformat}
CBO (Calcite Return Path): Add QBId to TableScan digest
---
Key: HIVE-10380
URL: https://issues.apache.org/jira/browse/HIVE-10380
Project: Hive
Issue Type: Sub-task
Components: CBO
Affects Versions: cbo-branch
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: cbo-branch
Attachments: HIVE-10380.cbo.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics
[
https://issues.apache.org/jira/browse/HIVE-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500436#comment-14500436
]
Sushanth Sowmyan commented on HIVE-10228:
-
Thanks Alan, I have created HIVE-10381 for that other issue, and added comments
in code to expand on what DROP TABLE FOR REPLICATION is doing.
Changes to Hive Export/Import/DropTable/DropPartition to support replication
semantics
--
Key: HIVE-10228
URL: https://issues.apache.org/jira/browse/HIVE-10228
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10228.2.patch, HIVE-10228.3.patch,
HIVE-10228.4.patch, HIVE-10228.5.patch, HIVE-10228.patch
We need to update a couple of hive commands to support replication semantics.
To wit, we need the following:
EXPORT ... [FOR [METADATA] REPLICATION(“comment”)]
Export will now support an extra optional clause to tell it that this export
is being prepared for the purpose of replication. There is also an additional
optional clause here, that allows for the export to be a metadata-only
export, to handle cases of capturing the diff for alter statements, for
example.
Also, if done for replication, the non-presence of a table, or a table being
a view/offline table/non-native table is not considered an error, and
instead, will result in a successful no-op.
IMPORT ... (as normal) – but handles new semantics
No syntax changes for import, but import will have to change to be able to
handle all the permutations of export dumps possible. Also, import will have
to ensure that it should update the object only if the update being imported
is not older than the state of the object. Also, import currently does not
work with dbname.tablename kind of specification, this should be fixed to
work.
DROP TABLE ... FOR REPLICATION('eventid')
Drop Table now has an additional clause, to specify that this drop table is
being done for replication purposes, and that the dop should not actually
drop the table if the table is newer than that event id specified.
ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid')
Similarly, Drop Partition also has an equivalent change to Drop Table.
=
In addition, we introduce a new property repl.last.id, which when tagged on
to table properties or partition properties on a replication-destination,
holds the effective state identifier of the object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10352) Investigate the failure of union_view.q
[ https://issues.apache.org/jira/browse/HIVE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10352: - Component/s: Tests Investigate the failure of union_view.q Key: HIVE-10352 URL: https://issues.apache.org/jira/browse/HIVE-10352 Project: Hive Issue Type: Bug Components: Tests Reporter: Aihua Xu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10307) Support to use number literals in partition column
[ https://issues.apache.org/jira/browse/HIVE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500498#comment-14500498 ] Lefty Leverenz commented on HIVE-10307: --- Found a typo in one of the parameter descriptions -- see review board. Support to use number literals in partition column -- Key: HIVE-10307 URL: https://issues.apache.org/jira/browse/HIVE-10307 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 1.0.0 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Attachments: HIVE-10307.1.patch, HIVE-10307.2.patch, HIVE-10307.patch Data types like TinyInt, SmallInt, BigInt or Decimal can be expressed as literals with postfix like Y, S, L, or BD appended to the number. These literals work in most Hive queries, but do not when they are used as partition column value. For a partitioned table like: create table partcoltypenum (key int, value string) partitioned by (tint tinyint, sint smallint, bint bigint); insert into partcoltypenum partition (tint=100Y, sint=1S, bint=1000L) select key, value from src limit 30; Queries like select, describe and drop partition do not work. For an example select * from partcoltypenum where tint=100Y and sint=1S and bint=1000L; does not return any rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9917) After HIVE-3454 is done, make int to timestamp conversion configurable
[ https://issues.apache.org/jira/browse/HIVE-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500516#comment-14500516 ] Aihua Xu commented on HIVE-9917: [~jdere] I updated the unit tests to check the old behavior and added the unit tests to compare the new/old behaviors. The test failures above are not related to the patch. Anything else to check? After HIVE-3454 is done, make int to timestamp conversion configurable -- Key: HIVE-9917 URL: https://issues.apache.org/jira/browse/HIVE-9917 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-9917.patch After HIVE-3454 is fixed, we will have correct behavior of converting int to timestamp. While the customers are using such incorrect behavior for so long, better to make it configurable so that in one release, it will default to old/inconsistent way and the next release will default to new/consistent way. And then we will deprecate it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9917) After HIVE-3454 is done, make int to timestamp conversion configurable
[ https://issues.apache.org/jira/browse/HIVE-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500527#comment-14500527 ] Jason Dere commented on HIVE-9917: -- I'll try to take a look later today After HIVE-3454 is done, make int to timestamp conversion configurable -- Key: HIVE-9917 URL: https://issues.apache.org/jira/browse/HIVE-9917 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-9917.patch After HIVE-3454 is fixed, we will have correct behavior of converting int to timestamp. While the customers are using such incorrect behavior for so long, better to make it configurable so that in one release, it will default to old/inconsistent way and the next release will default to new/consistent way. And then we will deprecate it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10331) ORC : Is null SARG filters out all row groups written in old ORC format
[
https://issues.apache.org/jira/browse/HIVE-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500531#comment-14500531
]
Mostafa Mokhtar commented on HIVE-10331:
[~prasanth_j]
These failures are unrelated
{code}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
{code}
ORC : Is null SARG filters out all row groups written in old ORC format
---
Key: HIVE-10331
URL: https://issues.apache.org/jira/browse/HIVE-10331
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.1.0
Reporter: Mostafa Mokhtar
Assignee: Mostafa Mokhtar
Fix For: 1.2.0
Attachments: HIVE-10331.01.patch, HIVE-10331.02.patch,
HIVE-10331.03.patch, HIVE-10331.03.patch, HIVE-10331.04.patch,
HIVE-10331.05.patch
Queries are returning wrong results as all row groups gets filtered out and
no rows get scanned.
{code}
SELECT
count(*)
FROM
store_sales
WHERE
ss_addr_sk IS NULL
{code}
With hive.optimize.index.filter disabled we get the correct results
In pickRowGroups stats show that hasNull_ is fales, while the rowgroup
actually has null.
Same query runs fine for newly loaded ORC tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10268) Merge cbo branch into trunk
[
https://issues.apache.org/jira/browse/HIVE-10268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500455#comment-14500455
]
Hive QA commented on HIVE-10268:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726051/HIVE-10268.4.patch
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 8712 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_timestamp_funcs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_4
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3478/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3478/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3478/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726051 - PreCommit-HIVE-TRUNK-Build
Merge cbo branch into trunk
---
Key: HIVE-10268
URL: https://issues.apache.org/jira/browse/HIVE-10268
Project: Hive
Issue Type: Task
Components: CBO
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-10268.1.patch, HIVE-10268.2.patch,
HIVE-10268.3.patch, HIVE-10268.4.patch, HIVE-10268.patch
Merge patch generated on basis of diffs of trunk with cbo-branch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10382) Aggregate stats cache for RDBMS based metastore codepath
[ https://issues.apache.org/jira/browse/HIVE-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10382: Issue Type: Improvement (was: Bug) Aggregate stats cache for RDBMS based metastore codepath Key: HIVE-10382 URL: https://issues.apache.org/jira/browse/HIVE-10382 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 1.2.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Similar to the work done on the HBase branch (HIVE-9693), the stats cache can potentially have performance gains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10380: --- Attachment: HIVE-10380.01.cbo.patch New patch after discussion with [~jpullokkaran]. CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.01.cbo.patch, HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10352) Investigate the failure of union_view.q
[ https://issues.apache.org/jira/browse/HIVE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-10352: -- Attachment: HIVE-10352.1.patch Investigate the failure of union_view.q Key: HIVE-10352 URL: https://issues.apache.org/jira/browse/HIVE-10352 Project: Hive Issue Type: Bug Components: Tests Reporter: Aihua Xu Attachments: HIVE-10352.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10352) Investigate the failure of union_view.q
[ https://issues.apache.org/jira/browse/HIVE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-10352: - Assignee: Vikram Dixit K Investigate the failure of union_view.q Key: HIVE-10352 URL: https://issues.apache.org/jira/browse/HIVE-10352 Project: Hive Issue Type: Bug Components: Tests Reporter: Aihua Xu Assignee: Vikram Dixit K Attachments: HIVE-10352.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10352) Investigate the failure of union_view.q
[ https://issues.apache.org/jira/browse/HIVE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500567#comment-14500567 ] Thejas M Nair commented on HIVE-10352: -- +1 Investigate the failure of union_view.q Key: HIVE-10352 URL: https://issues.apache.org/jira/browse/HIVE-10352 Project: Hive Issue Type: Bug Components: Tests Reporter: Aihua Xu Attachments: HIVE-10352.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500593#comment-14500593 ] Laljo John Pullokkaran commented on HIVE-10380: --- +1 CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.01.cbo.patch, HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10307) Support to use number literals in partition column
[ https://issues.apache.org/jira/browse/HIVE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10307: --- Attachment: HIVE-10307.3.patch Correct the typo in parameter description. Thanks [~leftylev] ! Support to use number literals in partition column -- Key: HIVE-10307 URL: https://issues.apache.org/jira/browse/HIVE-10307 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 1.0.0 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Attachments: HIVE-10307.1.patch, HIVE-10307.2.patch, HIVE-10307.3.patch, HIVE-10307.patch Data types like TinyInt, SmallInt, BigInt or Decimal can be expressed as literals with postfix like Y, S, L, or BD appended to the number. These literals work in most Hive queries, but do not when they are used as partition column value. For a partitioned table like: create table partcoltypenum (key int, value string) partitioned by (tint tinyint, sint smallint, bint bigint); insert into partcoltypenum partition (tint=100Y, sint=1S, bint=1000L) select key, value from src limit 30; Queries like select, describe and drop partition do not work. For an example select * from partcoltypenum where tint=100Y and sint=1S and bint=1000L; does not return any rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10384) RetryingMetaStoreClient does not retry wrapped TTransportExceptions
[
https://issues.apache.org/jira/browse/HIVE-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang reassigned HIVE-10384:
--
Assignee: Chaoyu Tang
RetryingMetaStoreClient does not retry wrapped TTransportExceptions
---
Key: HIVE-10384
URL: https://issues.apache.org/jira/browse/HIVE-10384
Project: Hive
Issue Type: Bug
Components: Clients
Reporter: Eric Liang
Assignee: Chaoyu Tang
This bug is very similar to HIVE-9436, in that a TTransportException wrapped
in a MetaException will not be retried. RetryingMetaStoreClient has a block
of code above the MetaException handler that retries thrift exceptions, but
this doesn't work when the exception is wrapped.
{code}
if ((e.getCause() instanceof TApplicationException) ||
(e.getCause() instanceof TProtocolException) ||
(e.getCause() instanceof TTransportException)) {
caughtException = (TException) e.getCause();
} else if ((e.getCause() instanceof MetaException)
e.getCause().getMessage().matches((?s).*JDO[a-zA-Z]*Exception.*)) {
caughtException = (MetaException) e.getCause();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500756#comment-14500756 ] Pengcheng Xiong commented on HIVE-10380: This patch + (HIVE-10375) works for cbo_union.q. ccing [~ashutoshc] and [~jpullokkaran] . [~jpullokkaran] said he had a better patch to replace HIVE-10375 CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.01.cbo.patch, HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10384) RetryingMetaStoreClient does not retry wrapped TTransportExceptions
[
https://issues.apache.org/jira/browse/HIVE-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Liang updated HIVE-10384:
--
Description:
This bug is very similar to HIVE-9436, in that a TTransportException wrapped in
a MetaException will not be retried. RetryingMetaStoreClient has a block of
code above the MetaException handler that retries thrift exceptions, but this
doesn't work when the exception is wrapped.
{code}
if ((e.getCause() instanceof TApplicationException) ||
(e.getCause() instanceof TProtocolException) ||
(e.getCause() instanceof TTransportException)) {
caughtException = (TException) e.getCause();
} else if ((e.getCause() instanceof MetaException)
e.getCause().getMessage().matches((?s).*JDO[a-zA-Z]*Exception.*))
{
caughtException = (MetaException) e.getCause();
{code}
was:This bug is very similar to HIVE-9436, in that a TTransportException
wrapped in a MetaException will not be retried.
RetryingMetaStoreClient does not retry wrapped TTransportExceptions
---
Key: HIVE-10384
URL: https://issues.apache.org/jira/browse/HIVE-10384
Project: Hive
Issue Type: Bug
Components: Clients
Reporter: Eric Liang
This bug is very similar to HIVE-9436, in that a TTransportException wrapped
in a MetaException will not be retried. RetryingMetaStoreClient has a block
of code above the MetaException handler that retries thrift exceptions, but
this doesn't work when the exception is wrapped.
{code}
if ((e.getCause() instanceof TApplicationException) ||
(e.getCause() instanceof TProtocolException) ||
(e.getCause() instanceof TTransportException)) {
caughtException = (TException) e.getCause();
} else if ((e.getCause() instanceof MetaException)
e.getCause().getMessage().matches((?s).*JDO[a-zA-Z]*Exception.*)) {
caughtException = (MetaException) e.getCause();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8648) numRows cannot be set by user
[
https://issues.apache.org/jira/browse/HIVE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500785#comment-14500785
]
Sergio Peña commented on HIVE-8648:
---
Hi [~ashutoshc] Could you help me know what kind of ALTER operations can cause
to invalidate these stats?
I tried running alter drop partitions while debugging, but the
{{updateUnpartitionedTableStatsFast}} is not called.
numRows cannot be set by user
-
Key: HIVE-8648
URL: https://issues.apache.org/jira/browse/HIVE-8648
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8648.1.patch, HIVE-8648.patch
Since HIVE-3777 users who want to set the number of rows for a table, must do
as follows:
{noformat}
alter table ... set tblproperties ('numRows' = '12345',
'STATS_GENERATED_VIA_STATS_TASK' = 'true');
{noformat}
Which is strange because (1) users can know the numbers of rows and (2) the
stat is not generated by a stats task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[
https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500627#comment-14500627
]
Hive QA commented on HIVE-8950:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726052/HIVE-8950.7.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8750 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3479/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3479/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3479/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726052 - PreCommit-HIVE-TRUNK-Build
Add support in ParquetHiveSerde to create table schema from a parquet file
--
Key: HIVE-8950
URL: https://issues.apache.org/jira/browse/HIVE-8950
Project: Hive
Issue Type: Improvement
Reporter: Ashish K Singh
Assignee: Ashish K Singh
Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch,
HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch,
HIVE-8950.patch
PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without
having to specify the column names and types. As, parquet files store schema
in their footer, it is possible to generate hive schema from parquet file's
metadata. This will improve usability of parquet backed tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9923) No clear message when from is missing
[
https://issues.apache.org/jira/browse/HIVE-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500633#comment-14500633
]
Chao commented on HIVE-9923:
OK, will take a look at this.
No clear message when from is missing
---
Key: HIVE-9923
URL: https://issues.apache.org/jira/browse/HIVE-9923
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jeff Zhang
Assignee: Yongzhi Chen
Attachments: HIVE-9923.1.patch, HIVE-9923.2.patch
For the following sql, from is missing but it throw NPE which is not clear
for user.
{code}
hive insert overwrite directory '/tmp/hive-3' select sb1.name, sb2.age
student_bucketed sb1 join student_bucketed sb2 on sb1.name=sb2.name;
FAILED: NullPointerException null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10227) Concrete implementation of Export/Import based ReplicationTaskFactory
[ https://issues.apache.org/jira/browse/HIVE-10227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10227: Attachment: HIVE-10227.2.patch Updating patch after initial feedback from Mithun on adding HCatClient.getConfVal, and using that to change how ReplicationTask.create and its factory loading works, instead of statically initializing a HiveConf. Also, added batching semantics for HCatReplicationTaskIterator so that even if a user specifies -1 as their max_events, we should not have OOM issues. In addition, HCatReplicationTaskIterator always returns max_events entries if available, irrespective of whether or not there are max_events relevant events in the next eventFrom+max_events events(old behaviour was that it'd return at-most max_events events as it simply passed these parameters to the metastore). Basically, HCatReplicationTaskIterator has been made more robust, and is now the primary interface point to its users. Concrete implementation of Export/Import based ReplicationTaskFactory - Key: HIVE-10227 URL: https://issues.apache.org/jira/browse/HIVE-10227 Project: Hive Issue Type: Sub-task Components: Import/Export Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-10227.2.patch, HIVE-10227.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9182) avoid FileSystem.getAclStatus rpc call for filesystems that don't support acl
[ https://issues.apache.org/jira/browse/HIVE-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500846#comment-14500846 ] Szehon Ho commented on HIVE-9182: - +1, thanks for pointing this out avoid FileSystem.getAclStatus rpc call for filesystems that don't support acl - Key: HIVE-9182 URL: https://issues.apache.org/jira/browse/HIVE-9182 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Thejas M Nair Assignee: Abdelrahman Shettia Fix For: 1.2.0 Attachments: HIVE-9182.2.patch, HIVE-9182.3.patch, HIVE-9182.3.patch File systems such as s3, wasp (azure) don't implement Hadoop FileSystem acl functionality. Hadoop23Shims has code that calls getAclStatus on file systems. Instead of calling getAclStatus and catching the exception, we can also check FsPermission#getAclBit . Additionally, instead of catching all exceptions for calls to getAclStatus and ignoring them, it is better to just catch UnsupportedOperationException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10386) CBO (Calcite Return Path): Disable Trivial Project Removal on ret path
[ https://issues.apache.org/jira/browse/HIVE-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-10386: -- Attachment: HIVE-10386.patch CBO (Calcite Return Path): Disable Trivial Project Removal on ret path -- Key: HIVE-10386 URL: https://issues.apache.org/jira/browse/HIVE-10386 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10386.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10385) Optionally disable partition creation to speedup ETL jobs
[ https://issues.apache.org/jira/browse/HIVE-10385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slava Markeyev updated HIVE-10385: -- Attachment: HIVE-10385.patch Optionally disable partition creation to speedup ETL jobs - Key: HIVE-10385 URL: https://issues.apache.org/jira/browse/HIVE-10385 Project: Hive Issue Type: Improvement Components: Hive Reporter: Slava Markeyev Priority: Minor Attachments: HIVE-10385.patch ETL jobs that create dynamic partitions with high cardinality perform the expensive step of metastore partition creation after query completion. Until bulk partition creation can be optimized there should be a way of optionally skipping this step. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics
[
https://issues.apache.org/jira/browse/HIVE-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500822#comment-14500822
]
Sushanth Sowmyan commented on HIVE-10228:
-
The reported failed tests are not related to this patch.
Changes to Hive Export/Import/DropTable/DropPartition to support replication
semantics
--
Key: HIVE-10228
URL: https://issues.apache.org/jira/browse/HIVE-10228
Project: Hive
Issue Type: Sub-task
Components: Import/Export
Affects Versions: 1.2.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: HIVE-10228.2.patch, HIVE-10228.3.patch,
HIVE-10228.4.patch, HIVE-10228.5.patch, HIVE-10228.patch
We need to update a couple of hive commands to support replication semantics.
To wit, we need the following:
EXPORT ... [FOR [METADATA] REPLICATION(“comment”)]
Export will now support an extra optional clause to tell it that this export
is being prepared for the purpose of replication. There is also an additional
optional clause here, that allows for the export to be a metadata-only
export, to handle cases of capturing the diff for alter statements, for
example.
Also, if done for replication, the non-presence of a table, or a table being
a view/offline table/non-native table is not considered an error, and
instead, will result in a successful no-op.
IMPORT ... (as normal) – but handles new semantics
No syntax changes for import, but import will have to change to be able to
handle all the permutations of export dumps possible. Also, import will have
to ensure that it should update the object only if the update being imported
is not older than the state of the object. Also, import currently does not
work with dbname.tablename kind of specification, this should be fixed to
work.
DROP TABLE ... FOR REPLICATION('eventid')
Drop Table now has an additional clause, to specify that this drop table is
being done for replication purposes, and that the dop should not actually
drop the table if the table is newer than that event id specified.
ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid')
Similarly, Drop Partition also has an equivalent change to Drop Table.
=
In addition, we introduce a new property repl.last.id, which when tagged on
to table properties or partition properties on a replication-destination,
holds the effective state identifier of the object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10380) CBO (Calcite Return Path): Add QBId to TableScan digest
[ https://issues.apache.org/jira/browse/HIVE-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-10380: -- Attachment: HIVE-10380.02.patch CBO (Calcite Return Path): Add QBId to TableScan digest --- Key: HIVE-10380 URL: https://issues.apache.org/jira/browse/HIVE-10380 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10380.01.cbo.patch, HIVE-10380.02.patch, HIVE-10380.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
