[jira] [Commented] (HIVE-14155) Vectorization: Custom UDF Vectorization annotations are ignored
[
https://issues.apache.org/jira/browse/HIVE-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360388#comment-15360388
]
Hive QA commented on HIVE-14155:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815927/HIVE-14155.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10292 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/356/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/356/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-356/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815927 - PreCommit-HIVE-MASTER-Build
> Vectorization: Custom UDF Vectorization annotations are ignored
> ---
>
> Key: HIVE-14155
> URL: https://issues.apache.org/jira/browse/HIVE-14155
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14155.1.patch
>
>
> {code}
> @VectorizedExpressions(value = { VectorStringRot13.class })
> {code}
> in a custom UDF Is ignored because the check for annotations happens after
> custom UDF detection.
> The custom UDF codepath is on the fail-over track of annotation lookups, so
> the detection during validation of SEL is sufficient, instead of during
> expression creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[
https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360367#comment-15360367
]
Hive QA commented on HIVE-14147:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815905/HIVE-14147.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10293 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/355/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/355/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-355/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815905 - PreCommit-HIVE-MASTER-Build
> Hive PPD might remove predicates when they are defined as a simple node e.g.
> "WHERE pred"
> -
>
> Key: HIVE-14147
> URL: https://issues.apache.org/jira/browse/HIVE-14147
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-14147.01.patch, HIVE-14147.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360349#comment-15360349
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815890/HIVE-14035.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10292 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/354/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/354/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-354/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815890 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[
https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360323#comment-15360323
]
Hive QA commented on HIVE-13560:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815831/HIVE-13560.7.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 85 failed/errored test(s), 9869 tests
executed
*Failed tests:*
{noformat}
TestBeelineArgParsing - did not produce a TEST-*.xml file
TestGenMapRedUtilsUsePartitionColumnsPositive - did not produce a TEST-*.xml
file
TestHCatDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatExternalDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatExternalNonPartitioned - did not produce a TEST-*.xml file
TestHCatExternalPartitioned - did not produce a TEST-*.xml file
TestHCatLoader - did not produce a TEST-*.xml file
TestHCatLoaderComplexSchema - did not produce a TEST-*.xml file
TestHCatLoaderEncryption - did not produce a TEST-*.xml file
TestHCatMutableDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatMutableNonPartitioned - did not produce a TEST-*.xml file
TestHCatMutablePartitioned - did not produce a TEST-*.xml file
TestHCatNonPartitioned - did not produce a TEST-*.xml file
TestHCatPartitioned - did not produce a TEST-*.xml file
TestHCatStorer - did not produce a TEST-*.xml file
TestHCatStorerMulti - did not produce a TEST-*.xml file
TestOrcFile - did not produce a TEST-*.xml file
TestPositivePartitionPrunerCompactExpr - did not produce a TEST-*.xml file
TestSessionManagerMetrics - did not produce a TEST-*.xml file
TestSessionState - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
on
testCliDriver_vector_complex_all(org.apache.hadoop.hive.cli.TestMiniLlapCliDriver)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
on
testCliDriver_vector_complex_join(org.apache.hadoop.hive.cli.TestMiniLlapCliDriver)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_empty_dir_in_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_external_table_with_space_in_location_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_file_with_header_footer
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_gen_udf_example_add10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_import_exported_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_input16_cc
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_insert_overwrite_directory2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_leftsemijoin_mr
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_fs2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
[jira] [Updated] (HIVE-14155) Vectorization: Custom UDF Vectorization annotations are ignored
[
https://issues.apache.org/jira/browse/HIVE-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14155:
---
Status: Patch Available (was: Open)
> Vectorization: Custom UDF Vectorization annotations are ignored
> ---
>
> Key: HIVE-14155
> URL: https://issues.apache.org/jira/browse/HIVE-14155
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14155.1.patch
>
>
> {code}
> @VectorizedExpressions(value = { VectorStringRot13.class })
> {code}
> in a custom UDF Is ignored because the check for annotations happens after
> custom UDF detection.
> The custom UDF codepath is on the fail-over track of annotation lookups, so
> the detection during validation of SEL is sufficient, instead of during
> expression creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14155) Vectorization: Custom UDF Vectorization annotations are ignored
[
https://issues.apache.org/jira/browse/HIVE-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14155:
---
Attachment: HIVE-14155.1.patch
> Vectorization: Custom UDF Vectorization annotations are ignored
> ---
>
> Key: HIVE-14155
> URL: https://issues.apache.org/jira/browse/HIVE-14155
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14155.1.patch
>
>
> {code}
> @VectorizedExpressions(value = { VectorStringRot13.class })
> {code}
> in a custom UDF Is ignored because the check for annotations happens after
> custom UDF detection.
> The custom UDF codepath is on the fail-over track of annotation lookups, so
> the detection during validation of SEL is sufficient, instead of during
> expression creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13925) ETL optimizations
[ https://issues.apache.org/jira/browse/HIVE-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek reassigned HIVE-13925: --- Assignee: Abhishek > ETL optimizations > - > > Key: HIVE-13925 > URL: https://issues.apache.org/jira/browse/HIVE-13925 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Abhishek >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14149) Joda Time causes an AmazonS3Exception on Hadoop3.0.0
[
https://issues.apache.org/jira/browse/HIVE-14149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360285#comment-15360285
]
Hive QA commented on HIVE-14149:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815827/HIVE-14149.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10292 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_timestamp_ints_casts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_windowing_gby
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/351/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/351/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-351/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815827 - PreCommit-HIVE-MASTER-Build
> Joda Time causes an AmazonS3Exception on Hadoop3.0.0
>
>
> Key: HIVE-14149
> URL: https://issues.apache.org/jira/browse/HIVE-14149
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
> Attachments: HIVE-14149.1.patch
>
>
> Java1.8u60 and higher cause Joda Time 2.5 to incorrectly format timezones,
> which leads to the aws server rejecting requests with the aws sdk hadoop3.0
> uses. This means any queries involving the s3a connector will return the
> following AmazonS3Exception:
> {code}
> com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication
> requires a valid Date or x-amz-date header
> {code}
> The fix for this is to update Joda Time from 2.5 to 2.8.1. See here for
> details:
> https://github.com/aws/aws-sdk-java/issues/444
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13749) Memory leak in Hive Metastore
[
https://issues.apache.org/jira/browse/HIVE-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360253#comment-15360253
]
Hive QA commented on HIVE-13749:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815813/HIVE-13749.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/350/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/350/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-350/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815813 - PreCommit-HIVE-MASTER-Build
> Memory leak in Hive Metastore
> -
>
> Key: HIVE-13749
> URL: https://issues.apache.org/jira/browse/HIVE-13749
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13749.1.patch, HIVE-13749.patch, Top_Consumers7.html
>
>
> Looking a heap dump of 10GB, a large number of Configuration objects(> 66k
> instances) are being retained. These objects along with its retained set is
> occupying about 95% of the heap space. This leads to HMS crashes every few
> days.
> I will attach an exported snapshot from the eclipse MAT.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360244#comment-15360244
]
Sergio Peña commented on HIVE-13884:
[~leftylev] Here's the description of the new parameter:
*hive.metastore.limit.partition.request*
This limits the number of partitions that can be requested from the metastore
for a given table. A query will not be executed if it attempts to fetch more
partitions per table than the limit configured. This parameter is preferred
over hive.limit.query.max.table.partition (deprecated). The default value is -1
(unlimited).
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13884:
---
Labels: TODOC2.2 (was: )
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13884:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Tests are not related to this patch. I tested locally, and previous jobs are
failing with them too.
Thanks guys for the review. I committed this to 2.2
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14148) Add branch-2.1 branch to pre-commit tests
[ https://issues.apache.org/jira/browse/HIVE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14148: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > Add branch-2.1 branch to pre-commit tests > - > > Key: HIVE-14148 > URL: https://issues.apache.org/jira/browse/HIVE-14148 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.2.0 > > Attachments: HIVE-14148.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360201#comment-15360201
]
Hive QA commented on HIVE-10100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815803/HIVE-10100.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_dboutput
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-349/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815803 - PreCommit-HIVE-MASTER-Build
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch,
> HIVE-10100.3.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14142) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path for Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360166#comment-15360166
]
Hive QA commented on HIVE-14142:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815800/HIVE-14142.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/348/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/348/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-348/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815800 - PreCommit-HIVE-MASTER-Build
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> for Hive on Spark
> ---
>
> Key: HIVE-14142
> URL: https://issues.apache.org/jira/browse/HIVE-14142
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14142.1.patch
>
>
> Similar to HIVE-14037, seems HOS also has the same issue. The jars in
> hive.reloadable.aux.jars.path are not available during runtime.
> {noformat}
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:232)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> xudf.XAdd
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:134)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1365)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.isDeterministic(FunctionRegistry.java:1328)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:100)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:74)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:406)
>
[jira] [Work started] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14147 started by Jesus Camacho Rodriguez. -- > Hive PPD might remove predicates when they are defined as a simple node e.g. > "WHERE pred" > - > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Status: Patch Available (was: In Progress) > Hive PPD might remove predicates when they are defined as a simple node e.g. > "WHERE pred" > - > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Attachment: HIVE-14147.01.patch > Hive PPD might remove predicates when they are defined as a simple node e.g. > "WHERE pred" > - > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Status: Open (was: Patch Available) > Hive PPD might remove predicates when they are defined as a simple node e.g. > "WHERE pred" > - > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[
https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360131#comment-15360131
]
Hive QA commented on HIVE-13369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815793/HIVE-13369.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
org.apache.hadoop.hive.ql.io.TestAcidUtils.testBaseDeltas
org.apache.hadoop.hive.ql.io.TestAcidUtils.testOriginalDeltas
org.apache.hadoop.hive.ql.io.TestAcidUtils.testOverlapingDelta
org.apache.hadoop.hive.ql.io.TestAcidUtils.testOverlapingDelta2
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorPartitionCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorPartitionCompactionNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorTableCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMinorPartitionCompaction
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMinorTableCompaction
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/347/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/347/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-347/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815793 - PreCommit-HIVE-MASTER-Build
> AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing
> the "best" base file
> --
>
> Key: HIVE-13369
> URL: https://issues.apache.org/jira/browse/HIVE-13369
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Blocker
> Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch
>
>
> The JavaDoc on getAcidState() reads, in part:
> "Note that because major compactions don't
>preserve the history, we can't use a base directory that includes a
>transaction id that we must exclude."
> which is correct but there is nothing in the code that does this.
> And if we detect a situation where txn X must be excluded but and there are
> deltas that contain X, we'll have to abort the txn. This can't (reasonably)
> happen with auto commit mode, but with multi statement txns it's possible.
> Suppose some long running txn starts and lock in snapshot at 17 (HWM). An
> hour later it decides to access some partition for which all txns < 20 (for
> example) have already been compacted (i.e. GC'd).
> ==
> Here is a more concrete example. Let's say the file for table A are as
> follows and created in the order listed.
> delta_4_4
> delta_5_5
> delta_4_5
> base_5
> delta_16_16
> delta_17_17
> base_17 (for example user ran major compaction)
> let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20
> and ExceptionList=<16>
> Assume that all txns <= 20 commit.
> Reader can't use base_17 because it has result of txn16. So it should chose
> base_5 "TxnBase bestBase" in _getChildState()_.
> Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and
> delta_17_17 in _Directory_ object. This would represent acceptable snapshot
> for such reader.
> The issue is if at the same time the Cleaner process is running. It
[jira] [Issue Comment Deleted] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-14143:
-
Comment: was deleted
(was: Patch updated.)
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.1.patch, HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-14143:
-
Attachment: HIVE-14143.1.patch
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.1.patch, HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-14143:
-
Attachment: HIVE-14143.1.patch
Patch updated.
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-14143:
-
Attachment: (was: HIVE-14143.1.patch)
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14038) miscellaneous acid improvements
[ https://issues.apache.org/jira/browse/HIVE-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360108#comment-15360108 ] Jesus Camacho Rodriguez commented on HIVE-14038: Sure, no need for approval, go ahead! Thanks > miscellaneous acid improvements > --- > > Key: HIVE-14038 > URL: https://issues.apache.org/jira/browse/HIVE-14038 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14038.2.patch, HIVE-14038.3.patch, > HIVE-14038.8.patch, HIVE-14038.patch > > > 1. fix thread name inHouseKeeperServiceBase (currently they are all > "org.apache.hadoop.hive.ql.txn.compactor.HouseKeeperServiceBase$1-0") > 2. dump metastore configs from HiveConf on start up to help record values of > properties > 3. add some tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360107#comment-15360107 ] Jesus Camacho Rodriguez commented on HIVE-13725: Go ahead! Thanks > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch, HIVE-13725.5.patch, addendum.txt > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[
https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360091#comment-15360091
]
Hive QA commented on HIVE-14147:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815776/HIVE-14147.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 114 failed/errored test(s), 10288 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_id3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_window
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_interval_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join44
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_disablecbo_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_with_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_special_character_in_tabnames_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_explain_rewrite
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqual_corr_expr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_offcbo
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionall_join_nullconstant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionall_unbalancedppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin_3way
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query12
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query13
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query15
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query17
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query18
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query19
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query20
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query21
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query22
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query25
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: Open) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Attachment: HIVE-14035.07.patch Enable rolling downgrade whereby legacy type becomes the default behavior for newly created acid tables > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Open (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14148) Add branch-2.1 branch to pre-commit tests
[ https://issues.apache.org/jira/browse/HIVE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360076#comment-15360076 ] Ashutosh Chauhan commented on HIVE-14148: - +1 [~spena] Once you commit this, you may want to send out a note on dev@ list to provide instructions on how to get patch tested for branch-2.1 > Add branch-2.1 branch to pre-commit tests > - > > Key: HIVE-14148 > URL: https://issues.apache.org/jira/browse/HIVE-14148 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14148.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360057#comment-15360057
]
Nemon Lou commented on HIVE-14143:
--
Referring to ORC and LazySimpleSerde, rawDataSize is calculated without any
care of column projection.
So rawDataSize calculation for RCFile can be the same way.Right?
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360049#comment-15360049
]
Nemon Lou commented on HIVE-14143:
--
Agreed. As describe in TableScanDesc.java
{code}
// Both neededColumnIDs and neededColumns should never be null.
// When neededColumnIDs is an empty list,
// it means no needed column (e.g. we do not need any column to evaluate
// SELECT count(*) FROM t).
private List neededColumnIDs;
{code}
I must has been misleading by the following code in HiveInputFormat.java:
{code}
private void pushProjection(final JobConf newjob, final StringBuilder
readColumnsBuffer,
final StringBuilder readColumnNamesBuffer) {
String readColIds = readColumnsBuffer.toString();
String readColNames = readColumnNamesBuffer.toString();
boolean readAllColumns = readColIds.isEmpty() ? true : false;
newjob.setBoolean(ColumnProjectionUtils.READ_ALL_COLUMNS, readAllColumns);
...
}
{code}
The solution is not clear for me . Any suggestions?
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14139) NPE dropping permanent function
[
https://issues.apache.org/jira/browse/HIVE-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360046#comment-15360046
]
Hive QA commented on HIVE-14139:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815715/HIVE-14139.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/345/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/345/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-345/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815715 - PreCommit-HIVE-MASTER-Build
> NPE dropping permanent function
> ---
>
> Key: HIVE-14139
> URL: https://issues.apache.org/jira/browse/HIVE-14139
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14139.1.patch
>
>
> To reproduce:
> 1. Start a CLI session and create a permanent function.
> 2. Exit current CLI session.
> 3. Start a new CLI session and drop the function.
> Stack trace:
> {noformat}
> FAILED: error during drop function: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Registry.removePersistentFunctionUnderLock(Registry.java:513)
> at
> org.apache.hadoop.hive.ql.exec.Registry.unregisterFunction(Registry.java:501)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.unregisterPermanentFunction(FunctionRegistry.java:1532)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.dropPermanentFunction(FunctionTask.java:228)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:95)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1860)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1316)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1073)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14122) VectorMapOperator: Missing update to AbstractMapOperator::numRows
[ https://issues.apache.org/jira/browse/HIVE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14122: --- Resolution: Fixed Fix Version/s: 2.2.0 Release Note: VectorMapOperator: Missing update to AbstractMapOperator::numRows (Gopal V, reviewed by Sergey Shelukhin) Status: Resolved (was: Patch Available) Pushed to master, thanks [~sershe]! > VectorMapOperator: Missing update to AbstractMapOperator::numRows > - > > Key: HIVE-14122 > URL: https://issues.apache.org/jira/browse/HIVE-14122 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0, 2.2.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-14122.1.patch > > > The INPUT_RECORDS counter is out of sync with the actual # of rows-read in > vectorized and non-vectorized modes. > This means Tez record summaries are off by a large margin or is 0 for those > vertices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14122) VectorMapOperator: Missing update to AbstractMapOperator::numRows
[ https://issues.apache.org/jira/browse/HIVE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360023#comment-15360023 ] Gopal V commented on HIVE-14122: [~sershe]: The vector_complex queries are failing with real errors, but have been failing for a while before this patch. > VectorMapOperator: Missing update to AbstractMapOperator::numRows > - > > Key: HIVE-14122 > URL: https://issues.apache.org/jira/browse/HIVE-14122 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0, 2.2.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-14122.1.patch > > > The INPUT_RECORDS counter is out of sync with the actual # of rows-read in > vectorized and non-vectorized modes. > This means Tez record summaries are off by a large margin or is 0 for those > vertices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14143) RawDataSize of RCFile is zero after analyze
[
https://issues.apache.org/jira/browse/HIVE-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360012#comment-15360012
]
Pengcheng Xiong commented on HIVE-14143:
[~nemon], thanks a lot for your explanation. I think current assumption that
"empty column ids means read all columns" is confusing and misleading. I would
prefer the following assumption:
{code}
getNeededColumnIDs==null or empty ===means==> do not need any columns
{code}
If you agree, could you please change the code accordingly? Thanks.
> RawDataSize of RCFile is zero after analyze
>
>
> Key: HIVE-14143
> URL: https://issues.apache.org/jira/browse/HIVE-14143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-14143.patch
>
>
> After running the following analyze command ,rawDataSize becomes zero for
> rcfile tables.
> {noformat}
> analyze table RCFILE_TABLE compute statistics ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
