[jira] [Updated] (HIVE-10514) Fix MiniCliDriver tests failure
[
https://issues.apache.org/jira/browse/HIVE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-10514:
-
Attachment: HIVE-10514.1.patch
Tested with this patch locally and it seems to work. I can upload an improved
version of the patch if someone can verify this patch and the approach.
Thanks
Hari
> Fix MiniCliDriver tests failure
> ---
>
> Key: HIVE-10514
> URL: https://issues.apache.org/jira/browse/HIVE-10514
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Szehon Ho
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10514.1.patch
>
>
> The MinimrCliDriver tests always fail to run.
> This can be reproduced by the following, run the command:
> {noformat}
> mvn -B test -Phadoop-2 -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_map_operators.q,join1.q,bucketmapjoin7.q,udf_using.q
> {noformat}
> And the following exception comes:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project hive-it-qfile: Compilation failure
> [ERROR]
> /Users/szehon/repos/apache-hive-git/hive/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriver.java:[100,22]
> code too large
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10514) Fix MiniCliDriver tests failure
[
https://issues.apache.org/jira/browse/HIVE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan reassigned HIVE-10514:
Assignee: Hari Sankar Sivarama Subramaniyan
> Fix MiniCliDriver tests failure
> ---
>
> Key: HIVE-10514
> URL: https://issues.apache.org/jira/browse/HIVE-10514
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Szehon Ho
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> The MinimrCliDriver tests always fail to run.
> This can be reproduced by the following, run the command:
> {noformat}
> mvn -B test -Phadoop-2 -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_map_operators.q,join1.q,bucketmapjoin7.q,udf_using.q
> {noformat}
> And the following exception comes:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project hive-it-qfile: Compilation failure
> [ERROR]
> /Users/szehon/repos/apache-hive-git/hive/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriver.java:[100,22]
> code too large
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10518) Replace US_ASCII charset with US-ASCII in encode and decode description
[
https://issues.apache.org/jira/browse/HIVE-10518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-10518:
---
Attachment: HIVE-10518.1.patch
patch #1
> Replace US_ASCII charset with US-ASCII in encode and decode description
> ---
>
> Key: HIVE-10518
> URL: https://issues.apache.org/jira/browse/HIVE-10518
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Trivial
> Attachments: HIVE-10518.1.patch
>
>
> GenericUDFEncode line 45 says
> {code}
> extended = "Possible options for the character set are 'US_ASCII',
> 'ISO-8859-1',\n" +
> {code}
> But Java Charset API says that valid ASCII charset value is US-ASCII
> https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8165) Annotation changes for replication
[ https://issues.apache.org/jira/browse/HIVE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan reassigned HIVE-8165: -- Assignee: Sushanth Sowmyan > Annotation changes for replication > -- > > Key: HIVE-8165 > URL: https://issues.apache.org/jira/browse/HIVE-8165 > Project: Hive > Issue Type: Sub-task > Components: Import/Export >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-8165.patch > > > We need to make a couple of changes for annotating the recent changes. > a) Marking old notification listener in HCatalog as @Deprecated, linking > instead to the new repl/ module. > b) Mark the new interfaces as @Evolving @Unstable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8165) Annotation changes for replication
[ https://issues.apache.org/jira/browse/HIVE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516481#comment-14516481 ] Sushanth Sowmyan commented on HIVE-8165: [~alangates], could you please review? > Annotation changes for replication > -- > > Key: HIVE-8165 > URL: https://issues.apache.org/jira/browse/HIVE-8165 > Project: Hive > Issue Type: Sub-task > Components: Import/Export >Reporter: Sushanth Sowmyan > Attachments: HIVE-8165.patch > > > We need to make a couple of changes for annotating the recent changes. > a) Marking old notification listener in HCatalog as @Deprecated, linking > instead to the new repl/ module. > b) Mark the new interfaces as @Evolving @Unstable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-8165) Annotation changes for replication
[
https://issues.apache.org/jira/browse/HIVE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516479#comment-14516479
]
Sushanth Sowmyan edited comment on HIVE-8165 at 4/28/15 6:22 AM:
-
Patch attached that does the following:
a) Marks new HCatClient calls and ReplicationTask as
\@InterfaceAudience.LimitedPrivate(\{"Apache Hive, Apache Falcon"\}) and
\@InterfaceStability.Evolving
b) Marks new MetastoreClient calls as
\@InterfaceAudience.LimitedPrivate(\{"HCatalog"\})
was (Author: sushanth):
Patch attached that does the following:
a) Marks new HCatClient calls and ReplicationTask as
@InterfaceAudience.LimitedPrivate({"Apache Hive, Apache Falcon"}) and
@InterfaceStability.Evolving
b) Marks new MetastoreClient calls as
@InterfaceAudience.LimitedPrivate({"HCatalog"})
> Annotation changes for replication
> --
>
> Key: HIVE-8165
> URL: https://issues.apache.org/jira/browse/HIVE-8165
> Project: Hive
> Issue Type: Sub-task
> Components: Import/Export
>Reporter: Sushanth Sowmyan
> Attachments: HIVE-8165.patch
>
>
> We need to make a couple of changes for annotating the recent changes.
> a) Marking old notification listener in HCatalog as @Deprecated, linking
> instead to the new repl/ module.
> b) Mark the new interfaces as @Evolving @Unstable
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8165) Annotation changes for replication
[
https://issues.apache.org/jira/browse/HIVE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-8165:
---
Attachment: HIVE-8165.patch
Patch attached that does the following:
a) Marks new HCatClient calls and ReplicationTask as
@InterfaceAudience.LimitedPrivate({"Apache Hive, Apache Falcon"}) and
@InterfaceStability.Evolving
b) Marks new MetastoreClient calls as
@InterfaceAudience.LimitedPrivate({"HCatalog"})
> Annotation changes for replication
> --
>
> Key: HIVE-8165
> URL: https://issues.apache.org/jira/browse/HIVE-8165
> Project: Hive
> Issue Type: Sub-task
> Components: Import/Export
>Reporter: Sushanth Sowmyan
> Attachments: HIVE-8165.patch
>
>
> We need to make a couple of changes for annotating the recent changes.
> a) Marking old notification listener in HCatalog as @Deprecated, linking
> instead to the new repl/ module.
> b) Mark the new interfaces as @Evolving @Unstable
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10447) Beeline JDBC Driver to support 2 way SSL
[ https://issues.apache.org/jira/browse/HIVE-10447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516471#comment-14516471 ] Lefty Leverenz commented on HIVE-10447: --- Does this need documentation? If so, please add a TODOC1.2 label. Here's where the doc would go: * [HiveServer2 Clients | https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients] > Beeline JDBC Driver to support 2 way SSL > > > Key: HIVE-10447 > URL: https://issues.apache.org/jira/browse/HIVE-10447 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 1.2.0 > > Attachments: HIVE-10447.1.patch, HIVE-10447.2.patch, > HIVE-10447.2.patch > > > This jira should cover 2-way SSL authentication between the JDBC Client and > server which requires the driver to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10485) Create md5 UDF
[
https://issues.apache.org/jira/browse/HIVE-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-10485:
---
Attachment: HIVE-10485.2.patch
patch #2
- extend from UDF
- support BytesWritable
> Create md5 UDF
> --
>
> Key: HIVE-10485
> URL: https://issues.apache.org/jira/browse/HIVE-10485
> Project: Hive
> Issue Type: Task
> Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
> Attachments: HIVE-10485.1.patch, HIVE-10485.2.patch
>
>
> MD5(str)
> Calculates an MD5 128-bit checksum for the string. The value is returned as a
> string of 32 hex digits, or NULL if the argument was NULL. The return value
> can, for example, be used as a hash key.
> Example:
> {code}
> SELECT MD5('udf_md5');
> 'ce62ef0d2d27dc37b6d488b92f4b24fd'
> {code}
> online md5 generator: http://www.md5.cz/
> MySQL has md5 function:
> https://dev.mysql.com/doc/refman/5.5/en/encryption-functions.html#function_md5
> PostgreSQL also has md5 function:
> http://www.postgresql.org/docs/9.1/static/functions-string.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516466#comment-14516466
]
Hive QA commented on HIVE-10476:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728696/HIVE-10476.2-spark.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8721 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucket6.q-scriptfile1_win.q-quotedid_smb.q-and-1-more - did
not produce a TEST-*.xml file
TestMinimrCliDriver-bucketizedhiveinputformat.q-empty_dir_in_table.q - did not
produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-infer_bucket_sort_map_operators.q-load_hdfs_file_with_space_in_the_name.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-import_exported_table.q-truncate_column_buckets.q-bucket_num_reducers2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-infer_bucket_sort_num_buckets.q-parallel_orderby.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-join1.q-infer_bucket_sort_bucketed_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-bucket5.q-infer_bucket_sort_merge.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-input16_cc.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-bucket_num_reducers.q-scriptfile1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx_cbo_2.q-bucketmapjoin6.q-bucket4.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-reduce_deduplicate.q-infer_bucket_sort_dyn_part.q-udf_using.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-uber_reduce.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-stats_counter_partitioned.q-external_table_with_space_in_location_path.q-disable_merge_for_bucketing.q-and-1-more
- did not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/845/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/845/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-845/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728696 - PreCommit-HIVE-SPARK-Build
> Hive query should fail when it fails to initialize a session in
> SetSparkReducerParallelism [Spark Branch]
> -
>
> Key: HIVE-10476
> URL: https://issues.apache.org/jira/browse/HIVE-10476
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HIVE-10476.1-spark.patch, HIVE-10476.2-spark.patch
>
>
> Currently, for a Hive query HoS need to get a session
> a session twice, once in SparkSetReducerParallelism, and another when
> submitting the actual job.
> The issue is that sometimes there's problem when launching a Yarn application
> (e.g., don't have permission), then user will have to wait for two timeouts,
> because both session initializations will fail. This turned out to happen
> frequently.
> This JIRA proposes to fail the query in SparkSetReducerParallelism, when it
> cannot initialize the session.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10519) Move TestGenericUDF classes to udf.generic package
[
https://issues.apache.org/jira/browse/HIVE-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-10519:
---
Description:
The following TestGenericUDF classes are located in udf package instead of
udf.generic.
{code}
TestGenericUDFDate.java
TestGenericUDFDateAdd.java
TestGenericUDFDateDiff.java
TestGenericUDFDateSub.java
TestGenericUDFUtils.java
{code}
was:
5 TestGenericUDF classes are located in udf package instead of udf.generic
{code}
TestGenericUDFDate.java
TestGenericUDFDateAdd.java
TestGenericUDFDateDiff.java
TestGenericUDFDateSub.java
TestGenericUDFUtils.java
{code}
> Move TestGenericUDF classes to udf.generic package
> --
>
> Key: HIVE-10519
> URL: https://issues.apache.org/jira/browse/HIVE-10519
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Trivial
>
> The following TestGenericUDF classes are located in udf package instead of
> udf.generic.
> {code}
> TestGenericUDFDate.java
> TestGenericUDFDateAdd.java
> TestGenericUDFDateDiff.java
> TestGenericUDFDateSub.java
> TestGenericUDFUtils.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10518) Replace US_ASCII charset with US-ASCII in encode and decode description
[
https://issues.apache.org/jira/browse/HIVE-10518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-10518:
---
Description:
GenericUDFEncode line 45 says
{code}
extended = "Possible options for the character set are 'US_ASCII',
'ISO-8859-1',\n" +
{code}
But Java Charset API says that valid ASCII charset value is US-ASCII
https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html
> Replace US_ASCII charset with US-ASCII in encode and decode description
> ---
>
> Key: HIVE-10518
> URL: https://issues.apache.org/jira/browse/HIVE-10518
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Trivial
>
> GenericUDFEncode line 45 says
> {code}
> extended = "Possible options for the character set are 'US_ASCII',
> 'ISO-8859-1',\n" +
> {code}
> But Java Charset API says that valid ASCII charset value is US-ASCII
> https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10517) HCatPartition should not be created with "" as location in tests
[ https://issues.apache.org/jira/browse/HIVE-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10517: Attachment: (was: HIVE-10517.patch) > HCatPartition should not be created with "" as location in tests > > > Key: HIVE-10517 > URL: https://issues.apache.org/jira/browse/HIVE-10517 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 1.2.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-10517.patch > > > Tests in TestHCatClient and TestCommands wind up instantiating HCatPartition > with a dummy empty String as location. This causes test failures when run > against an existing metastore, as introduced by HIVE-10074. > We need to instantiate actual values instead of dummy "" strings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10517) HCatPartition should not be created with "" as location in tests
[ https://issues.apache.org/jira/browse/HIVE-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10517: Attachment: HIVE-10517.patch > HCatPartition should not be created with "" as location in tests > > > Key: HIVE-10517 > URL: https://issues.apache.org/jira/browse/HIVE-10517 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 1.2.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-10517.patch > > > Tests in TestHCatClient and TestCommands wind up instantiating HCatPartition > with a dummy empty String as location. This causes test failures when run > against an existing metastore, as introduced by HIVE-10074. > We need to instantiate actual values instead of dummy "" strings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516425#comment-14516425
]
Sushanth Sowmyan commented on HIVE-9674:
Since this jira seems to not be picked up by precommit tests automatically, I
manually queued one over at
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/
, build#3626
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10191) ORC: Cleanup writer per-row synchronization
[
https://issues.apache.org/jira/browse/HIVE-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516421#comment-14516421
]
Hive QA commented on HIVE-10191:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728490/HIVE-10191.4.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8818 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3618/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3618/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3618/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728490 - PreCommit-HIVE-TRUNK-Build
> ORC: Cleanup writer per-row synchronization
> ---
>
> Key: HIVE-10191
> URL: https://issues.apache.org/jira/browse/HIVE-10191
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-10191.1.patch, HIVE-10191.2.patch,
> HIVE-10191.3.patch, HIVE-10191.4.patch
>
>
> ORC writers were originally meant to be thread-safe, but in the present day
> implementation each ORC writer is entirely share-nothing which converts most
> of the synchronized blocks in ORC as entirely uncontested locks.
> These uncontested locks prevent the JVM from inlining/optimizing these
> methods, while adding no extra thread-safety to the ORC writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10517) HCatPartition should not be created with "" as location in tests
[ https://issues.apache.org/jira/browse/HIVE-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10517: Attachment: HIVE-10517.patch Patch attached. [~thejas], could you please review? > HCatPartition should not be created with "" as location in tests > > > Key: HIVE-10517 > URL: https://issues.apache.org/jira/browse/HIVE-10517 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 1.2.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-10517.patch > > > Tests in TestHCatClient and TestCommands wind up instantiating HCatPartition > with a dummy empty String as location. This causes test failures when run > against an existing metastore, as introduced by HIVE-10074. > We need to instantiate actual values instead of dummy "" strings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10498) LLAP: Resolve everything in llap-daemon-site.xml
[ https://issues.apache.org/jira/browse/HIVE-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516382#comment-14516382 ] Lefty Leverenz commented on HIVE-10498: --- Thanks Gopal. > LLAP: Resolve everything in llap-daemon-site.xml > > > Key: HIVE-10498 > URL: https://issues.apache.org/jira/browse/HIVE-10498 > Project: Hive > Issue Type: Sub-task >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10498.patch > > > Configuring a sequence of hadoop execution parameters via llap-daemon-site.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10498) LLAP: Resolve everything in llap-daemon-site.xml
[ https://issues.apache.org/jira/browse/HIVE-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516381#comment-14516381 ] Lefty Leverenz commented on HIVE-10498: --- Thanks Gopal. > LLAP: Resolve everything in llap-daemon-site.xml > > > Key: HIVE-10498 > URL: https://issues.apache.org/jira/browse/HIVE-10498 > Project: Hive > Issue Type: Sub-task >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10498.patch > > > Configuring a sequence of hadoop execution parameters via llap-daemon-site.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10498) LLAP: Resolve everything in llap-daemon-site.xml
[ https://issues.apache.org/jira/browse/HIVE-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516378#comment-14516378 ] Gopal V commented on HIVE-10498: [~leftylev]: no docs for this - the .xml files updated are used in unit-tests (for coverage) > LLAP: Resolve everything in llap-daemon-site.xml > > > Key: HIVE-10498 > URL: https://issues.apache.org/jira/browse/HIVE-10498 > Project: Hive > Issue Type: Sub-task >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10498.patch > > > Configuring a sequence of hadoop execution parameters via llap-daemon-site.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516362#comment-14516362 ] Rui Li commented on HIVE-10476: --- +1 > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch, HIVE-10476.2-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-10476: Attachment: HIVE-10476.2-spark.patch > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch, HIVE-10476.2-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-10476: Attachment: (was: HIVE-10476.2-spark.patch) > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-10476: Attachment: HIVE-10476.2-spark.patch > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch, HIVE-10476.2-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10498) LLAP: Resolve everything in llap-daemon-site.xml
[ https://issues.apache.org/jira/browse/HIVE-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516326#comment-14516326 ] Lefty Leverenz commented on HIVE-10498: --- Does this need documentation? If so, add link "relates to" HIVE-9850 (documentation for llap). > LLAP: Resolve everything in llap-daemon-site.xml > > > Key: HIVE-10498 > URL: https://issues.apache.org/jira/browse/HIVE-10498 > Project: Hive > Issue Type: Sub-task >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10498.patch > > > Configuring a sequence of hadoop execution parameters via llap-daemon-site.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10500) Repeated deadlocks in underlying RDBMS cause transaction or lock failure
[
https://issues.apache.org/jira/browse/HIVE-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516321#comment-14516321
]
Hive QA commented on HIVE-10500:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728482/HIVE-10050.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8818 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3617/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3617/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3617/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728482 - PreCommit-HIVE-TRUNK-Build
> Repeated deadlocks in underlying RDBMS cause transaction or lock failure
>
>
> Key: HIVE-10500
> URL: https://issues.apache.org/jira/browse/HIVE-10500
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.14.0, 1.0.0, 1.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-10050.patch
>
>
> In some cases in a busy system, deadlocks in the metastore RDBMS can cause
> failures in Hive locks and transactions when using DbTxnManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10297) Package a default ivysettings.xml file in hive-exec jar.
[ https://issues.apache.org/jira/browse/HIVE-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Nag resolved HIVE-10297. -- Resolution: Duplicate The issue is will be resolved in HIVE-10251 > Package a default ivysettings.xml file in hive-exec jar. > > > Key: HIVE-10297 > URL: https://issues.apache.org/jira/browse/HIVE-10297 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Anant Nag >Assignee: Anant Nag > > Currently DependencyResolver searches for the ivysettings.xml file in the > HIVE_HOME, HIVE_CONF_DIR directories and the classpath. We should package a > default ivysettings file in the hive-exec jar so that we fall back to it if > no ivysettings file is found in any of the above locations. > The default ivysettings.xml file could be named > hive-ivysettings-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516278#comment-14516278 ] Rui Li commented on HIVE-10476: --- Yeah that looks good to me. > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516272#comment-14516272
]
Chao Sun commented on HIVE-10476:
-
OK, how about this:
{code}
...
sparkMemoryAndCores = sparkSession.getMemoryAndCores();
} catch (HiveException e) {
throw new SemanticException("Failed to get a spark session: " + e);
} catch (Exception e) {
LOG.warn("Failed to get spark memory/core info", e);
}
...
{code}
I think it should still continue if failed to get mem/core info.
> Hive query should fail when it fails to initialize a session in
> SetSparkReducerParallelism [Spark Branch]
> -
>
> Key: HIVE-10476
> URL: https://issues.apache.org/jira/browse/HIVE-10476
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HIVE-10476.1-spark.patch
>
>
> Currently, for a Hive query HoS need to get a session
> a session twice, once in SparkSetReducerParallelism, and another when
> submitting the actual job.
> The issue is that sometimes there's problem when launching a Yarn application
> (e.g., don't have permission), then user will have to wait for two timeouts,
> because both session initializations will fail. This turned out to happen
> frequently.
> This JIRA proposes to fail the query in SparkSetReducerParallelism, when it
> cannot initialize the session.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10511) Replacing the implementation of Hive CLI using Beeline
[ https://issues.apache.org/jira/browse/HIVE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516260#comment-14516260 ] Xuefu Zhang commented on HIVE-10511: [~Ferd], Please note that the document link is a little outdated in the sense that some of functionality has already removed such as accepting hostname and port, which is meant for HiveServer1. Therefore, we should focus whatever remains. The syntax of the command line might also has difference with Beeline, which needs to be identified. As to a branch for this development, I think it makes sense. I will discuss in dev list and create it accordingly. > Replacing the implementation of Hive CLI using Beeline > -- > > Key: HIVE-10511 > URL: https://issues.apache.org/jira/browse/HIVE-10511 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > > Hive CLI is a legacy tool which had two main use cases: > 1. a thick client for SQL on hadoop > 2. a command line tool for HiveServer1. > HiveServer1 is already deprecated and removed from Hive code base, so use > case #2 is out of the question. For #1, Beeline provides or is supposed to > provides equal functionality, yet is implemented differently from Hive CLI. > As it has been a while that Hive community has been recommending Beeline + > HS2 configuration, ideally we should deprecating Hive CLI. Because of wide > use of Hive CLI, we instead propose replacing Hive CLI's implementation with > Beeline plus embedded HS2 so that Hive community only needs to maintain a > single code path. In this way, Hive CLI is just an alias to Beeline at either > shell script level or at high code level. The goal is that no changes or > minimum changes are expected from existing user scrip using Hive CLI. > This is an Umbrella JIRA covering all tasks related to this initiative. Over > the last year or two, Beeline has been improved significantly to match what > Hive CLI offers. Still, there may still be some gaps or deficiency to be > discovered and fixed. In the meantime, we also want to make sure the enough > tests are included and performance impact is identified and addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10516) Measure Hive CLI's performance difference before and after implementation is switched
[ https://issues.apache.org/jira/browse/HIVE-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu reassigned HIVE-10516: --- Assignee: Ferdinand Xu > Measure Hive CLI's performance difference before and after implementation is > switched > - > > Key: HIVE-10516 > URL: https://issues.apache.org/jira/browse/HIVE-10516 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10501) remove HashTableSinkDesc fields which hide base class fields
[
https://issues.apache.org/jira/browse/HIVE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516255#comment-14516255
]
Hive QA commented on HIVE-10501:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728477/HIVE-10501.1.patch
{color:red}ERROR:{color} -1 due to 313 failed/errored test(s), 8818 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestCliDriv
[jira] [Commented] (HIVE-10511) Replacing the implementation of Hive CLI using Beeline
[ https://issues.apache.org/jira/browse/HIVE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516251#comment-14516251 ] Ferdinand Xu commented on HIVE-10511: - This is what hive cli supported: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli Currently qtest framework is using clidriver to process command. I think the first step should be replacing it with beeline. I do believe we can resolve some compatibility issues after resolving all failed issues caused by the refactory. [~xuefuz], do you think we need to create a branch to do the replacing work? Thank you! > Replacing the implementation of Hive CLI using Beeline > -- > > Key: HIVE-10511 > URL: https://issues.apache.org/jira/browse/HIVE-10511 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > > Hive CLI is a legacy tool which had two main use cases: > 1. a thick client for SQL on hadoop > 2. a command line tool for HiveServer1. > HiveServer1 is already deprecated and removed from Hive code base, so use > case #2 is out of the question. For #1, Beeline provides or is supposed to > provides equal functionality, yet is implemented differently from Hive CLI. > As it has been a while that Hive community has been recommending Beeline + > HS2 configuration, ideally we should deprecating Hive CLI. Because of wide > use of Hive CLI, we instead propose replacing Hive CLI's implementation with > Beeline plus embedded HS2 so that Hive community only needs to maintain a > single code path. In this way, Hive CLI is just an alias to Beeline at either > shell script level or at high code level. The goal is that no changes or > minimum changes are expected from existing user scrip using Hive CLI. > This is an Umbrella JIRA covering all tasks related to this initiative. Over > the last year or two, Beeline has been improved significantly to match what > Hive CLI offers. Still, there may still be some gaps or deficiency to be > discovered and fixed. In the meantime, we also want to make sure the enough > tests are included and performance impact is identified and addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-1161) Hive Replication
[ https://issues.apache.org/jira/browse/HIVE-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-1161. -- Resolution: Duplicate > Hive Replication > > > Key: HIVE-1161 > URL: https://issues.apache.org/jira/browse/HIVE-1161 > Project: Hive > Issue Type: New Feature > Components: Contrib >Reporter: Edward Capriolo >Assignee: SHAILESH PILARE >Priority: Minor > > Users may want to replicate data between two distinct hadoop clusters or two > hive warehouses on the same cluster. > Users may want to replicate entire catalogs or possibly on a table by table > basis. Should this process be batch driven or a be a full time running > application? What are some practical requirements, what are the limitations? > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10511) Replacing the implementation of Hive CLI using Beeline
[ https://issues.apache.org/jira/browse/HIVE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned HIVE-10511: -- Assignee: Ferdinand Xu (was: Xuefu Zhang) Assign this to Ferdinand as he has expressed his interest of working on this. > Replacing the implementation of Hive CLI using Beeline > -- > > Key: HIVE-10511 > URL: https://issues.apache.org/jira/browse/HIVE-10511 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > > Hive CLI is a legacy tool which had two main use cases: > 1. a thick client for SQL on hadoop > 2. a command line tool for HiveServer1. > HiveServer1 is already deprecated and removed from Hive code base, so use > case #2 is out of the question. For #1, Beeline provides or is supposed to > provides equal functionality, yet is implemented differently from Hive CLI. > As it has been a while that Hive community has been recommending Beeline + > HS2 configuration, ideally we should deprecating Hive CLI. Because of wide > use of Hive CLI, we instead propose replacing Hive CLI's implementation with > Beeline plus embedded HS2 so that Hive community only needs to maintain a > single code path. In this way, Hive CLI is just an alias to Beeline at either > shell script level or at high code level. The goal is that no changes or > minimum changes are expected from existing user scrip using Hive CLI. > This is an Umbrella JIRA covering all tasks related to this initiative. Over > the last year or two, Beeline has been improved significantly to match what > Hive CLI offers. Still, there may still be some gaps or deficiency to be > discovered and fixed. In the meantime, we also want to make sure the enough > tests are included and performance impact is identified and addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10514) Fix MiniCliDriver tests failure
[
https://issues.apache.org/jira/browse/HIVE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516175#comment-14516175
]
Ferdinand Xu commented on HIVE-10514:
-
Only a rough idea, how about separating the code into several parts?
> Fix MiniCliDriver tests failure
> ---
>
> Key: HIVE-10514
> URL: https://issues.apache.org/jira/browse/HIVE-10514
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Szehon Ho
>
> The MinimrCliDriver tests always fail to run.
> This can be reproduced by the following, run the command:
> {noformat}
> mvn -B test -Phadoop-2 -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_map_operators.q,join1.q,bucketmapjoin7.q,udf_using.q
> {noformat}
> And the following exception comes:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project hive-it-qfile: Compilation failure
> [ERROR]
> /Users/szehon/repos/apache-hive-git/hive/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriver.java:[100,22]
> code too large
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10382) Aggregate stats cache for RDBMS based metastore codepath
[
https://issues.apache.org/jira/browse/HIVE-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516167#comment-14516167
]
Hive QA commented on HIVE-10382:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728475/HIVE-10382.3.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 8822 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3615/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3615/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3615/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728475 - PreCommit-HIVE-TRUNK-Build
> Aggregate stats cache for RDBMS based metastore codepath
>
>
> Key: HIVE-10382
> URL: https://issues.apache.org/jira/browse/HIVE-10382
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 1.2.0
>
> Attachments: HIVE-10382.1.patch, HIVE-10382.1.patch,
> HIVE-10382.2.patch, HIVE-10382.2.patch, HIVE-10382.3.patch
>
>
> Similar to the work done on the HBase branch (HIVE-9693), the stats cache can
> potentially have performance gains.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10511) Replacing the implementation of Hive CLI using Beeline
[ https://issues.apache.org/jira/browse/HIVE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned HIVE-10511: -- Assignee: Xuefu Zhang > Replacing the implementation of Hive CLI using Beeline > -- > > Key: HIVE-10511 > URL: https://issues.apache.org/jira/browse/HIVE-10511 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > > Hive CLI is a legacy tool which had two main use cases: > 1. a thick client for SQL on hadoop > 2. a command line tool for HiveServer1. > HiveServer1 is already deprecated and removed from Hive code base, so use > case #2 is out of the question. For #1, Beeline provides or is supposed to > provides equal functionality, yet is implemented differently from Hive CLI. > As it has been a while that Hive community has been recommending Beeline + > HS2 configuration, ideally we should deprecating Hive CLI. Because of wide > use of Hive CLI, we instead propose replacing Hive CLI's implementation with > Beeline plus embedded HS2 so that Hive community only needs to maintain a > single code path. In this way, Hive CLI is just an alias to Beeline at either > shell script level or at high code level. The goal is that no changes or > minimum changes are expected from existing user scrip using Hive CLI. > This is an Umbrella JIRA covering all tasks related to this initiative. Over > the last year or two, Beeline has been improved significantly to match what > Hive CLI offers. Still, there may still be some gaps or deficiency to be > discovered and fixed. In the meantime, we also want to make sure the enough > tests are included and performance impact is identified and addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516148#comment-14516148
]
Hive QA commented on HIVE-9152:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728616/HIVE-9152.4-spark.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8722 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucket6.q-scriptfile1_win.q-quotedid_smb.q-and-1-more - did
not produce a TEST-*.xml file
TestMinimrCliDriver-bucketizedhiveinputformat.q-empty_dir_in_table.q - did not
produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-infer_bucket_sort_map_operators.q-load_hdfs_file_with_space_in_the_name.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-import_exported_table.q-truncate_column_buckets.q-bucket_num_reducers2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-infer_bucket_sort_num_buckets.q-parallel_orderby.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-join1.q-infer_bucket_sort_bucketed_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-bucket5.q-infer_bucket_sort_merge.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-input16_cc.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-bucket_num_reducers.q-scriptfile1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx_cbo_2.q-bucketmapjoin6.q-bucket4.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-reduce_deduplicate.q-infer_bucket_sort_dyn_part.q-udf_using.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-uber_reduce.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-stats_counter_partitioned.q-external_table_with_space_in_location_path.q-disable_merge_for_bucketing.q-and-1-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_dynamic_partition_pruning
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/844/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/844/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-844/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728616 - PreCommit-HIVE-SPARK-Build
> Dynamic Partition Pruning [Spark Branch]
>
>
> Key: HIVE-9152
> URL: https://issues.apache.org/jira/browse/HIVE-9152
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Brock Noland
>Assignee: Chao Sun
> Attachments: HIVE-9152.1-spark.patch, HIVE-9152.2-spark.patch,
> HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch
>
>
> Tez implemented dynamic partition pruning in HIVE-7826. This is a nice
> optimization and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516133#comment-14516133 ] Rui Li commented on HIVE-10476: --- One minor: if the session fails to initialize, we'll get HiveException. Maybe we can use that to make the error log more accurate, e.g. failure to create session or failure to get mem/core info? > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-10476.1-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7150) FileInputStream is not closed in HiveConnection#getHttpClient()
[
https://issues.apache.org/jira/browse/HIVE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516124#comment-14516124
]
Gabor Liptak commented on HIVE-7150:
Is there a way to force a Hive QA build? Thanks
> FileInputStream is not closed in HiveConnection#getHttpClient()
> ---
>
> Key: HIVE-7150
> URL: https://issues.apache.org/jira/browse/HIVE-7150
> Project: Hive
> Issue Type: Bug
>Reporter: Ted Yu
> Labels: jdbc
> Fix For: 1.2.0
>
> Attachments: HIVE-7150.1.patch
>
>
> Here is related code:
> {code}
> sslTrustStore.load(new FileInputStream(sslTrustStorePath),
> sslTrustStorePassword.toCharArray());
> {code}
> The FileInputStream is not closed upon returning from the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10513) [CBO] return path : Fix create_func1.q for return path
[ https://issues.apache.org/jira/browse/HIVE-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10513: Summary: [CBO] return path : Fix create_func1.q for return path (was: Fix create_func1.q for return path) > [CBO] return path : Fix create_func1.q for return path > -- > > Key: HIVE-10513 > URL: https://issues.apache.org/jira/browse/HIVE-10513 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > throws class cast exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10506) CBO (Calcite Return Path): Disallow return path to be enable if CBO is off
[ https://issues.apache.org/jira/browse/HIVE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516118#comment-14516118 ] Laljo John Pullokkaran commented on HIVE-10506: --- [~jcamachorodriguez] I think its not sufficient to look at static configuration (Hive Conf); we also need to look at run time exceptions. I.e we really need to verify if this is CBO optimized plan that is translated to hive ops. Even if CBO is enabled it may not kick in due to unsupported operators and it may fail to optimize/translate (missing col stats, missing feature, bugs...) we need to handle these cases. > CBO (Calcite Return Path): Disallow return path to be enable if CBO is off > -- > > Key: HIVE-10506 > URL: https://issues.apache.org/jira/browse/HIVE-10506 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 1.2.0 > > Attachments: HIVE-10506.patch > > > If hive.cbo.enable=false and hive.cbo.returnpath=true then some optimizations > would kick in. It's quite possible that in customer environment, they might > end up in these scenarios; we should prevent it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9365) The Metastore should take port configuration from hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516114#comment-14516114
]
Reuben Kuhnert commented on HIVE-9365:
--
Seems like only these two tests:
{code}
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
{code}
are actual failures. Looks like it happened in the build before mine as well:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3612/
(Build 3612)
and
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3609/
(3609)
and others. My build is (#3613). Tested these locally, and they seem to work.
> The Metastore should take port configuration from hive-site.xml
> ---
>
> Key: HIVE-9365
> URL: https://issues.apache.org/jira/browse/HIVE-9365
> Project: Hive
> Issue Type: Improvement
>Reporter: Nicolas Thiébaud
>Assignee: Reuben Kuhnert
>Priority: Minor
> Labels: metastore
> Attachments: HIVE-9365.01.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> As opposed to the cli. Having this configuration in the launcher script
> create fragmentation and does is not consistent with the way the hive stack
> is configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10512) CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516116#comment-14516116
]
Ashutosh Chauhan commented on HIVE-10512:
-
or rather change DefaultAnnotate to do above.
> CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
> ---
>
> Key: HIVE-10512
> URL: https://issues.apache.org/jira/browse/HIVE-10512
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10512.patch
>
>
> When return path is on, SMB conversion is throwing an Exception in some cases.
> The problem can be reproduced with auto_join32.q. The Exception with the
> following stacktrace is thrown:
> {noformat}
> java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FilterOperator
> cannot be cast to org.apache.hadoop.hive.ql.exec.TableScanOperator
> at
> org.apache.hadoop.hive.ql.parse.TableAccessAnalyzer.genRootTableScan(TableAccessAnalyzer.java:243)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.checkConvertBucketMapJoin(AbstractBucketJoinProc.java:226)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToBucketMapJoin(AbstractSMBJoinProc.java:497)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToSMBJoin(AbstractSMBJoinProc.java:414)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:45)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:201)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-10512) CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516105#comment-14516105
]
Ashutosh Chauhan edited comment on HIVE-10512 at 4/28/15 1:11 AM:
--
Shall we generalize Filter/Select Annotate to match on any operator and if # of
parent operator = 1, then call propagateAlias() ?
was (Author: ashutoshc):
Shall we generalize Filter/Select Annotate to match on any operator and if # of
parent operator = 1, then call propagateAnnotation() ?
> CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
> ---
>
> Key: HIVE-10512
> URL: https://issues.apache.org/jira/browse/HIVE-10512
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10512.patch
>
>
> When return path is on, SMB conversion is throwing an Exception in some cases.
> The problem can be reproduced with auto_join32.q. The Exception with the
> following stacktrace is thrown:
> {noformat}
> java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FilterOperator
> cannot be cast to org.apache.hadoop.hive.ql.exec.TableScanOperator
> at
> org.apache.hadoop.hive.ql.parse.TableAccessAnalyzer.genRootTableScan(TableAccessAnalyzer.java:243)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.checkConvertBucketMapJoin(AbstractBucketJoinProc.java:226)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToBucketMapJoin(AbstractSMBJoinProc.java:497)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToSMBJoin(AbstractSMBJoinProc.java:414)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:45)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:201)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10512) CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516105#comment-14516105
]
Ashutosh Chauhan commented on HIVE-10512:
-
Shall we generalize Filter/Select Annotate to match on any operator and if # of
parent operator = 1, then call propagateAnnotation() ?
> CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
> ---
>
> Key: HIVE-10512
> URL: https://issues.apache.org/jira/browse/HIVE-10512
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10512.patch
>
>
> When return path is on, SMB conversion is throwing an Exception in some cases.
> The problem can be reproduced with auto_join32.q. The Exception with the
> following stacktrace is thrown:
> {noformat}
> java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FilterOperator
> cannot be cast to org.apache.hadoop.hive.ql.exec.TableScanOperator
> at
> org.apache.hadoop.hive.ql.parse.TableAccessAnalyzer.genRootTableScan(TableAccessAnalyzer.java:243)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.checkConvertBucketMapJoin(AbstractBucketJoinProc.java:226)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToBucketMapJoin(AbstractSMBJoinProc.java:497)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToSMBJoin(AbstractSMBJoinProc.java:414)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:45)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:201)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10512) CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516075#comment-14516075
]
Jesus Camacho Rodriguez commented on HIVE-10512:
cc'ed [~jpullokkaran]
> CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
> ---
>
> Key: HIVE-10512
> URL: https://issues.apache.org/jira/browse/HIVE-10512
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10512.patch
>
>
> When return path is on, SMB conversion is throwing an Exception in some cases.
> The problem can be reproduced with auto_join32.q. The Exception with the
> following stacktrace is thrown:
> {noformat}
> java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FilterOperator
> cannot be cast to org.apache.hadoop.hive.ql.exec.TableScanOperator
> at
> org.apache.hadoop.hive.ql.parse.TableAccessAnalyzer.genRootTableScan(TableAccessAnalyzer.java:243)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.checkConvertBucketMapJoin(AbstractBucketJoinProc.java:226)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToBucketMapJoin(AbstractSMBJoinProc.java:497)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToSMBJoin(AbstractSMBJoinProc.java:414)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:45)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:201)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10512) CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10512:
---
Attachment: HIVE-10512.patch
[~ashutoshc], could you take a look? The problem was that we were storing all
operators in aliasToOpInfo DS in the Join descriptors, leading to orphan
operators when some other optimization would replace them. This patch fixes the
problem by only adding the operators that are actually needed, as it is done by
SemanticAnalyzer. Thanks
> CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException
> ---
>
> Key: HIVE-10512
> URL: https://issues.apache.org/jira/browse/HIVE-10512
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10512.patch
>
>
> When return path is on, SMB conversion is throwing an Exception in some cases.
> The problem can be reproduced with auto_join32.q. The Exception with the
> following stacktrace is thrown:
> {noformat}
> java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FilterOperator
> cannot be cast to org.apache.hadoop.hive.ql.exec.TableScanOperator
> at
> org.apache.hadoop.hive.ql.parse.TableAccessAnalyzer.genRootTableScan(TableAccessAnalyzer.java:243)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.checkConvertBucketMapJoin(AbstractBucketJoinProc.java:226)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToBucketMapJoin(AbstractSMBJoinProc.java:497)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertJoinToSMBJoin(AbstractSMBJoinProc.java:414)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:45)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:201)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1161) Hive Replication
[ https://issues.apache.org/jira/browse/HIVE-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516066#comment-14516066 ] Ashutosh Chauhan commented on HIVE-1161: I think this can be resolved as fix, since feature is largely gonna be available in 1.2. Any future feature request or bug report can be raised in follow-up jiras. > Hive Replication > > > Key: HIVE-1161 > URL: https://issues.apache.org/jira/browse/HIVE-1161 > Project: Hive > Issue Type: New Feature > Components: Contrib >Reporter: Edward Capriolo >Assignee: SHAILESH PILARE >Priority: Minor > > Users may want to replicate data between two distinct hadoop clusters or two > hive warehouses on the same cluster. > Users may want to replicate entire catalogs or possibly on a table by table > basis. Should this process be batch driven or a be a full time running > application? What are some practical requirements, what are the limitations? > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10403) Add n-way join support for Hybrid Grace Hash Join
[
https://issues.apache.org/jira/browse/HIVE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516051#comment-14516051
]
Hive QA commented on HIVE-10403:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728474/HIVE-10403.04.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8819 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3614/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3614/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728474 - PreCommit-HIVE-TRUNK-Build
> Add n-way join support for Hybrid Grace Hash Join
> -
>
> Key: HIVE-10403
> URL: https://issues.apache.org/jira/browse/HIVE-10403
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-10403.01.patch, HIVE-10403.02.patch,
> HIVE-10403.03.patch, HIVE-10403.04.patch
>
>
> Currently Hybrid Grace Hash Join only supports 2-way join (one big table and
> one small table). This task will enable n-way join (one big table and
> multiple small tables).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1161) Hive Replication
[ https://issues.apache.org/jira/browse/HIVE-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516040#comment-14516040 ] Sushanth Sowmyan commented on HIVE-1161: Hi, FYI, most of HIVE-7973 is now committed and will be part of the upcoming hive 1.2 release. [~ashutoshc], any opinions on what should be done with this jira? > Hive Replication > > > Key: HIVE-1161 > URL: https://issues.apache.org/jira/browse/HIVE-1161 > Project: Hive > Issue Type: New Feature > Components: Contrib >Reporter: Edward Capriolo >Assignee: SHAILESH PILARE >Priority: Minor > > Users may want to replicate data between two distinct hadoop clusters or two > hive warehouses on the same cluster. > Users may want to replicate entire catalogs or possibly on a table by table > basis. Should this process be batch driven or a be a full time running > application? What are some practical requirements, what are the limitations? > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10511) Replacing the implementation of Hive CLI using Beeline
[ https://issues.apache.org/jira/browse/HIVE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10511: --- Summary: Replacing the implementation of Hive CLI using Beeline (was: Unify Hive CLI and Beeline) > Replacing the implementation of Hive CLI using Beeline > -- > > Key: HIVE-10511 > URL: https://issues.apache.org/jira/browse/HIVE-10511 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Xuefu Zhang > > Hive CLI is a legacy tool which had two main use cases: > 1. a thick client for SQL on hadoop > 2. a command line tool for HiveServer1. > HiveServer1 is already deprecated and removed from Hive code base, so use > case #2 is out of the question. For #1, Beeline provides or is supposed to > provides equal functionality, yet is implemented differently from Hive CLI. > As it has been a while that Hive community has been recommending Beeline + > HS2 configuration, ideally we should deprecating Hive CLI. Because of wide > use of Hive CLI, we instead propose replacing Hive CLI's implementation with > Beeline plus embedded HS2 so that Hive community only needs to maintain a > single code path. In this way, Hive CLI is just an alias to Beeline at either > shell script level or at high code level. The goal is that no changes or > minimum changes are expected from existing user scrip using Hive CLI. > This is an Umbrella JIRA covering all tasks related to this initiative. Over > the last year or two, Beeline has been improved significantly to match what > Hive CLI offers. Still, there may still be some gaps or deficiency to be > discovered and fixed. In the meantime, we also want to make sure the enough > tests are included and performance impact is identified and addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-9152: --- Attachment: HIVE-9152.4-spark.patch > Dynamic Partition Pruning [Spark Branch] > > > Key: HIVE-9152 > URL: https://issues.apache.org/jira/browse/HIVE-9152 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Brock Noland >Assignee: Chao Sun > Attachments: HIVE-9152.1-spark.patch, HIVE-9152.2-spark.patch, > HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch > > > Tez implemented dynamic partition pruning in HIVE-7826. This is a nice > optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9451) Add max size of column dictionaries to ORC metadata
[ https://issues.apache.org/jira/browse/HIVE-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515880#comment-14515880 ] Prasanth Jayachandran commented on HIVE-9451: - Patch looks good to me. +1. Pending tests. ORC concatenation is not affected by this patch :). Minor nit: LOG.info instead of sysout in ColumnStatisticsImpl and WriterImpl? > Add max size of column dictionaries to ORC metadata > --- > > Key: HIVE-9451 > URL: https://issues.apache.org/jira/browse/HIVE-9451 > Project: Hive > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 1.2.0 > > Attachments: HIVE-9451.patch > > > To predict the amount of memory required to read an ORC file we need to know > the size of the dictionaries for the columns that we are reading. I propose > adding the number of bytes for each column's dictionary to the stripe's > column statistics. The file's column statistics would have the maximum > dictionary size for each column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10510) Change 1.2.0-SNAPSHOT to 1.2.0 in branch-1.2
[ https://issues.apache.org/jira/browse/HIVE-10510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan resolved HIVE-10510. - Resolution: Fixed Fix Version/s: 1.2.0 > Change 1.2.0-SNAPSHOT to 1.2.0 in branch-1.2 > > > Key: HIVE-10510 > URL: https://issues.apache.org/jira/browse/HIVE-10510 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 1.2.0 > > Attachments: HIVE-10510.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10510) Change 1.2.0-SNAPSHOT to 1.2.0 in branch-1.2
[ https://issues.apache.org/jira/browse/HIVE-10510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10510: Attachment: HIVE-10510.patch > Change 1.2.0-SNAPSHOT to 1.2.0 in branch-1.2 > > > Key: HIVE-10510 > URL: https://issues.apache.org/jira/browse/HIVE-10510 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-10510.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10509) Bump trunk version to 1.3 as branch-1.2 has been created.
[ https://issues.apache.org/jira/browse/HIVE-10509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan resolved HIVE-10509. - Resolution: Fixed Fix Version/s: 1.3.0 > Bump trunk version to 1.3 as branch-1.2 has been created. > - > > Key: HIVE-10509 > URL: https://issues.apache.org/jira/browse/HIVE-10509 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 1.3.0 > > Attachments: HIVE-10509.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10509) Bump trunk version to 1.3 as branch-1.2 has been created.
[ https://issues.apache.org/jira/browse/HIVE-10509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-10509: Attachment: HIVE-10509.patch Patch attached to capture the update from 1.2 to 1.3 > Bump trunk version to 1.3 as branch-1.2 has been created. > - > > Key: HIVE-10509 > URL: https://issues.apache.org/jira/browse/HIVE-10509 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 1.3.0 > > Attachments: HIVE-10509.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10507) Expose RetryingMetastoreClient to other external users of metastore client like Flume and Storm.
[ https://issues.apache.org/jira/browse/HIVE-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10507: - Attachment: HIVE-10507.1.patch cc-ing [~thejas] to review. Thanks Hari > Expose RetryingMetastoreClient to other external users of metastore client > like Flume and Storm. > - > > Key: HIVE-10507 > URL: https://issues.apache.org/jira/browse/HIVE-10507 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10507.1.patch > > > HiveMetastoreClient is now being relied upon by external clients like Flume > and Storm for streaming. > When the thrift connection between MetaStoreClient and the meta store is > broken (due to intermittent network issues or restarting of metastore) the > Metastore does not handle the connection error and automatically re-establish > the connection. Currently the client process needs to be restarted to > re-establish the connection. > The request here is consider supporting the following behavior: For each API > invocation on the MetastoreClient, it should try to restablish the connection > (if needed) once. And if that does not work out then throw a specific > exception indicating the same. The client could then handle the issue by > retrying the same API after some delay. By catching the specific connection > exception, the client could decide how many times to retry before aborting. > Hive does this internally using RetryingMetastoreClient. This jira is suppose > to expose this mechanism to other users of that interface. This is useful for > users of this interface, and from metastore HA point of view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10071) CBO (Calcite Return Path): Join to MultiJoin rule
[ https://issues.apache.org/jira/browse/HIVE-10071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10071: --- Summary: CBO (Calcite Return Path): Join to MultiJoin rule (was: CBO (Calcite Return Path): Join to MultiJoin rule [CBO Branch]) > CBO (Calcite Return Path): Join to MultiJoin rule > - > > Key: HIVE-10071 > URL: https://issues.apache.org/jira/browse/HIVE-10071 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: cbo-branch >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: cbo-branch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10493) Merge multiple joins when join keys are the same
[ https://issues.apache.org/jira/browse/HIVE-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-10493. Resolution: Duplicate > Merge multiple joins when join keys are the same > > > Key: HIVE-10493 > URL: https://issues.apache.org/jira/browse/HIVE-10493 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 1.2.0 > > > CBO return path: auto_join3.q is joined on the same key from 3 sources. It is > translated into 2 map joins. Need to merge them into a single one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9365) The Metastore should take port configuration from hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515829#comment-14515829
]
Hive QA commented on HIVE-9365:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728472/HIVE-9365.01.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8818 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3613/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3613/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3613/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728472 - PreCommit-HIVE-TRUNK-Build
> The Metastore should take port configuration from hive-site.xml
> ---
>
> Key: HIVE-9365
> URL: https://issues.apache.org/jira/browse/HIVE-9365
> Project: Hive
> Issue Type: Improvement
>Reporter: Nicolas Thiébaud
>Assignee: Reuben Kuhnert
>Priority: Minor
> Labels: metastore
> Attachments: HIVE-9365.01.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> As opposed to the cli. Having this configuration in the launcher script
> create fragmentation and does is not consistent with the way the hive stack
> is configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez resolved HIVE-10462.
Resolution: Fixed
We follow up the issue raised by [~jpullokkaran] in HIVE-10506.
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10506) CBO (Calcite Return Path): Disallow return path to be enable if CBO is off
[ https://issues.apache.org/jira/browse/HIVE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10506: --- Description: If hive.cbo.enable=false and hive.co.returnpath=true then some optimizations would kick in. It's quite possible that in customer environment, they might end up in these scenarios; we should prevent it. > CBO (Calcite Return Path): Disallow return path to be enable if CBO is off > -- > > Key: HIVE-10506 > URL: https://issues.apache.org/jira/browse/HIVE-10506 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 1.2.0 > > Attachments: HIVE-10506.patch > > > If hive.cbo.enable=false and hive.co.returnpath=true then some optimizations > would kick in. It's quite possible that in customer environment, they might > end up in these scenarios; we should prevent it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10506) CBO (Calcite Return Path): Disallow return path to be enable if CBO is off
[ https://issues.apache.org/jira/browse/HIVE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10506: --- Description: If hive.cbo.enable=false and hive.cbo.returnpath=true then some optimizations would kick in. It's quite possible that in customer environment, they might end up in these scenarios; we should prevent it. (was: If hive.cbo.enable=false and hive.co.returnpath=true then some optimizations would kick in. It's quite possible that in customer environment, they might end up in these scenarios; we should prevent it.) > CBO (Calcite Return Path): Disallow return path to be enable if CBO is off > -- > > Key: HIVE-10506 > URL: https://issues.apache.org/jira/browse/HIVE-10506 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 1.2.0 > > Attachments: HIVE-10506.patch > > > If hive.cbo.enable=false and hive.cbo.returnpath=true then some optimizations > would kick in. It's quite possible that in customer environment, they might > end up in these scenarios; we should prevent it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10372) Update parquet version to 1.6.0
[ https://issues.apache.org/jira/browse/HIVE-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515812#comment-14515812 ] Sushanth Sowmyan commented on HIVE-10372: - Cherry-picked into 1.2. Thanks Ferdinand, Xuefu! > Update parquet version to 1.6.0 > --- > > Key: HIVE-10372 > URL: https://issues.apache.org/jira/browse/HIVE-10372 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: 1.2.0 > > Attachments: HIVE-10372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10372) Update parquet version to 1.6.0
[ https://issues.apache.org/jira/browse/HIVE-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515808#comment-14515808 ] Sushanth Sowmyan commented on HIVE-10372: - Weird. I do see it now. Not sure why - I did see patches after yours in there when I last looked. Sorry, I must have misread. :) I will pull it into branch-1.2. Thanks! > Update parquet version to 1.6.0 > --- > > Key: HIVE-10372 > URL: https://issues.apache.org/jira/browse/HIVE-10372 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: 1.2.0 > > Attachments: HIVE-10372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10506) CBO (Calcite Return Path): Disallow return path to be enable if CBO is off
[ https://issues.apache.org/jira/browse/HIVE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10506: --- Attachment: HIVE-10506.patch [~jpullokkaran], could you take a look? This solves the issue raised in HIVE-10462. Thanks > CBO (Calcite Return Path): Disallow return path to be enable if CBO is off > -- > > Key: HIVE-10506 > URL: https://issues.apache.org/jira/browse/HIVE-10506 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 1.2.0 > > Attachments: HIVE-10506.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10504) ORC date column statistics should return primitive object instead of writable
[ https://issues.apache.org/jira/browse/HIVE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-10504: - Attachment: HIVE-10504.1.patch > ORC date column statistics should return primitive object instead of writable > - > > Key: HIVE-10504 > URL: https://issues.apache.org/jira/browse/HIVE-10504 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-10504.1.patch > > > Date column statistics is inconsistent with other column statistics. It > returns DateWritable as opposed to primitive variant Date. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10372) Update parquet version to 1.6.0
[
https://issues.apache.org/jira/browse/HIVE-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515170#comment-14515170
]
Xuefu Zhang commented on HIVE-10372:
Strange. I do see this (across machines):
{code}
commit e11450cf323f8a2eae5d9feb56562532e2e5f6d7
Author: Xuefu Zhang
Date: Mon Apr 27 06:35:44 2015 -0700
HIVE-10372: Update parquet version to 1.6.0 (Ferdinand via Xuefu)
{code}
Could you double check?
> Update parquet version to 1.6.0
> ---
>
> Key: HIVE-10372
> URL: https://issues.apache.org/jira/browse/HIVE-10372
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: 1.2.0
>
> Attachments: HIVE-10372.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10307) Support to use number literals in partition column
[ https://issues.apache.org/jira/browse/HIVE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515161#comment-14515161 ] Jimmy Xiang commented on HIVE-10307: +1 > Support to use number literals in partition column > -- > > Key: HIVE-10307 > URL: https://issues.apache.org/jira/browse/HIVE-10307 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10307.1.patch, HIVE-10307.2.patch, > HIVE-10307.3.patch, HIVE-10307.4.patch, HIVE-10307.5.patch, > HIVE-10307.6.patch, HIVE-10307.patch > > > Data types like TinyInt, SmallInt, BigInt or Decimal can be expressed as > literals with postfix like Y, S, L, or BD appended to the number. These > literals work in most Hive queries, but do not when they are used as > partition column value. For a partitioned table like: > create table partcoltypenum (key int, value string) partitioned by (tint > tinyint, sint smallint, bint bigint); > insert into partcoltypenum partition (tint=100Y, sint=1S, > bint=1000L) select key, value from src limit 30; > Queries like select, describe and drop partition do not work. For an example > select * from partcoltypenum where tint=100Y and sint=1S and > bint=1000L; > does not return any rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515154#comment-14515154
]
Jesus Camacho Rodriguez commented on HIVE-10462:
Sure, you are right, I'll open a new JIRA case and submit a fix for that. Thanks
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515151#comment-14515151
]
Laljo John Pullokkaran commented on HIVE-10462:
---
[~jcamachorodriguez] If hive.cbo.enable is false and hive.co.returnpath=true
then this optimization would kick in. Its quiet possible that in customer env
they might end up in these scenarios. Its an easy fix why not prevent it?
Also Trivial ProjectRemoval rule we disable it if return path is on; there also
we need to check if cbo is enabled.
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515144#comment-14515144
]
Jesus Camacho Rodriguez commented on HIVE-10462:
[~jpullokkaran], as far as I have seen, we never double check for CBO and
return path; that's why I only check if the return path is on. For join
translation based on algorithm, we haven't commit it yet. Or am I missing
something? When return path is on, the new transformation rule is applied.
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laljo John Pullokkaran reopened HIVE-10462:
---
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515139#comment-14515139
]
Laljo John Pullokkaran commented on HIVE-10462:
---
[~jcamachorodriguez] This transformation should be applied only if CBO is
enabled, CBO Return Path is enabled, & We haven't performed join algo
selection/translation. As the code exists it doesn't seems to be doing it.
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10450) More than one TableScan in MapWork not supported in Vectorization -- causes query to fail during vectorization
[
https://issues.apache.org/jira/browse/HIVE-10450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515132#comment-14515132
]
Matt McCline commented on HIVE-10450:
-
[~gopalv] thank you for reviewing and helping solve the problem.
Note the problem can also manifest in MR as a NullPointerException with this
stack trace (when tables common column types but with different number of
columns):
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.setBatch(VectorExtractRow.java:705)
at
org.apache.hadoop.hive.ql.exec.vector.VectorExtractRowDynBatch.setBatchOnEntry(VectorExtractRowDynBatch.java:34)
at
org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:89)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
... 10 more
{noformat}
> More than one TableScan in MapWork not supported in Vectorization -- causes
> query to fail during vectorization
> ---
>
> Key: HIVE-10450
> URL: https://issues.apache.org/jira/browse/HIVE-10450
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-10450.01.patch, HIVE-10450.01.patch,
> HIVE-10450.02.patch, HIVE-10450.03.patch, HIVE-10450.04.patch
>
>
> [~gopalv] found a error with this query:
> {noformat}
> explain select
> s_state, count(1)
> from store_sales,
> store,
> date_dim
> where store_sales.ss_sold_date_sk = date_dim.d_date_sk and
>store_sales.ss_store_sk = store.s_store_sk and
>store.s_state in ('KS','AL', 'MN', 'AL', 'SC', 'VT')
> group by s_state
> order by s_state
> limit 100;
> {noformat}
> Stack trace:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:676)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:735)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:59)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:422)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:354)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:322)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:877)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at org.apache.hadoop.
[jira] [Commented] (HIVE-10441) Fix confusing log statement in SessionState about hive.execution.engine setting
[
https://issues.apache.org/jira/browse/HIVE-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515115#comment-14515115
]
Jason Dere commented on HIVE-10441:
---
committed to branch-1.2
> Fix confusing log statement in SessionState about hive.execution.engine
> setting
> ---
>
> Key: HIVE-10441
> URL: https://issues.apache.org/jira/browse/HIVE-10441
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 1.2.0
>
> Attachments: HIVE-10441.1.patch
>
>
> {code}
> LOG.info("No Tez session required at this point. hive.execution.engine=mr.");
> {code}
> This statement is misleading. It is true that it is printed in the case that
> Tez session does not need to be created, but it is not necessarily true that
> hive.execution.engine=mr - it could be Spark, or it could even be set to Tez
> but the Session determined that a Tez Session did not need to be created
> (which is the case for HiveServer2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10441) Fix confusing log statement in SessionState about hive.execution.engine setting
[
https://issues.apache.org/jira/browse/HIVE-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10441:
--
Fix Version/s: (was: 1.3.0)
1.2.0
> Fix confusing log statement in SessionState about hive.execution.engine
> setting
> ---
>
> Key: HIVE-10441
> URL: https://issues.apache.org/jira/browse/HIVE-10441
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 1.2.0
>
> Attachments: HIVE-10441.1.patch
>
>
> {code}
> LOG.info("No Tez session required at this point. hive.execution.engine=mr.");
> {code}
> This statement is misleading. It is true that it is printed in the case that
> Tez session does not need to be created, but it is not necessarily true that
> hive.execution.engine=mr - it could be Spark, or it could even be set to Tez
> but the Session determined that a Tez Session did not need to be created
> (which is the case for HiveServer2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10477) Provide option to disable Spark tests
[
https://issues.apache.org/jira/browse/HIVE-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515099#comment-14515099
]
Sushanth Sowmyan commented on HIVE-10477:
-
+1, will go ahead and commit to master and branch-1.2.
> Provide option to disable Spark tests
> --
>
> Key: HIVE-10477
> URL: https://issues.apache.org/jira/browse/HIVE-10477
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10477.01.patch
>
>
> The following is one of the reasons why we might want to provide an option to
> disable spark tests :
> In the current master branch, unit tests fail with windows OS because of the
> dependency on "bash" executable in itests/hive-unit/pom.xml around these
> lines :
> {code}
>
> failonerror="true">
>
>
>
> {code}
> We should provide an option to disable spark tests in OSes like Windows
> where bash might be absent. That being mentioned, spark tests will be enabled
> by default in pre-commit test runs and should still continue to work as it is
> in the master branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10421) DROP TABLE with qualified table name ignores database name when checking partitions
[ https://issues.apache.org/jira/browse/HIVE-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-10421: -- Fix Version/s: (was: 1.3.0) 1.2.0 > DROP TABLE with qualified table name ignores database name when checking > partitions > --- > > Key: HIVE-10421 > URL: https://issues.apache.org/jira/browse/HIVE-10421 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 1.2.0 > > Attachments: HIVE-10421.1.patch, HIVE-10421.2.patch > > > Hive was only recently changed to allow "drop table dbname.tabname". However > DDLTask.dropTable() is still using an older version of > Hive.getPartitionNames(), which only took in a single string for the table > name, rather than the database and table names. As a result Hive is filling > in the current database name as the dbname during the listPartitions call to > the MetaStore. > It also appears that on the Hive Metastore side, in the non-auth path there > is no validation to check that the dbname.tablename actually exists - this > call simply returns back an empty list of partitions, which causes the table > to be dropped without checking any of the partition information. I will open > a separate issue for this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10421) DROP TABLE with qualified table name ignores database name when checking partitions
[ https://issues.apache.org/jira/browse/HIVE-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515090#comment-14515090 ] Jason Dere commented on HIVE-10421: --- Committed to 1.2 > DROP TABLE with qualified table name ignores database name when checking > partitions > --- > > Key: HIVE-10421 > URL: https://issues.apache.org/jira/browse/HIVE-10421 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 1.2.0 > > Attachments: HIVE-10421.1.patch, HIVE-10421.2.patch > > > Hive was only recently changed to allow "drop table dbname.tabname". However > DDLTask.dropTable() is still using an older version of > Hive.getPartitionNames(), which only took in a single string for the table > name, rather than the database and table names. As a result Hive is filling > in the current database name as the dbname during the listPartitions call to > the MetaStore. > It also appears that on the Hive Metastore side, in the non-auth path there > is no validation to check that the dbname.tablename actually exists - this > call simply returns back an empty list of partitions, which causes the table > to be dropped without checking any of the partition information. I will open > a separate issue for this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10499) Ensure Session/ZooKeeperClient instances are closed
[
https://issues.apache.org/jira/browse/HIVE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515076#comment-14515076
]
Hive QA commented on HIVE-10499:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728459/HIVE-10499.patch
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 8813 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
TestSchedulerQueue - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3612/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3612/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728459 - PreCommit-HIVE-TRUNK-Build
> Ensure Session/ZooKeeperClient instances are closed
> ---
>
> Key: HIVE-10499
> URL: https://issues.apache.org/jira/browse/HIVE-10499
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-10499.patch
>
>
> Some Session/ZooKeeperClient instances are not closed in some scenario. We
> need to make sure they are always closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10428) NPE in RegexSerDe using HCat
[
https://issues.apache.org/jira/browse/HIVE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515071#comment-14515071
]
Sushanth Sowmyan commented on HIVE-10428:
-
+1 for inclusion to 1.2.
Added to
https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
> NPE in RegexSerDe using HCat
>
>
> Key: HIVE-10428
> URL: https://issues.apache.org/jira/browse/HIVE-10428
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-10428.1.patch
>
>
> When HCatalog calls to table with "org.apache.hadoop.hive.serde2.RegexSerDe",
> when doing Hcatalog call to get read the table, it throws exception:
> {noformat}
> 15/04/21 14:07:31 INFO security.TokenCache: Got dt for hdfs://hdpsecahdfs;
> Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpsecahdfs, Ident:
> (HDFS_DELEGATION_TOKEN token 1478 for haha)
> 15/04/21 14:07:31 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> Splits len : 1
> SplitInfo : [hdpseca03.seca.hwxsup.com, hdpseca04.seca.hwxsup.com,
> hdpseca05.seca.hwxsup.com]
> 15/04/21 14:07:31 INFO mapreduce.InternalUtil: Initializing
> org.apache.hadoop.hive.serde2.RegexSerDe with properties
> {name=casetest.regex_table, numFiles=1, columns.types=string,string,
> serialization.format=1, columns=id,name, rawDataSize=0, numRows=0,
> output.format.string=%1$s %2$s,
> serialization.lib=org.apache.hadoop.hive.serde2.RegexSerDe,
> COLUMN_STATS_ACCURATE=true, totalSize=25, serialization.null.format=\N,
> input.regex=([^ ]*) ([^ ]*), transient_lastDdlTime=1429590172}
> 15/04/21 14:07:31 WARN serde2.RegexSerDe: output.format.string has been
> deprecated
> Exception in thread "main" java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Splitter.split(Splitter.java:371)
> at
> org.apache.hadoop.hive.serde2.RegexSerDe.initialize(RegexSerDe.java:155)
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:49)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:518)
> at
> org.apache.hive.hcatalog.mapreduce.InternalUtil.initializeDeserializer(InternalUtil.java:156)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createDeserializer(HCatRecordReader.java:127)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:92)
> at HCatalogSQLMR.main(HCatalogSQLMR.java:81)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10372) Update parquet version to 1.6.0
[ https://issues.apache.org/jira/browse/HIVE-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515068#comment-14515068 ] Sushanth Sowmyan commented on HIVE-10372: - Approved for branch-1.2. Xuefu, I do not see this commit on master. Could you please verify? > Update parquet version to 1.6.0 > --- > > Key: HIVE-10372 > URL: https://issues.apache.org/jira/browse/HIVE-10372 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: 1.2.0 > > Attachments: HIVE-10372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10428) NPE in RegexSerDe using HCat
[
https://issues.apache.org/jira/browse/HIVE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515067#comment-14515067
]
Jason Dere commented on HIVE-10428:
---
[~sushanth] another one I'd like to get in for 1.2
> NPE in RegexSerDe using HCat
>
>
> Key: HIVE-10428
> URL: https://issues.apache.org/jira/browse/HIVE-10428
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-10428.1.patch
>
>
> When HCatalog calls to table with "org.apache.hadoop.hive.serde2.RegexSerDe",
> when doing Hcatalog call to get read the table, it throws exception:
> {noformat}
> 15/04/21 14:07:31 INFO security.TokenCache: Got dt for hdfs://hdpsecahdfs;
> Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpsecahdfs, Ident:
> (HDFS_DELEGATION_TOKEN token 1478 for haha)
> 15/04/21 14:07:31 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> Splits len : 1
> SplitInfo : [hdpseca03.seca.hwxsup.com, hdpseca04.seca.hwxsup.com,
> hdpseca05.seca.hwxsup.com]
> 15/04/21 14:07:31 INFO mapreduce.InternalUtil: Initializing
> org.apache.hadoop.hive.serde2.RegexSerDe with properties
> {name=casetest.regex_table, numFiles=1, columns.types=string,string,
> serialization.format=1, columns=id,name, rawDataSize=0, numRows=0,
> output.format.string=%1$s %2$s,
> serialization.lib=org.apache.hadoop.hive.serde2.RegexSerDe,
> COLUMN_STATS_ACCURATE=true, totalSize=25, serialization.null.format=\N,
> input.regex=([^ ]*) ([^ ]*), transient_lastDdlTime=1429590172}
> 15/04/21 14:07:31 WARN serde2.RegexSerDe: output.format.string has been
> deprecated
> Exception in thread "main" java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Splitter.split(Splitter.java:371)
> at
> org.apache.hadoop.hive.serde2.RegexSerDe.initialize(RegexSerDe.java:155)
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:49)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:518)
> at
> org.apache.hive.hcatalog.mapreduce.InternalUtil.initializeDeserializer(InternalUtil.java:156)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createDeserializer(HCatRecordReader.java:127)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:92)
> at HCatalogSQLMR.main(HCatalogSQLMR.java:81)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10428) NPE in RegexSerDe using HCat
[
https://issues.apache.org/jira/browse/HIVE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515022#comment-14515022
]
Ashutosh Chauhan commented on HIVE-10428:
-
+1
> NPE in RegexSerDe using HCat
>
>
> Key: HIVE-10428
> URL: https://issues.apache.org/jira/browse/HIVE-10428
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-10428.1.patch
>
>
> When HCatalog calls to table with "org.apache.hadoop.hive.serde2.RegexSerDe",
> when doing Hcatalog call to get read the table, it throws exception:
> {noformat}
> 15/04/21 14:07:31 INFO security.TokenCache: Got dt for hdfs://hdpsecahdfs;
> Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpsecahdfs, Ident:
> (HDFS_DELEGATION_TOKEN token 1478 for haha)
> 15/04/21 14:07:31 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> Splits len : 1
> SplitInfo : [hdpseca03.seca.hwxsup.com, hdpseca04.seca.hwxsup.com,
> hdpseca05.seca.hwxsup.com]
> 15/04/21 14:07:31 INFO mapreduce.InternalUtil: Initializing
> org.apache.hadoop.hive.serde2.RegexSerDe with properties
> {name=casetest.regex_table, numFiles=1, columns.types=string,string,
> serialization.format=1, columns=id,name, rawDataSize=0, numRows=0,
> output.format.string=%1$s %2$s,
> serialization.lib=org.apache.hadoop.hive.serde2.RegexSerDe,
> COLUMN_STATS_ACCURATE=true, totalSize=25, serialization.null.format=\N,
> input.regex=([^ ]*) ([^ ]*), transient_lastDdlTime=1429590172}
> 15/04/21 14:07:31 WARN serde2.RegexSerDe: output.format.string has been
> deprecated
> Exception in thread "main" java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Splitter.split(Splitter.java:371)
> at
> org.apache.hadoop.hive.serde2.RegexSerDe.initialize(RegexSerDe.java:155)
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:49)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:518)
> at
> org.apache.hive.hcatalog.mapreduce.InternalUtil.initializeDeserializer(InternalUtil.java:156)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createDeserializer(HCatRecordReader.java:127)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:92)
> at HCatalogSQLMR.main(HCatalogSQLMR.java:81)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10307) Support to use number literals in partition column
[ https://issues.apache.org/jira/browse/HIVE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10307: --- Attachment: HIVE-10307.6.patch [~jxiang] Updated patch based on your comments. Also fixed two more test failures after rebase. > Support to use number literals in partition column > -- > > Key: HIVE-10307 > URL: https://issues.apache.org/jira/browse/HIVE-10307 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10307.1.patch, HIVE-10307.2.patch, > HIVE-10307.3.patch, HIVE-10307.4.patch, HIVE-10307.5.patch, > HIVE-10307.6.patch, HIVE-10307.patch > > > Data types like TinyInt, SmallInt, BigInt or Decimal can be expressed as > literals with postfix like Y, S, L, or BD appended to the number. These > literals work in most Hive queries, but do not when they are used as > partition column value. For a partitioned table like: > create table partcoltypenum (key int, value string) partitioned by (tint > tinyint, sint smallint, bint bigint); > insert into partcoltypenum partition (tint=100Y, sint=1S, > bint=1000L) select key, value from src limit 30; > Queries like select, describe and drop partition do not work. For an example > select * from partcoltypenum where tint=100Y and sint=1S and > bint=1000L; > does not return any rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10451) PTF deserializer fails if values are not used in reducer
[ https://issues.apache.org/jira/browse/HIVE-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514988#comment-14514988 ] Ashutosh Chauhan commented on HIVE-10451: - Committed to 1.2 > PTF deserializer fails if values are not used in reducer > -- > > Key: HIVE-10451 > URL: https://issues.apache.org/jira/browse/HIVE-10451 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing, Query Processor >Affects Versions: 0.14.0, 1.0.0, 1.1.0 >Reporter: Gopal V >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10451.1.patch, HIVE-10451.1.patch, HIVE-10451.patch > > > In this particular case no values are needed from reducer to complete > processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10451) PTF deserializer fails if values are not used in reducer
[ https://issues.apache.org/jira/browse/HIVE-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10451: Component/s: (was: Logical Optimizer) Query Processor > PTF deserializer fails if values are not used in reducer > -- > > Key: HIVE-10451 > URL: https://issues.apache.org/jira/browse/HIVE-10451 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing, Query Processor >Affects Versions: 0.14.0, 1.0.0, 1.1.0 >Reporter: Gopal V >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10451.1.patch, HIVE-10451.1.patch, HIVE-10451.patch > > > In this particular case no values are needed from reducer to complete > processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10451) PTF deserializer fails if values are not used in reducer
[ https://issues.apache.org/jira/browse/HIVE-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10451: Component/s: PTF-Windowing > PTF deserializer fails if values are not used in reducer > -- > > Key: HIVE-10451 > URL: https://issues.apache.org/jira/browse/HIVE-10451 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing, Query Processor >Affects Versions: 0.14.0, 1.0.0, 1.1.0 >Reporter: Gopal V >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10451.1.patch, HIVE-10451.1.patch, HIVE-10451.patch > > > In this particular case no values are needed from reducer to complete > processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10462:
Fix Version/s: (was: 1.3.0)
1.2.0
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10451) PTF deserializer fails if values are not used in reducer
[ https://issues.apache.org/jira/browse/HIVE-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10451: Fix Version/s: (was: 1.3.0) 1.2.0 > PTF deserializer fails if values are not used in reducer > -- > > Key: HIVE-10451 > URL: https://issues.apache.org/jira/browse/HIVE-10451 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 0.14.0, 1.0.0, 1.1.0 >Reporter: Gopal V >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10451.1.patch, HIVE-10451.1.patch, HIVE-10451.patch > > > In this particular case no values are needed from reducer to complete > processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10462) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514987#comment-14514987
]
Ashutosh Chauhan commented on HIVE-10462:
-
Committed to 1.2
> CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
> ---
>
> Key: HIVE-10462
> URL: https://issues.apache.org/jira/browse/HIVE-10462
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.0
>
> Attachments: HIVE-10462.01.patch, HIVE-10462.02.patch,
> HIVE-10462.03.patch, HIVE-10462.patch
>
>
> When the return path is on, the mapjoin conversion optimization fails as some
> DS in the Join descriptor have not been initialized properly.
> The failure can be reproduced with auto_join4.q. In particular, the following
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: null
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:203)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
> ...
> {noformat}
> SMBJoin conversion is not triggered either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10490) HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry
[ https://issues.apache.org/jira/browse/HIVE-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10490: Fix Version/s: (was: 1.3.0) 1.2.0 > HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry > -- > > Key: HIVE-10490 > URL: https://issues.apache.org/jira/browse/HIVE-10490 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 1.2.0 >Reporter: Jagruti Varia >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10490.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10490) HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry
[ https://issues.apache.org/jira/browse/HIVE-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514985#comment-14514985 ] Ashutosh Chauhan commented on HIVE-10490: - Committed to 1.2 > HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry > -- > > Key: HIVE-10490 > URL: https://issues.apache.org/jira/browse/HIVE-10490 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 1.2.0 >Reporter: Jagruti Varia >Assignee: Ashutosh Chauhan > Fix For: 1.2.0 > > Attachments: HIVE-10490.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10441) Fix confusing log statement in SessionState about hive.execution.engine setting
[
https://issues.apache.org/jira/browse/HIVE-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514978#comment-14514978
]
Sushanth Sowmyan commented on HIVE-10441:
-
Approved for branch-1.2. Added to
https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
Please feel free to commit to branch-1.2.
> Fix confusing log statement in SessionState about hive.execution.engine
> setting
> ---
>
> Key: HIVE-10441
> URL: https://issues.apache.org/jira/browse/HIVE-10441
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 1.3.0
>
> Attachments: HIVE-10441.1.patch
>
>
> {code}
> LOG.info("No Tez session required at this point. hive.execution.engine=mr.");
> {code}
> This statement is misleading. It is true that it is printed in the case that
> Tez session does not need to be created, but it is not necessarily true that
> hive.execution.engine=mr - it could be Spark, or it could even be set to Tez
> but the Session determined that a Tez Session did not need to be created
> (which is the case for HiveServer2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9508) MetaStore client socket connection should have a lifetime
[ https://issues.apache.org/jira/browse/HIVE-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514965#comment-14514965 ] Sushanth Sowmyan commented on HIVE-9508: Approved to port to branch-1.2 pending rebase that works appropriately in branch-1.2. Added to https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status > MetaStore client socket connection should have a lifetime > - > > Key: HIVE-9508 > URL: https://issues.apache.org/jira/browse/HIVE-9508 > Project: Hive > Issue Type: Sub-task > Components: CLI, Metastore >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9508.1.patch, HIVE-9508.2.patch, HIVE-9508.3.patch > > > Currently HiveMetaStoreClient (or SessionHMSC) is connected to one Metastore > server until the connection is closed or there is a problem. I would like to > introduce the concept of a MetaStore client socket life time. The MS client > will reconnect if the socket lifetime is reached. This will help during > rolling upgrade of Metastore. > When there are multiple Metastore servers behind a VIP (load balancer), it is > easy to take one server out of rotation and wait for 10+ mins for all > existing connections will die down (if the lifetime is 5mins say) and the > server can be updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10437) NullPointerException on queries where map/reduce is not involved on tables with partitions
[
https://issues.apache.org/jira/browse/HIVE-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10437:
Component/s: Serializers/Deserializers
> NullPointerException on queries where map/reduce is not involved on tables
> with partitions
> --
>
> Key: HIVE-10437
> URL: https://issues.apache.org/jira/browse/HIVE-10437
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Demeter Sztanko
>Assignee: Ashutosh Chauhan
>Priority: Critical
> Attachments: HIVE-10437.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> On a table with partitions, whenever I try to do a simple query which tells
> hive not to execute mapreduce but just read data straight from hdfs, it
> raises an exception:
> {code}
> create external table jsonbug(
> a int,
> b string
> )
> PARTITIONED BY (
> `c` string)
> ROW FORMAT SERDE
> 'org.openx.data.jsonserde.JsonSerDe'
> WITH SERDEPROPERTIES (
> 'ignore.malformed.json'='true')
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> '/tmp/jsonbug';
> ALTER TABLE jsonbug ADD PARTITION(c='1');
> {code}
> Runnin simple
> {code}
> select * from jsonbug;
> {code}
> Raises the following exception:
> {code}
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception nulljava.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.needConversion(FetchOperator.java:607)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:578)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:455)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> {code}
> It works fine if I execute a query involving map/reduce job though.
> This problem occurs only when using SerDe's created for hive versions pre
> 1.1.0, those which do not have @SerDeSpec annotation specified. Most of the
> third party SerDE's, including hcat's JsonSerde have this problem as well.
> It seems like changes made in HIVE-7977 introduce this bug. See
> org.apache.hadoop.hive.ql.exec.FetchOperator.needConversion(FetchOperator.java:607)
> {code}
> Class tableSerDe = tableDesc.getDeserializerClass();
> String[] schemaProps = AnnotationUtils.getAnnotation(tableSerDe,
> SerDeSpec.class).schemaProps();
> {code}
> And it also seems like a relatively easy fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
