[jira] [Commented] (HIVE-10038) Add Calcite's ProjectMergeRule.
[
https://issues.apache.org/jira/browse/HIVE-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379550#comment-14379550
]
Hive QA commented on HIVE-10038:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707107/HIVE-10038.3.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 8336 tests executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_id1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3144/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3144/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3144/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707107 - PreCommit-HIVE-TRUNK-Build
Add Calcite's ProjectMergeRule.
---
Key: HIVE-10038
URL: https://issues.apache.org/jira/browse/HIVE-10038
Project: Hive
Issue Type: New Feature
Components: CBO, Logical Optimizer
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-10038.2.patch, HIVE-10038.3.patch, HIVE-10038.patch
Helps to improve latency by shortening operator pipeline. Folds adjacent
projections in one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10078) Optionally allow logging of records processed in fixed intervals
[
https://issues.apache.org/jira/browse/HIVE-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379914#comment-14379914
]
Hive QA commented on HIVE-10078:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707111/HIVE-10078.1.patch
{color:red}ERROR:{color} -1 due to 1711 failed/errored test(s), 8337 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alias_casted_column
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_char1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_char2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_concatenate_indexed_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition_authorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_invalidate_column_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_serde2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_varchar1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_varchar2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_view_as_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_tbl_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_array_map_access_nonconstant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_3
[jira] [Commented] (HIVE-10077) Use new ParquetInputSplit constructor API
[
https://issues.apache.org/jira/browse/HIVE-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379844#comment-14379844
]
Hive QA commented on HIVE-10077:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707110/HIVE-10077.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3146/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3146/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3146/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/spark-client-1.2.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/spark-client-1.2.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/spark-client/1.2.0-SNAPSHOT/spark-client-1.2.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/spark-client/1.2.0-SNAPSHOT/spark-client-1.2.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 1.2.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/ql (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as {KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as {KW_REGEXP, KW_RLIKE} KW_UNION KW_FROM using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as {KW_REGEXP, KW_RLIKE} KW_UNION KW_REDUCE
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as {KW_REGEXP, KW_RLIKE} KW_UNION KW_MAP using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as
[jira] [Commented] (HIVE-1575) get_json_object does not support JSON array at the root level
[
https://issues.apache.org/jira/browse/HIVE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380071#comment-14380071
]
Hive QA commented on HIVE-1575:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707117/HIVE-1575.4.patch
{color:green}SUCCESS:{color} +1 8339 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3148/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3148/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3148/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707117 - PreCommit-HIVE-TRUNK-Build
get_json_object does not support JSON array at the root level
-
Key: HIVE-1575
URL: https://issues.apache.org/jira/browse/HIVE-1575
Project: Hive
Issue Type: Improvement
Components: UDF
Affects Versions: 0.7.0
Reporter: Steven Wong
Assignee: Alexander Pivovarov
Attachments:
0001-Updated-UDFJson-to-allow-arrays-as-a-root-object.patch,
HIVE-1575.2.patch, HIVE-1575.3.patch, HIVE-1575.4.patch
Currently, get_json_object(json_txt, path) always returns null if json_txt is
not a JSON object (e.g. is a JSON array) at the root level.
I have a table column of JSON arrays at the root level, but I can't parse it
because of that.
get_json_object should accept any JSON value (string, number, object, array,
true, false, null), not just object, at the root level. In other words, it
should behave as if it were named get_json_value or simply get_json.
Per the JSON RFC, an array is indeed legal top-level JSON-text
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9839) HiveServer2 leaks OperationHandle on async queries which fail at compile phase
[ https://issues.apache.org/jira/browse/HIVE-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379998#comment-14379998 ] Jimmy Xiang commented on HIVE-9839: --- Thanks for making the change. +1 pending on test. HiveServer2 leaks OperationHandle on async queries which fail at compile phase -- Key: HIVE-9839 URL: https://issues.apache.org/jira/browse/HIVE-9839 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0, 0.13.1, 1.0.0, 1.1.0 Reporter: Nemon Lou Assignee: Nemon Lou Priority: Critical Attachments: HIVE-9839.patch, HIVE-9839.patch, HIVE-9839.patch, HIVE-9839.patch, OperationHandleMonitor.java Using beeline to connect to HiveServer2.And type the following: drop table if exists table_not_exists; select * from table_not_exists; There will be an OperationHandle object staying in HiveServer2's memory for ever even after quit from beeline . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-10087:
-
Attachment: HIVE-10087.patch
The following is the output with the proposed fix with --silent=true and
--silent=false (query is echoed back)
{code}
$ beeline -u jdbc:hive2://localhost.com:1/default --showHeader=false
--silent=true -f query.sql
+---++--+
| true | t |
| false | f |
| TRUE | T |
| FALSE | F |
| ZERO | 0 |
+---++--+
$ beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=false -f query.sql
0: jdbc:hive2://localhost select * from booleantest limit 5;
+---++--+
| true | t |
| false | f |
| TRUE | T |
| FALSE | F |
| ZERO | 0 |
+---++--+
5 rows selected (1.087 seconds)
0: jdbc:hive2://localhost
Closing: 0: jdbc:hive2://localhost:1/default
{code}
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10073) Runtime exception when querying HBase with Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-10073:
---
Attachment: HIVE-10073.1-spark.patch
Runtime exception when querying HBase with Spark [Spark Branch]
---
Key: HIVE-10073
URL: https://issues.apache.org/jira/browse/HIVE-10073
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: spark-branch
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: HIVE-10073.1-spark.patch
When querying HBase with Spark, we got
{noformat}
Caused by: java.lang.IllegalArgumentException: Must specify table name
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:188)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveOutputFormat(HiveFileFormatUtils.java:276)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveOutputFormat(HiveFileFormatUtils.java:266)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:331)
{noformat}
But it works fine for MapReduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380320#comment-14380320
]
Mithun Radhakrishnan commented on HIVE-9674:
Sush, could you please review this one? I'd like to avoid another rebase.
*DropPartitionEvent should handle partition-sets.
-
Key: HIVE-9674
URL: https://issues.apache.org/jira/browse/HIVE-9674
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9674.2.patch, HIVE-9736.3.patch, HIVE-9736.4.patch
Dropping a set of N partitions from a table currently results in N
DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
is wasteful, especially so for large N. It also makes it impossible to even
try to run authorization-checks on all partitions in a batch.
Taking the cue from HIVE-9609, we should compose an {{IterablePartition}}
in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10078) Optionally allow logging of records processed in fixed intervals
[
https://issues.apache.org/jira/browse/HIVE-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380323#comment-14380323
]
Prasanth Jayachandran commented on HIVE-10078:
--
[~hagleitn] Can you take a look at the failures? I suspect its because of
hiveconf.getLongVar(). The default value should be 0L instead of 0 I guess.
Here is the exception
{code}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 23 more
Caused by: java.lang.AssertionError: hive.log.every.n.records
at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:2408)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:432)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:119)
... 23 more
{code}
Optionally allow logging of records processed in fixed intervals
Key: HIVE-10078
URL: https://issues.apache.org/jira/browse/HIVE-10078
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-10078.1.patch
Tasks today log progress (records in/records out) on an exponential scale (1,
10, 100, ...). Sometimes it's helpful to be able to switch to fixed interval.
That can help debugging certain issues that look like a hang, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380336#comment-14380336
]
Xuefu Zhang commented on HIVE-10087:
Patch looks good. One minor thing, I noticed there is a bland line in the
console output for -f when --silent=true. Is there a way to get rid of that?
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9780) Add another level of explain for RDBMS audience
[ https://issues.apache.org/jira/browse/HIVE-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9780: -- Attachment: HIVE-9780.03.patch Add another level of explain for RDBMS audience --- Key: HIVE-9780 URL: https://issues.apache.org/jira/browse/HIVE-9780 Project: Hive Issue Type: Improvement Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Minor Attachments: HIVE-9780.01.patch, HIVE-9780.02.patch, HIVE-9780.03.patch Current Hive Explain (default) is targeted at MR Audience. We need a new level of explain plan to be targeted at RDBMS audience. The explain requires these: 1) The focus needs to be on what part of the query is being executed rather than internals of the engines 2) There needs to be a clearly readable tree of operations 3) Examples - Table scan should mention the table being scanned, the Sarg, the size of table and expected cardinality after the Sarg'ed read. The join should mention the table being joined with and the join condition. The aggregate should mention the columns in the group-by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10090) Add connection manager for Tephra
[
https://issues.apache.org/jira/browse/HIVE-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380488#comment-14380488
]
Alan Gates commented on HIVE-10090:
---
I have not yet run this on a stand alone instance of Tephra, but I have run it
with the unit tests and an in memory version of Tephra and it works. To run
with this rather than the no-transactions connections just add
{{-Dhive.metastore.hbase.connection.class=org.apache.hadoop.hive.metastore.hbase.TephraHBaseConnection}}
to your mvn test command.
Add connection manager for Tephra
-
Key: HIVE-10090
URL: https://issues.apache.org/jira/browse/HIVE-10090
Project: Hive
Issue Type: Sub-task
Components: Metastore
Affects Versions: hbase-metastore-branch
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-10090.patch
The task is to create an implementation of HBaseConnection that will use
Tephra for transaction management.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10069) CBO (Calcite Return Path): Ambiguity table name causes problem in field trimmer [CBO Branch]
[ https://issues.apache.org/jira/browse/HIVE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380507#comment-14380507 ] Jesus Camacho Rodriguez commented on HIVE-10069: [~jpullokkaran], can you check it? Thanks CBO (Calcite Return Path): Ambiguity table name causes problem in field trimmer [CBO Branch] Key: HIVE-10069 URL: https://issues.apache.org/jira/browse/HIVE-10069 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-10069.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380536#comment-14380536
]
Ashutosh Chauhan commented on HIVE-10085:
-
Can you create a RB entry for this?
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8817) Create unit test where we insert into an encrypted table and then read from it with pig
[
https://issues.apache.org/jira/browse/HIVE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380623#comment-14380623
]
Hive QA commented on HIVE-8817:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704794/HIVE-8817.patch
{color:green}SUCCESS:{color} +1 8342 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3151/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3151/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3151/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704794 - PreCommit-HIVE-TRUNK-Build
Create unit test where we insert into an encrypted table and then read from
it with pig
---
Key: HIVE-8817
URL: https://issues.apache.org/jira/browse/HIVE-8817
Project: Hive
Issue Type: Sub-task
Affects Versions: encryption-branch
Reporter: Brock Noland
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-8817.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9518) Implement MONTHS_BETWEEN aligned with Oracle one
[ https://issues.apache.org/jira/browse/HIVE-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-9518: -- Attachment: HIVE-9518.5.patch patch #5 refactored the code Implement MONTHS_BETWEEN aligned with Oracle one Key: HIVE-9518 URL: https://issues.apache.org/jira/browse/HIVE-9518 Project: Hive Issue Type: Bug Reporter: Xiaobing Zhou Assignee: Alexander Pivovarov Attachments: HIVE-9518.1.patch, HIVE-9518.2.patch, HIVE-9518.3.patch, HIVE-9518.4.patch, HIVE-9518.5.patch This is used to track work to build Oracle like months_between. Here's semantics: MONTHS_BETWEEN returns number of months between dates date1 and date2. If date1 is later than date2, then the result is positive. If date1 is earlier than date2, then the result is negative. If date1 and date2 are either the same days of the month or both last days of months, then the result is always an integer. Otherwise Oracle Database calculates the fractional portion of the result based on a 31-day month and considers the difference in time components date1 and date2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380419#comment-14380419
]
Naveen Gangam commented on HIVE-10087:
--
Thanks Xuefu, I will take a look where this is from. I suspect this might be
from jline printing an empty buffer that holds the prompt the query string.
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9486) Use session classloader instead of application loader
[ https://issues.apache.org/jira/browse/HIVE-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380424#comment-14380424 ] Ashutosh Chauhan commented on HIVE-9486: Changes in JobState was made for completeness sake (of having an uniform way of class loading in Hive), no other particular reason AFAIK. I am fine with just using Class.forName() there instead, since this call is on remote short lived launcher process, where issue described here is likely irrelevant. Use session classloader instead of application loader - Key: HIVE-9486 URL: https://issues.apache.org/jira/browse/HIVE-9486 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 1.2.0 Attachments: HIVE-9486.1.patch.txt, HIVE-9486.2.patch.txt From http://www.mail-archive.com/dev@hive.apache.org/msg107615.html Looks reasonable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380438#comment-14380438
]
Gunther Hagleitner commented on HIVE-9976:
--
it seems you're setting numexpectedevents to 0 first and then turn around and
call decrement. Why not just set to -1? Also - why atomic integers? as far as i
can tell all access to these maps is synchronized.
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Fix For: 1.0.1
Attachments: HIVE-9976.1.patch, llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-10085:
Attachment: HIVE-10085.patch
When the sizes RowSchema and the columns after pruning are the same, right now,
we don't update SEL operator info in ColumnPrunerLateralViewForwardProc while
the order of the columns may get reordered and require to update SEL operator
as well in this issue.
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10090) Add connection manager for Tephra
[ https://issues.apache.org/jira/browse/HIVE-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-10090: -- Attachment: HIVE-10090.patch Add connection manager for Tephra - Key: HIVE-10090 URL: https://issues.apache.org/jira/browse/HIVE-10090 Project: Hive Issue Type: Sub-task Components: Metastore Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-10090.patch The task is to create an implementation of HBaseConnection that will use Tephra for transaction management. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9563) CBO(Calcite Return Path): Translate GB to Hive OP [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-9563: - Attachment: HIVE-9563.patch CBO(Calcite Return Path): Translate GB to Hive OP [CBO branch] -- Key: HIVE-9563 URL: https://issues.apache.org/jira/browse/HIVE-9563 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-9563.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380446#comment-14380446
]
Gunther Hagleitner commented on HIVE-9976:
--
does it make sense to make initialize in the pruner private now? (can't be used
to init anymore - only from the constr). Also, the parameters aren't used
anymore, right?
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Fix For: 1.0.1
Attachments: HIVE-9976.1.patch, llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380625#comment-14380625
]
Aihua Xu commented on HIVE-10085:
-
RB entry: https://reviews.apache.org/r/32491/
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380426#comment-14380426
]
Gunther Hagleitner commented on HIVE-9976:
--
i see you've fixed calling close consistently on the data input stream. maybe
use try{}finally there?
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Fix For: 1.0.1
Attachments: HIVE-9976.1.patch, llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9976:
-
Fix Version/s: (was: 1.0.1)
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Attachments: HIVE-9976.1.patch, llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10091) Generate Hbase execution plan for partition filter conditions in HbaseStore api calls
[ https://issues.apache.org/jira/browse/HIVE-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10091: - Attachment: HIVE-10091.1.patch Generate Hbase execution plan for partition filter conditions in HbaseStore api calls - Key: HIVE-10091 URL: https://issues.apache.org/jira/browse/HIVE-10091 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: hbase-metastore-branch Attachments: HIVE-10091.1.patch RawStore functions that support partition filtering are the following - getPartitionsByExpr getPartitionsByFilter (takes filter string as argument, used from hcatalog) We need to generate a query execution plan in terms of Hbase scan api calls for a given filter condition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10091) Generate Hbase execution plan for partition filter conditions in HbaseStore api calls
[ https://issues.apache.org/jira/browse/HIVE-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10091: - Description: RawStore functions that support partition filtering are the following - getPartitionsByExpr getPartitionsByFilter (takes filter string as argument, used from hcatalog) We need to generate a query execution plan in terms of Hbase scan api calls for a given filter condition. NO PRECOMMIT TESTS was: RawStore functions that support partition filtering are the following - getPartitionsByExpr getPartitionsByFilter (takes filter string as argument, used from hcatalog) We need to generate a query execution plan in terms of Hbase scan api calls for a given filter condition. Generate Hbase execution plan for partition filter conditions in HbaseStore api calls - Key: HIVE-10091 URL: https://issues.apache.org/jira/browse/HIVE-10091 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: hbase-metastore-branch Attachments: HIVE-10091.1.patch RawStore functions that support partition filtering are the following - getPartitionsByExpr getPartitionsByFilter (takes filter string as argument, used from hcatalog) We need to generate a query execution plan in terms of Hbase scan api calls for a given filter condition. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1575) get_json_object does not support JSON array at the root level
[ https://issues.apache.org/jira/browse/HIVE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380723#comment-14380723 ] Jason Dere commented on HIVE-1575: -- +1 if tests look good get_json_object does not support JSON array at the root level - Key: HIVE-1575 URL: https://issues.apache.org/jira/browse/HIVE-1575 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.7.0 Reporter: Steven Wong Assignee: Alexander Pivovarov Attachments: 0001-Updated-UDFJson-to-allow-arrays-as-a-root-object.patch, HIVE-1575.2.patch, HIVE-1575.3.patch, HIVE-1575.4.patch, HIVE-1575.5.patch Currently, get_json_object(json_txt, path) always returns null if json_txt is not a JSON object (e.g. is a JSON array) at the root level. I have a table column of JSON arrays at the root level, but I can't parse it because of that. get_json_object should accept any JSON value (string, number, object, array, true, false, null), not just object, at the root level. In other words, it should behave as if it were named get_json_value or simply get_json. Per the JSON RFC, an array is indeed legal top-level JSON-text -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10078) Optionally allow logging of records processed in fixed intervals
[ https://issues.apache.org/jira/browse/HIVE-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380780#comment-14380780 ] Prasanth Jayachandran commented on HIVE-10078: -- LGTM, +1. Pending unit tests. Optionally allow logging of records processed in fixed intervals Key: HIVE-10078 URL: https://issues.apache.org/jira/browse/HIVE-10078 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-10078.1.patch, HIVE-10078.2.patch Tasks today log progress (records in/records out) on an exponential scale (1, 10, 100, ...). Sometimes it's helpful to be able to switch to fixed interval. That can help debugging certain issues that look like a hang, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10074) Ability to run HCat Client Unit tests in a system test setting
[
https://issues.apache.org/jira/browse/HIVE-10074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380727#comment-14380727
]
Sushanth Sowmyan commented on HIVE-10074:
-
The test failures do not look related to this patch. Still, might be worth
rerunning the tests to confirm. I'm going to cancel patch, reupload the same
patch and set available again.
Ability to run HCat Client Unit tests in a system test setting
--
Key: HIVE-10074
URL: https://issues.apache.org/jira/browse/HIVE-10074
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
Attachments: HIVE-10074.patch
Following testsuite
{{hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java}}
is a JUnit testsuite to test some basic HCat client API. During setup it
brings up a Hive Metastore with embedded Derby. The testsuite however will be
even more useful if it can be run against a running Hive Metastore
(transparent to whatever backing DB its running against).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10074) Ability to run HCat Client Unit tests in a system test setting
[
https://issues.apache.org/jira/browse/HIVE-10074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10074:
Attachment: HIVE-10074.1.patch
Ability to run HCat Client Unit tests in a system test setting
--
Key: HIVE-10074
URL: https://issues.apache.org/jira/browse/HIVE-10074
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
Attachments: HIVE-10074.1.patch, HIVE-10074.patch
Following testsuite
{{hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java}}
is a JUnit testsuite to test some basic HCat client API. During setup it
brings up a Hive Metastore with embedded Derby. The testsuite however will be
even more useful if it can be run against a running Hive Metastore
(transparent to whatever backing DB its running against).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380733#comment-14380733
]
Aihua Xu commented on HIVE-10085:
-
Please try again. First time to use the rbt tool. Seems I still need to set
permission and publish.
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-9582: --- Attachment: HIVE-9582.5.patch (Attaching rebased patch) HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380740#comment-14380740
]
Gunther Hagleitner commented on HIVE-9976:
--
+1
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Attachments: HIVE-9976.1.patch, HIVE-9976.2.patch,
llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380690#comment-14380690
]
Naveen Gangam commented on HIVE-10087:
--
This is coming from the jline code. As it reads from the script file character
by character, it adds it to the StringBuffer and also echoes it to the
terminal, if mask character is null. In this case, we have set the mask
character to null character value to prevent this echo. When it encounters a
EOL character in the file aka ;, it completes the line and prints a newline
character and returns the line to beeline to be executed.
{code}
snip
case ACCEPT_LINE:
return accept();
/snip
snip
public String accept() throws IOException {
moveToEnd();
println(); // output newline
flush();
return finishBuffer();
}
/snip
snip
public final void println() throws IOException {
print(CR);
}
/snip
{code}
If this newline were to be turned off, the first line of the result set would
be printed on the same line as the query. Something like this
{code}
$ beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=false --outputformat=tsv2 -f query.sql
scan complete in 2ms
Connecting to jdbc:hive2://localhost:1/default
0: jdbc:hive2://localhost select * from booleantest limit 5;true t
false f
TRUET
FALSE F
ZERO0
{code}
jline code does not provide any APIs to allow me to dictate this. Something
like this should fix this. I will file a jira against jline.
{code}
public final void println() throws IOException {
if (mask == null)
print(CR);
}
{code}
For now, we may have to get by with what we have.
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380721#comment-14380721
]
Naveen Gangam commented on HIVE-10087:
--
I have filed a new jira against jline to have this fixed.
https://github.com/jline/jline2/issues/181
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-1575) get_json_object does not support JSON array at the root level
[ https://issues.apache.org/jira/browse/HIVE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-1575: -- Attachment: HIVE-1575.5.patch patch #5. fixed 2 issues with initial json path validation get_json_object does not support JSON array at the root level - Key: HIVE-1575 URL: https://issues.apache.org/jira/browse/HIVE-1575 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.7.0 Reporter: Steven Wong Assignee: Alexander Pivovarov Attachments: 0001-Updated-UDFJson-to-allow-arrays-as-a-root-object.patch, HIVE-1575.2.patch, HIVE-1575.3.patch, HIVE-1575.4.patch, HIVE-1575.5.patch Currently, get_json_object(json_txt, path) always returns null if json_txt is not a JSON object (e.g. is a JSON array) at the root level. I have a table column of JSON arrays at the root level, but I can't parse it because of that. get_json_object should accept any JSON value (string, number, object, array, true, false, null), not just object, at the root level. In other words, it should behave as if it were named get_json_value or simply get_json. Per the JSON RFC, an array is indeed legal top-level JSON-text -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380812#comment-14380812 ] Sushanth Sowmyan commented on HIVE-9582: I've gone through this patch, and am +1 on the intent of the .4.patch. It did not apply cleanly on trunk, however, so I've rebased the patch slightly to make it patch. I'm uploading the .5.patch to reflect this change so we can get one more unit test run complete with the latest patch. That said, I do have one point of concern. Changing HCatUtil.getHiveClient to return IMetaStoreClient instead of a HMSC can potentially break oozie, falcon and sqoop compilation, not to mention other external user code that uses that method to get a HiveMetaStoreClient. I agree that it should have been IMSC all along, rather than HMSC, but it is now published interface. We could solve this as follows: a) create a new function getMetaStoreClient() which returns a IMSC. Change all code in HCat to refer to this function instead of the current getHiveClient() b) retain the existing getHiveClient() and have it continue to return a HMSC, and mark it for deprecation in 2 releases - i.e. deprecated in 1.2, gone in 1.3. I worried if this was possible given that you change the Cache class, but luckily, your underlying implementation of ICacheableMetaStoreClient is a CacheableHiveMetaStoreClient which is a HMSC, so this is still possible with minimal changes. Once we do this, we should communicate to oozie/falcon/sqoop developers to change their usage of this function to the more generic one. HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-9976:
-
Attachment: HIVE-9976.2.patch
Thanks for the review. Updated patch with comments addressed, and some more
changes.
bq. Not your fault - but there are 2 paths through HiveSplitGenerator.
Moved the methods into SplitGrouper. There's a static cache in there which
seems a little strange. Will create a follow up jira to investigate this. For
now I've changed that to a ConcurrentMap since split generation can run in
parallel.
bq. i see you've fixed calling close consistently on the data input stream.
maybe use try{}finally there?
Fixed. There was a bug with some of the other conditions which I'd changed.
Fixed that as well.
bq. it seems you're setting numexpectedevents to 0 first and then turn around
and call decrement. Why not just set to -1? Also - why atomic integers? as far
as i can tell all access to these maps is synchronized.
numExpectedEvents is decremented for each column for which a source will send
events. That's used to track total number of expected events from that source.
Added a comment for this.
Moved from AtomicIntegers to MutableInt - this was just to avoid re-inserting
the Integer into the map, and not for thread safety.
bq. does it make sense to make initialize in the pruner private now? (can't be
used to init anymore - only from the constr). Also, the parameters aren't used
anymore, right?
Done, along with some other methods.
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Attachments: HIVE-9976.1.patch, HIVE-9976.2.patch,
llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380680#comment-14380680
]
Ashutosh Chauhan commented on HIVE-10085:
-
Above link says:
{noformat}
You don't have access to this review request.
This review request is private. You must be a requested reviewer, either
directly or on a requested group, and have permission to access the repository
in order to view this review request.
{noformat}
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10078) Optionally allow logging of records processed in fixed intervals
[ https://issues.apache.org/jira/browse/HIVE-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-10078: -- Attachment: HIVE-10078.2.patch Optionally allow logging of records processed in fixed intervals Key: HIVE-10078 URL: https://issues.apache.org/jira/browse/HIVE-10078 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-10078.1.patch, HIVE-10078.2.patch Tasks today log progress (records in/records out) on an exponential scale (1, 10, 100, ...). Sometimes it's helpful to be able to switch to fixed interval. That can help debugging certain issues that look like a hang, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380792#comment-14380792
]
Hive QA commented on HIVE-9664:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707187/HIVE-9664.5.patch
{color:green}SUCCESS:{color} +1 8344 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3153/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3153/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3153/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707187 - PreCommit-HIVE-TRUNK-Build
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: hive, patch
Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4941) PTest2 Investigate Ignores
[
https://issues.apache.org/jira/browse/HIVE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380793#comment-14380793
]
Anthony Hsu commented on HIVE-4941:
---
I'm trying to run TestHiveMetaStore but I'm having trouble getting it to run.
I made the following changes to the root pom.xml file
{code}
diff --git a/pom.xml b/pom.xml
index 5d4f13c..a8098ce 100644
--- a/pom.xml
+++ b/pom.xml
@@ -712,12 +712,12 @@
configuration
excludes
exclude**/TestSerDe.java/exclude
-exclude**/TestHiveMetaStore.java/exclude
+!--exclude**/TestHiveMetaStore.java/exclude--
exclude**/ql/exec/vector/util/*.java/exclude
exclude**/ql/exec/vector/udf/legacy/*.java/exclude
exclude**/ql/exec/vector/udf/generic/*.java/exclude
exclude**/TestHiveServer2Concurrency.java/exclude
-exclude**/TestHiveMetaStore.java/exclude
+!--exclude**/TestHiveMetaStore.java/exclude--
/excludes
redirectTestOutputToFiletrue/redirectTestOutputToFile
reuseForksfalse/reuseForks
{code}
and then ran
{code}
mvn clean install -DskipTests -Phadoop-2
cd itests
mvn clean install -DskipTests -Phadoop-2
mvn test -Phadoop-2 -Dtest=TestHiveMetaStore
{code}
However, in the output, I see no tests were run:
{code}
---
T E S T S
---
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
{code}
A couple questions:
# Why is the TestHiveMetaStore test excluded?
# How do I run the TestHiveMetaStore test?
PTest2 Investigate Ignores
--
Key: HIVE-4941
URL: https://issues.apache.org/jira/browse/HIVE-4941
Project: Hive
Issue Type: Task
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Currently we excluding the following tests:
unitTests.exclude = TestHiveMetaStore TestSerDe TestBeeLineDriver
TestHiveServer2Concurrency TestJdbcDriver2 TestHiveServer2Concurrency
TestBeeLineDriver
some of them we got from the build files but I am not sure about
TestJdbcDriver2 for example. We should investigate why these are excluded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4941) PTest2 Investigate Ignores
[ https://issues.apache.org/jira/browse/HIVE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380823#comment-14380823 ] Anthony Hsu commented on HIVE-4941: --- I figured it out. I'm not supposed to run TestHiveMetaStore directly. I'm supposed to run the subclasses TestEmbeddedHiveMetaStore and TestRemoteHiveMetaStore instead. Running the subclasses works. PTest2 Investigate Ignores -- Key: HIVE-4941 URL: https://issues.apache.org/jira/browse/HIVE-4941 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Currently we excluding the following tests: unitTests.exclude = TestHiveMetaStore TestSerDe TestBeeLineDriver TestHiveServer2Concurrency TestJdbcDriver2 TestHiveServer2Concurrency TestBeeLineDriver some of them we got from the build files but I am not sure about TestJdbcDriver2 for example. We should investigate why these are excluded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381023#comment-14381023 ] Thiruvel Thirumoolan commented on HIVE-9582: Thanks [~sushanth] for your time and inputs. Much appreciated. I was of the opinion getHiveClient() was internal and not supposed to be used outside (yeah, we have lot of public methods). I asked Oozie team at Yahoo! to use HCatClient going forward, I am not sure those changes are in trunk. I didn't know falcon/sqoop might be using it. I will change as you suggested and update another patch. HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381040#comment-14381040 ] Sushanth Sowmyan commented on HIVE-9582: Admittedly, I also agree that we should slowly push people to use HCatClient, and away from HiveMetaStoreClient for our interface purposes, and so, eventually, we should internalize HCatUtil. HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380933#comment-14380933
]
Hive QA commented on HIVE-10087:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707246/HIVE-10087.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8337 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3154/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3154/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3154/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707246 - PreCommit-HIVE-TRUNK-Build
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10086) Hive throws error when accessing Parquet file schema using field name match
[
https://issues.apache.org/jira/browse/HIVE-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-10086:
---
Attachment: HIVE-10086.1.patch
Hive throws error when accessing Parquet file schema using field name match
---
Key: HIVE-10086
URL: https://issues.apache.org/jira/browse/HIVE-10086
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-10086.1.patch
When Hive table schema contains a portion of the schema of a Parquet file,
then the access to the values should work if the field names match the
schema. This does not work when a struct data type is in the schema, and
the Hive schema contains just a portion of the struct elements. Hive throws
an error instead.
This is the example and how to reproduce:
First, create a parquet table, and add some values on it:
{code}
CREATE TABLE test1 (id int, name string, address
structnumber:int,street:string,zip:string) STORED AS PARQUET;
INSERT INTO TABLE test1 SELECT 1, 'Roger',
named_struct('number',8600,'street','Congress Ave.','zip','87366') FROM
srcpart LIMIT 1;
{code}
Note: {{srcpart}} could be any table. It is just used to leverage the INSERT
statement.
The above table example generates the following Parquet file schema:
{code}
message hive_schema {
optional int32 id;
optional binary name (UTF8);
optional group address {
optional int32 number;
optional binary street (UTF8);
optional binary zip (UTF8);
}
}
{code}
Afterwards, I create a table that contains just a portion of the schema, and
load the Parquet file generated above, a query will fail on that table:
{code}
CREATE TABLE test1 (name string, address structstreet:string) STORED AS
PARQUET;
LOAD DATA LOCAL INPATH '/tmp/HiveGroup.parquet' OVERWRITE INTO TABLE test1;
hive SELECT name FROM test1;
OK
Roger
Time taken: 0.071 seconds, Fetched: 1 row(s)
hive SELECT address FROM test1;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.UnsupportedOperationException: Cannot inspect
org.apache.hadoop.io.IntWritable
Time taken: 0.085 seconds
{code}
I would expect that Parquet can access the matched names, but Hive throws an
error instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10086) Hive throws error when accessing Parquet file schema using field name match
[
https://issues.apache.org/jira/browse/HIVE-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-10086:
---
Attachment: HiveGroup.parquet
Hive throws error when accessing Parquet file schema using field name match
---
Key: HIVE-10086
URL: https://issues.apache.org/jira/browse/HIVE-10086
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-10086.1.patch, HiveGroup.parquet
When Hive table schema contains a portion of the schema of a Parquet file,
then the access to the values should work if the field names match the
schema. This does not work when a struct data type is in the schema, and
the Hive schema contains just a portion of the struct elements. Hive throws
an error instead.
This is the example and how to reproduce:
First, create a parquet table, and add some values on it:
{code}
CREATE TABLE test1 (id int, name string, address
structnumber:int,street:string,zip:string) STORED AS PARQUET;
INSERT INTO TABLE test1 SELECT 1, 'Roger',
named_struct('number',8600,'street','Congress Ave.','zip','87366') FROM
srcpart LIMIT 1;
{code}
Note: {{srcpart}} could be any table. It is just used to leverage the INSERT
statement.
The above table example generates the following Parquet file schema:
{code}
message hive_schema {
optional int32 id;
optional binary name (UTF8);
optional group address {
optional int32 number;
optional binary street (UTF8);
optional binary zip (UTF8);
}
}
{code}
Afterwards, I create a table that contains just a portion of the schema, and
load the Parquet file generated above, a query will fail on that table:
{code}
CREATE TABLE test1 (name string, address structstreet:string) STORED AS
PARQUET;
LOAD DATA LOCAL INPATH '/tmp/HiveGroup.parquet' OVERWRITE INTO TABLE test1;
hive SELECT name FROM test1;
OK
Roger
Time taken: 0.071 seconds, Fetched: 1 row(s)
hive SELECT address FROM test1;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.UnsupportedOperationException: Cannot inspect
org.apache.hadoop.io.IntWritable
Time taken: 0.085 seconds
{code}
I would expect that Parquet can access the matched names, but Hive throws an
error instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381017#comment-14381017
]
Mithun Radhakrishnan commented on HIVE-9582:
I wonder, should {{HCatUtils}} be package-protected?
HCatalog should use IMetaStoreClient interface
--
Key: HIVE-9582
URL: https://issues.apache.org/jira/browse/HIVE-9582
Project: Hive
Issue Type: Sub-task
Components: HCatalog, Metastore
Affects Versions: 0.14.0, 0.13.1
Reporter: Thiruvel Thirumoolan
Assignee: Thiruvel Thirumoolan
Labels: hcatalog, metastore, rolling_upgrade
Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch
Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
Hence during a failure, the client retries and possibly succeeds. But
HCatalog has long been using HiveMetaStoreClient directly and hence failures
are costly, especially if they are during the commit stage of a job. Its also
not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381036#comment-14381036 ] Sushanth Sowmyan commented on HIVE-9582: I think that was the goal behind InternalUtil, while HCatUtil was intended to be public-ish-facing. And even if there are functions in HCatUtil that might still be worth putting behind a InterfaceAudience.Private wall, getHiveClient, closeHiveClientQuietly and getHiveConf were intended to be public. HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380928#comment-14380928 ] Anthony Hsu commented on HIVE-9664: --- +1 (though I don't have Hive commit access) Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: hive, patch Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10086) Hive throws error when accessing Parquet file schema using field name match
[
https://issues.apache.org/jira/browse/HIVE-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380937#comment-14380937
]
Sergio Peña commented on HIVE-10086:
[~xuefuz] [~szehon] Could you help me review this patch?
Hive throws error when accessing Parquet file schema using field name match
---
Key: HIVE-10086
URL: https://issues.apache.org/jira/browse/HIVE-10086
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-10086.1.patch, HiveGroup.parquet
When Hive table schema contains a portion of the schema of a Parquet file,
then the access to the values should work if the field names match the
schema. This does not work when a struct data type is in the schema, and
the Hive schema contains just a portion of the struct elements. Hive throws
an error instead.
This is the example and how to reproduce:
First, create a parquet table, and add some values on it:
{code}
CREATE TABLE test1 (id int, name string, address
structnumber:int,street:string,zip:string) STORED AS PARQUET;
INSERT INTO TABLE test1 SELECT 1, 'Roger',
named_struct('number',8600,'street','Congress Ave.','zip','87366') FROM
srcpart LIMIT 1;
{code}
Note: {{srcpart}} could be any table. It is just used to leverage the INSERT
statement.
The above table example generates the following Parquet file schema:
{code}
message hive_schema {
optional int32 id;
optional binary name (UTF8);
optional group address {
optional int32 number;
optional binary street (UTF8);
optional binary zip (UTF8);
}
}
{code}
Afterwards, I create a table that contains just a portion of the schema, and
load the Parquet file generated above, a query will fail on that table:
{code}
CREATE TABLE test1 (name string, address structstreet:string) STORED AS
PARQUET;
LOAD DATA LOCAL INPATH '/tmp/HiveGroup.parquet' OVERWRITE INTO TABLE test1;
hive SELECT name FROM test1;
OK
Roger
Time taken: 0.071 seconds, Fetched: 1 row(s)
hive SELECT address FROM test1;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.UnsupportedOperationException: Cannot inspect
org.apache.hadoop.io.IntWritable
Time taken: 0.085 seconds
{code}
I would expect that Parquet can access the matched names, but Hive throws an
error instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10087) Beeline's --silent option should suppress query from being echoed when running with -f option
[
https://issues.apache.org/jira/browse/HIVE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380945#comment-14380945
]
Xuefu Zhang commented on HIVE-10087:
+1
Beeline's --silent option should suppress query from being echoed when
running with -f option
-
Key: HIVE-10087
URL: https://issues.apache.org/jira/browse/HIVE-10087
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 0.13.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor
Attachments: HIVE-10087.patch
The {{-e}} and the {{-f}} options behave differently.
{code}
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -f select.sql
0: jdbc:hive2://localhost:1/default select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives 299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
beeline -u jdbc:hive2://localhost:1/default --showHeader=false
--silent=true -e select * from sample_07 limit 5;
--
00- All Occupations 134354250 40690
11- Management occupations 6003930 96150
11-1011 Chief executives299160 151370
11-1021 General and operations managers 1655410 103780
11-1031 Legislators 61110 33880
--
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10086) Hive throws error when accessing Parquet file schema using field name match
[
https://issues.apache.org/jira/browse/HIVE-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381006#comment-14381006
]
Sergio Peña commented on HIVE-10086:
[~rdblue] Could you help me review this?
Hive throws error when accessing Parquet file schema using field name match
---
Key: HIVE-10086
URL: https://issues.apache.org/jira/browse/HIVE-10086
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-10086.1.patch, HiveGroup.parquet
When Hive table schema contains a portion of the schema of a Parquet file,
then the access to the values should work if the field names match the
schema. This does not work when a struct data type is in the schema, and
the Hive schema contains just a portion of the struct elements. Hive throws
an error instead.
This is the example and how to reproduce:
First, create a parquet table, and add some values on it:
{code}
CREATE TABLE test1 (id int, name string, address
structnumber:int,street:string,zip:string) STORED AS PARQUET;
INSERT INTO TABLE test1 SELECT 1, 'Roger',
named_struct('number',8600,'street','Congress Ave.','zip','87366') FROM
srcpart LIMIT 1;
{code}
Note: {{srcpart}} could be any table. It is just used to leverage the INSERT
statement.
The above table example generates the following Parquet file schema:
{code}
message hive_schema {
optional int32 id;
optional binary name (UTF8);
optional group address {
optional int32 number;
optional binary street (UTF8);
optional binary zip (UTF8);
}
}
{code}
Afterwards, I create a table that contains just a portion of the schema, and
load the Parquet file generated above, a query will fail on that table:
{code}
CREATE TABLE test1 (name string, address structstreet:string) STORED AS
PARQUET;
LOAD DATA LOCAL INPATH '/tmp/HiveGroup.parquet' OVERWRITE INTO TABLE test1;
hive SELECT name FROM test1;
OK
Roger
Time taken: 0.071 seconds, Fetched: 1 row(s)
hive SELECT address FROM test1;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.UnsupportedOperationException: Cannot inspect
org.apache.hadoop.io.IntWritable
Time taken: 0.085 seconds
{code}
I would expect that Parquet can access the matched names, but Hive throws an
error instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10072) Add vectorization support for Hybrid Grace Hash Join
[ https://issues.apache.org/jira/browse/HIVE-10072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-10072: - Attachment: HIVE-10072.06.patch Upload 6th patch for testing Add vectorization support for Hybrid Grace Hash Join Key: HIVE-10072 URL: https://issues.apache.org/jira/browse/HIVE-10072 Project: Hive Issue Type: Improvement Affects Versions: 1.2.0 Reporter: Wei Zheng Assignee: Wei Zheng Fix For: 1.2.0 Attachments: HIVE-10072.01.patch, HIVE-10072.02.patch, HIVE-10072.03.patch, HIVE-10072.04.patch, HIVE-10072.05.patch, HIVE-10072.06.patch This task is to enable vectorization support for Hybrid Grace Hash Join feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10077) Use new ParquetInputSplit constructor API
[ https://issues.apache.org/jira/browse/HIVE-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381165#comment-14381165 ] Ferdinand Xu commented on HIVE-10077: - Hi [~spena], could you help me review it though it depends on HIVE-10076? Use new ParquetInputSplit constructor API - Key: HIVE-10077 URL: https://issues.apache.org/jira/browse/HIVE-10077 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10077.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10062) HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
[
https://issues.apache.org/jira/browse/HIVE-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong reassigned HIVE-10062:
--
Assignee: Pengcheng Xiong
HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
-
Key: HIVE-10062
URL: https://issues.apache.org/jira/browse/HIVE-10062
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Priority: Critical
In q.test environment with src table, execute the following query:
{code}
CREATE TABLE DEST1(key STRING, value STRING) STORED AS TEXTFILE;
CREATE TABLE DEST2(key STRING, val1 STRING, val2 STRING) STORED AS TEXTFILE;
FROM (select 'tst1' as key, cast(count(1) as string) as value from src s1
UNION all
select s2.key as key, s2.value as value from src s2) unionsrc
INSERT OVERWRITE TABLE DEST1 SELECT unionsrc.key, COUNT(DISTINCT
SUBSTR(unionsrc.value,5)) GROUP BY unionsrc.key
INSERT OVERWRITE TABLE DEST2 SELECT unionsrc.key, unionsrc.value,
COUNT(DISTINCT SUBSTR(unionsrc.value,5))
GROUP BY unionsrc.key, unionsrc.value;
select * from DEST1;
select * from DEST2;
{code}
DEST1 and DEST2 should both have 310 rows. However, DEST2 only has 1 row
tst1500 1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9767) Fixes in Hive UDF to be usable in Pig
[ https://issues.apache.org/jira/browse/HIVE-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-9767: - Attachment: HIVE-9767.3.patch There is a NPE, fixed in HIVE-9767.3.patch. Fixes in Hive UDF to be usable in Pig - Key: HIVE-9767 URL: https://issues.apache.org/jira/browse/HIVE-9767 Project: Hive Issue Type: Bug Components: UDF Reporter: Daniel Dai Assignee: Daniel Dai Attachments: HIVE-9767.1.patch, HIVE-9767.2.patch, HIVE-9767.3.patch There are issues in UDF never get exposed because the execution path is never tested: # Assume the ObjectInspector to be WritableObjectInspector not the ObjectInspector pass to UDF # Assume the input parameter to be Writable not respecting the ObjectInspector pass to UDF # Assume ConstantObjectInspector to be WritableConstantXXXObjectInspector # The InputObjectInspector does not match OutputObjectInspector of previous stage in UDAF # The execution path involving convertIfNecessary is never been tested Attach a patch to fix those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9780) Add another level of explain for RDBMS audience
[
https://issues.apache.org/jira/browse/HIVE-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381174#comment-14381174
]
Hive QA commented on HIVE-9780:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707262/HIVE-9780.03.patch
{color:red}ERROR:{color} -1 due to 546 failed/errored test(s), 8340 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_columns
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_indexes_edge_cases
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_indexes_syntax
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tablestatus
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_showparts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_compare_java_string
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_logic_java_boolean
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_testlength
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_testlength2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29
[jira] [Commented] (HIVE-9518) Implement MONTHS_BETWEEN aligned with Oracle one
[
https://issues.apache.org/jira/browse/HIVE-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381252#comment-14381252
]
Hive QA commented on HIVE-9518:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707266/HIVE-9518.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8341 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3157/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3157/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707266 - PreCommit-HIVE-TRUNK-Build
Implement MONTHS_BETWEEN aligned with Oracle one
Key: HIVE-9518
URL: https://issues.apache.org/jira/browse/HIVE-9518
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Xiaobing Zhou
Assignee: Alexander Pivovarov
Attachments: HIVE-9518.1.patch, HIVE-9518.2.patch, HIVE-9518.3.patch,
HIVE-9518.4.patch, HIVE-9518.5.patch
This is used to track work to build Oracle like months_between. Here's
semantics:
MONTHS_BETWEEN returns number of months between dates date1 and date2. If
date1 is later than date2, then the result is positive. If date1 is earlier
than date2, then the result is negative. If date1 and date2 are either the
same days of the month or both last days of months, then the result is always
an integer. Otherwise Oracle Database calculates the fractional portion of
the result based on a 31-day month and considers the difference in time
components date1 and date2.
Should accept date, timestamp and string arguments in the format '-MM-dd'
or '-MM-dd HH:mm:ss'. The time part should be ignored.
The result should be rounded to 8 decimal places.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10072) Add vectorization support for Hybrid Grace Hash Join
[ https://issues.apache.org/jira/browse/HIVE-10072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381397#comment-14381397 ] Matt McCline commented on HIVE-10072: - I compared the 6th patch to the 3rd. Still looks good for launch after tests succeed. Add vectorization support for Hybrid Grace Hash Join Key: HIVE-10072 URL: https://issues.apache.org/jira/browse/HIVE-10072 Project: Hive Issue Type: Improvement Affects Versions: 1.2.0 Reporter: Wei Zheng Assignee: Wei Zheng Fix For: 1.2.0 Attachments: HIVE-10072.01.patch, HIVE-10072.02.patch, HIVE-10072.03.patch, HIVE-10072.04.patch, HIVE-10072.05.patch, HIVE-10072.06.patch This task is to enable vectorization support for Hybrid Grace Hash Join feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Hsu updated HIVE-9664: -- Description: Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ was: Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: hive, patch Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9891) LLAP: disable plan caching
[ https://issues.apache.org/jira/browse/HIVE-9891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9891. -- Resolution: Fixed Committed to branch. LLAP: disable plan caching -- Key: HIVE-9891 URL: https://issues.apache.org/jira/browse/HIVE-9891 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9891.1.patch Can't share the same plan objects in LLAP as they are used concurrently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381007#comment-14381007 ] Sushanth Sowmyan commented on HIVE-9582: +cc [~venkatnrangan] HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10085) Lateral view on top of a view throws RuntimeException
[
https://issues.apache.org/jira/browse/HIVE-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381066#comment-14381066
]
Hive QA commented on HIVE-10085:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707263/HIVE-10085.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8338 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt10
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3155/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3155/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707263 - PreCommit-HIVE-TRUNK-Build
Lateral view on top of a view throws RuntimeException
-
Key: HIVE-10085
URL: https://issues.apache.org/jira/browse/HIVE-10085
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-10085.patch
Following the following sqls to create table and view and execute a select
statement. It will throw the runtime exception:
{noformat}
FAILED: RuntimeException
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: map or list is
expected at function SIZE, but int is found
{noformat}
{noformat}
CREATE TABLE t1( symptom STRING, pattern ARRAYINT, occurrence INT, index
INT);
CREATE OR REPLACE VIEW v1 AS
SELECT TRIM(pd.symptom) AS symptom, pd.index, pd.pattern, pd.occurrence,
pd.occurrence as cnt from t1 pd;
SELECT pattern_data.symptom, pattern_data.index, pattern_data.occurrence,
pattern_data.cnt, size(pattern_data.pattern) as pattern_length,
pattern.pattern_id
FROM v1 pattern_data LATERAL VIEW explode(pattern) pattern AS pattern_id;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Nag updated HIVE-9664: Attachment: HIVE-9664.5.patch Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: hive, patch Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-5771: - Labels: (was: TODOC14) Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Fix For: 0.14.0 Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9839) HiveServer2 leaks OperationHandle on async queries which fail at compile phase
[
https://issues.apache.org/jira/browse/HIVE-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381329#comment-14381329
]
Nemon Lou commented on HIVE-9839:
-
Seems that the failure is unrelated.
Running this on my local computer doesn't fail:
{quote}
mvn test -Phadoop-2 -Dtest=TestCliDriver -Dqfile=schemeAuthority.q
{quote}
The result:
{quote}
Running org.apache.hadoop.hive.cli.TestCliDriver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.08 sec - in
org.apache.hadoop.hive.cli.TestCliDriver
{quote}
Submit again.
HiveServer2 leaks OperationHandle on async queries which fail at compile phase
--
Key: HIVE-9839
URL: https://issues.apache.org/jira/browse/HIVE-9839
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.14.0, 0.13.1, 1.0.0, 1.1.0
Reporter: Nemon Lou
Assignee: Nemon Lou
Priority: Critical
Attachments: HIVE-9839.patch, HIVE-9839.patch, HIVE-9839.patch,
HIVE-9839.patch, OperationHandleMonitor.java
Using beeline to connect to HiveServer2.And type the following:
drop table if exists table_not_exists;
select * from table_not_exists;
There will be an OperationHandle object staying in HiveServer2's memory for
ever even after quit from beeline .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381331#comment-14381331 ] Lefty Leverenz commented on HIVE-5771: -- Doc note: *hive.optimize.constant.propagation* is now documented in the wiki, so I'm removing the TODOC14 label. * [Configuration Properties -- hive.optimize.constant.propagation | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.constant.propagation] Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Fix For: 0.14.0 Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10082) LLAP: UnwrappedRowContainer throws exceptions
[
https://issues.apache.org/jira/browse/HIVE-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-10082:
--
Attachment: HIVE-10082.1.patch
LLAP: UnwrappedRowContainer throws exceptions
-
Key: HIVE-10082
URL: https://issues.apache.org/jira/browse/HIVE-10082
Project: Hive
Issue Type: Bug
Affects Versions: llap
Reporter: Gopal V
Assignee: Gunther Hagleitner
Fix For: llap
Attachments: HIVE-10082.1.patch
TPC-DS Query27 runs with map-joins enabled results in errors originating from
these lines in UnwrappedRowContainer::unwrap()
{code}
for (int index : valueIndex) {
if (index = 0) {
unwrapped.add(currentKey == null ? null : currentKey[index]);
} else {
unwrapped.add(values.get(-index - 1));
}
}
{code}
{code}
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at
org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.unwrap(UnwrapRowContainer.java:79)
at
org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.first(UnwrapRowContainer.java:62)
at
org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.first(UnwrapRowContainer.java:33)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:754)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:341)
{code}
This is intermittent and does not cause query failures as the retries
succeed, but slows down the query by an entire wave due to the retry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10090) Add connection manager for Tephra
[
https://issues.apache.org/jira/browse/HIVE-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381255#comment-14381255
]
Hive QA commented on HIVE-10090:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707280/HIVE-10090.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3158/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3158/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3158/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3158/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/results/clientpositive/show_functions.q.out'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/thirdparty
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-jmh/target itests/hive-unit/target itests/custom-serde/target
itests/util/target itests/qtest-spark/target hcatalog/target
hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target contrib/target service/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientpositive/udf_months_between.q.out
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFMonthsBetween.java
ql/src/test/queries/clientpositive/udf_months_between.q
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMonthsBetween.java
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1669247.
At revision 1669247.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707280 - PreCommit-HIVE-TRUNK-Build
Add connection manager for Tephra
-
Key: HIVE-10090
URL: https://issues.apache.org/jira/browse/HIVE-10090
Project: Hive
Issue Type: Sub-task
Components: Metastore
Affects Versions: hbase-metastore-branch
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-10090.patch
The task is to create an implementation of HBaseConnection that will use
Tephra for transaction management.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9976) Possible race condition in DynamicPartitionPruner for 200ms tasks
[
https://issues.apache.org/jira/browse/HIVE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381320#comment-14381320
]
Hive QA commented on HIVE-9976:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707303/HIVE-9976.2.patch
{color:green}SUCCESS:{color} +1 8347 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3159/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3159/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3159/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707303 - PreCommit-HIVE-TRUNK-Build
Possible race condition in DynamicPartitionPruner for 200ms tasks
--
Key: HIVE-9976
URL: https://issues.apache.org/jira/browse/HIVE-9976
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 1.0.0
Reporter: Gopal V
Assignee: Siddharth Seth
Attachments: HIVE-9976.1.patch, HIVE-9976.2.patch,
llap_vertex_200ms.png
Race condition in the DynamicPartitionPruner between
DynamicPartitionPruner::processVertex() and
DynamicPartitionpruner::addEvent() for tasks which respond with both the
result and success in a single heartbeat sequence.
{code}
2015-03-16 07:05:01,589 ERROR [InputInitializer [Map 1] #0]
tez.DynamicPartitionPruner: Expecting: 1, received: 0
2015-03-16 07:05:01,590 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Vertex Input: store_sales initializer failed,
vertex=vertex_1424502260528_1113_4_04 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Incorrect event count in
dynamic parition pruning
{code}
!llap_vertex_200ms.png!
All 4 upstream vertices of Map 1 need to finish within ~200ms to trigger
this, which seems to be consistently happening with LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
