[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed
[
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438260#comment-16438260
]
yunfei liu commented on HIVE-16041:
---
create a RB
> HCatalog doesn't delete temp _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch,
> HIVE-16041.3.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table, a
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the
> job failed.
> {quote}
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:45
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:51
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 00:57
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 01:28
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18862: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438259#comment-16438259 ] Zoltan Haindrich commented on HIVE-18862: - pushed to master. Thank you [~abstractdog] for fixing this! [~vgarg] I think this would be good to have on branch-3 also - would it be ok to push it there also? > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438257#comment-16438257
]
Hive QA commented on HIVE-18986:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} standalone-metastore: The patch generated 0 new +
808 unchanged - 3 fixed = 808 total (was 811) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10197/dev-support/hive-personality.sh
|
| git revision | master / 24969a8 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10197/yetus/patch-asflicense-problems.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10197/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Table rename will run java.lang.StackOverflowError in dataNucleus if the
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch,
> HIVE-18986.3.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would
> fail with the following stack trace. The issue is datanucleus can't handle
> the query with lots of colName='c1' && colName='c2' && ... .
>
> 2018-03-13 17:19:52,770 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]:
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table:
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler:
> [pool-5-thread-200]: j
[jira] [Assigned] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18862: --- Assignee: Zoltan Haindrich (was: Laszlo Bodor) > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18862: --- Assignee: Laszlo Bodor (was: Zoltan Haindrich) > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[
https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438250#comment-16438250
]
Hive QA commented on HIVE-19133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918787/HIVE-19133.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 143 failed/errored test(s), 14194 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[parquet_alter_part_table_drop_columns]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_add_months_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_add_months_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_array_contains_wrong1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDr
[jira] [Updated] (HIVE-16144) CompactionInfo doesn't have equals/hashCode but used in Set
[ https://issues.apache.org/jira/browse/HIVE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-16144: - Attachment: HIVE-16144.02.patch > CompactionInfo doesn't have equals/hashCode but used in Set > --- > > Key: HIVE-16144 > URL: https://issues.apache.org/jira/browse/HIVE-16144 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-16144.01.patch, HIVE-16144.02.patch > > > CompactionTxnHandler.findPotentialCompactions() uses a Set > but CompactionInfo doesn't have equals/hashCode. > should do the same as CompactionInfo.compareTo() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16144) CompactionInfo doesn't have equals/hashCode but used in Set
[ https://issues.apache.org/jira/browse/HIVE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438238#comment-16438238 ] Igor Kryvenko commented on HIVE-16144: -- Patch #3. Fix checkstyle > CompactionInfo doesn't have equals/hashCode but used in Set > --- > > Key: HIVE-16144 > URL: https://issues.apache.org/jira/browse/HIVE-16144 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-16144.01.patch, HIVE-16144.02.patch > > > CompactionTxnHandler.findPotentialCompactions() uses a Set > but CompactionInfo doesn't have equals/hashCode. > should do the same as CompactionInfo.compareTo() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19106) Hive ZooKeeper Locking - Throw and Log
[
https://issues.apache.org/jira/browse/HIVE-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438237#comment-16438237
]
Igor Kryvenko commented on HIVE-19106:
--
Patch #3. Fix checkstyle
> Hive ZooKeeper Locking - Throw and Log
> --
>
> Key: HIVE-19106
> URL: https://issues.apache.org/jira/browse/HIVE-19106
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Igor Kryvenko
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-19106.01.patch, HIVE-19106.02.patch,
> HIVE-19106.03.patch
>
>
> {code:java}
> ...
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> LOG.error("Node " + name + " can not be deleted after " +
> numRetriesForUnLock + " attempts.");
> throw new LockException(e);
> }
> }
> {code}
> Do not log and throw. Only throw and move the message into the
> {{LockException}}. There is already {{error}} level logging up the stack.
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L492-L495
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19106) Hive ZooKeeper Locking - Throw and Log
[
https://issues.apache.org/jira/browse/HIVE-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-19106:
-
Attachment: HIVE-19106.03.patch
> Hive ZooKeeper Locking - Throw and Log
> --
>
> Key: HIVE-19106
> URL: https://issues.apache.org/jira/browse/HIVE-19106
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Igor Kryvenko
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-19106.01.patch, HIVE-19106.02.patch,
> HIVE-19106.03.patch
>
>
> {code:java}
> ...
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> LOG.error("Node " + name + " can not be deleted after " +
> numRetriesForUnLock + " attempts.");
> throw new LockException(e);
> }
> }
> {code}
> Do not log and throw. Only throw and move the message into the
> {{LockException}}. There is already {{error}} level logging up the stack.
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L492-L495
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18816) CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
[
https://issues.apache.org/jira/browse/HIVE-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-18816:
-
Attachment: HIVE-18816.02.patch
> CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
> --
>
> Key: HIVE-18816
> URL: https://issues.apache.org/jira/browse/HIVE-18816
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18816.01.patch, HIVE-18816.02.patch
>
>
> *Reproducer*
> {code:sql}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> CREATE TABLE table_acid(d int, tz timestamp with local time zone)
> clustered by (d) into 2 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> {code}
> *Error*
> {code:sql}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException:
> Unknown primitive type TIMESTAMPLOCALTZ
> {code}
> *Error stack*
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: Unknown primitive type TIMESTAMPLOCALTZ
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:906)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4788)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:389)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2314)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1985)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1687)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1438)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1427)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1345)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1319)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_101]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_101]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_101]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:?]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:?]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:?]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> [junit-4.11.jar:?]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> [junit-4.11.jar:?]
> at org.junit.runners.ParentRunner.runL
[jira] [Commented] (HIVE-18816) CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
[
https://issues.apache.org/jira/browse/HIVE-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438236#comment-16438236
]
Igor Kryvenko commented on HIVE-18816:
--
Patch #2. Fix checkstyle
> CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
> --
>
> Key: HIVE-18816
> URL: https://issues.apache.org/jira/browse/HIVE-18816
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18816.01.patch, HIVE-18816.02.patch
>
>
> *Reproducer*
> {code:sql}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> CREATE TABLE table_acid(d int, tz timestamp with local time zone)
> clustered by (d) into 2 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> {code}
> *Error*
> {code:sql}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException:
> Unknown primitive type TIMESTAMPLOCALTZ
> {code}
> *Error stack*
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: Unknown primitive type TIMESTAMPLOCALTZ
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:906)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4788)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:389)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2314)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1985)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1687)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1438)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1427)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1345)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1319)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_101]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_101]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_101]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:?]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:?]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:?]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> [junit-4.11.jar:?]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
> [hive-it-util-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> [junit-4.11.jar:?]
[jira] [Updated] (HIVE-19213) Values query: Stop opening transactions on _dummy_table
[ https://issues.apache.org/jira/browse/HIVE-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-19213: --- Description: When the -query-cache is hit- query is a values clause, the ACID TxnManager opens a txn + lock + heartbeats on _dummy_database._dummy_table. This is blocking values clause from jumping through the fast-paths for transactions. was: When the query-cache is hit, the ACID TxnManager opens a txn + lock + heartbeats on _dummy_database._dummy_table. This is blocking the query-cache from jumping through the fast-paths for transactions. > Values query: Stop opening transactions on _dummy_table > --- > > Key: HIVE-19213 > URL: https://issues.apache.org/jira/browse/HIVE-19213 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > When the -query-cache is hit- query is a values clause, the ACID TxnManager > opens a txn + lock + heartbeats on _dummy_database._dummy_table. > This is blocking values clause from jumping through the fast-paths for > transactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19213) Values query: Stop opening transactions on _dummy_table
[ https://issues.apache.org/jira/browse/HIVE-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438235#comment-16438235 ] Gopal V commented on HIVE-19213: This turned out to be related to metadata optimizer than the query-cache. > Values query: Stop opening transactions on _dummy_table > --- > > Key: HIVE-19213 > URL: https://issues.apache.org/jira/browse/HIVE-19213 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > When the -query-cache is hit- query is a values clause, the ACID TxnManager > opens a txn + lock + heartbeats on _dummy_database._dummy_table. > This is blocking values clause from jumping through the fast-paths for > transactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19213) Values query: Stop opening transactions on _dummy_table
[ https://issues.apache.org/jira/browse/HIVE-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-19213: --- Summary: Values query: Stop opening transactions on _dummy_table (was: QueryCache: Stop opening transactions on _dummy_table) > Values query: Stop opening transactions on _dummy_table > --- > > Key: HIVE-19213 > URL: https://issues.apache.org/jira/browse/HIVE-19213 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > When the query-cache is hit, the ACID TxnManager opens a txn + lock + > heartbeats on _dummy_database._dummy_table. > This is blocking the query-cache from jumping through the fast-paths for > transactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[
https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438234#comment-16438234
]
Hive QA commented on HIVE-19133:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10195/dev-support/hive-personality.sh
|
| git revision | master / 24969a8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10195/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10195/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10195/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HS2 WebUI phase-wise performance metrics not showing correctly
> --
>
> Key: HIVE-19133
> URL: https://issues.apache.org/jira/browse/HIVE-19133
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Web UI
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19133.1.patch, HIVE-19133.2.patch,
> HIVE-19133.3.patch, WebUI-compile time query metrics.png
>
>
> The query specific WebUI metrics (go to drilldown -> performance logging) are
> not showing up in the correct phase and are often mixed up.
> Attaching screenshot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18816) CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
[
https://issues.apache.org/jira/browse/HIVE-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438230#comment-16438230
]
Hive QA commented on HIVE-18816:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918794/HIVE-18816.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 14112 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_likeany_wrong1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_map_keys_arg_num]
(batchId=96)
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testMetastoreVersion
(batchId=227)
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionMatching
(batchId=227)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=238)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10194/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10194/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10194/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hiv
[jira] [Commented] (HIVE-19086) Write notifications in bulk only when commitTransaction actually commits
[ https://issues.apache.org/jira/browse/HIVE-19086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438217#comment-16438217 ] Alexander Kolbasov commented on HIVE-19086: --- The post-event listener that is used by Sentry doesn't need to know every ID. In this case it needs to know at least the max ID for the whole sequence of events. Is there any way this can be communicated to the non-transactional listener? > Write notifications in bulk only when commitTransaction actually commits > > > Key: HIVE-19086 > URL: https://issues.apache.org/jira/browse/HIVE-19086 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alexander Kolbasov >Assignee: Vihang Karajgaonkar >Priority: Major > > This is an optimization that is targeting reducing the amount of time the > global DB lock is held for notifications. > The idea is to collect all notifications and only push them when > commitTransaction() actually commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18816) CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
[
https://issues.apache.org/jira/browse/HIVE-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438216#comment-16438216
]
Hive QA commented on HIVE-18816:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 2 new + 101 unchanged - 0
fixed = 103 total (was 101) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10194/dev-support/hive-personality.sh
|
| git revision | master / 24969a8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10194/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10194/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10194/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CREATE TABLE (ACID) doesn't work with TIMESTAMPLOCALTZ column type
> --
>
> Key: HIVE-18816
> URL: https://issues.apache.org/jira/browse/HIVE-18816
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18816.01.patch
>
>
> *Reproducer*
> {code:sql}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> CREATE TABLE table_acid(d int, tz timestamp with local time zone)
> clustered by (d) into 2 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> {code}
> *Error*
> {code:sql}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException:
> Unknown primitive type TIMESTAMPLOCALTZ
> {code}
> *Error stack*
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: Unknown primitive type TIMESTAMPLOCALTZ
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:906)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4788)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:389)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.execut
[jira] [Updated] (HIVE-19104) When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-19104:
--
Attachment: HIVE-19104.6.patch
> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch,
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same
> JDBC url, and warehouse directory. This can cause problem in the tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19104) When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438215#comment-16438215
]
Peter Vary commented on HIVE-19104:
---
[~stakiar]: The {{WarehouseInstance}} explicitly sets the JDBC Url. Since it is
used in multiple test cases (for example replication scenario tests) and I am
not sure if it is intentional or not, I decided to keep the original JDBC url
in this case. So based on your earlier suggestion I created a method with which
we can explicitly forbid the url change.
Run every failed tests manually, and with the exception of
{{TestTriggersMoveWorkloadManager}} all of them was passed. This last one is a
known flaky
> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch,
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same
> JDBC url, and warehouse directory. This can cause problem in the tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19104) When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-19104:
--
Attachment: (was: HIVE-19104.6.patch)
> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch,
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same
> JDBC url, and warehouse directory. This can cause problem in the tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19104) When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-19104:
--
Attachment: HIVE-19104.6.patch
> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch,
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same
> JDBC url, and warehouse directory. This can cause problem in the tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438214#comment-16438214 ] Gopal V commented on HIVE-19214: Block-padding to false - the padding is irrelevant for the small stripe flushing & is just wasting space. LGTM +1 > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19214.1.patch, HIVE-19214.2.patch > > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19106) Hive ZooKeeper Locking - Throw and Log
[
https://issues.apache.org/jira/browse/HIVE-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438211#comment-16438211
]
Hive QA commented on HIVE-19106:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918790/HIVE-19106.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 13401 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,l
[jira] [Commented] (HIVE-19212) Fix findbugs yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438209#comment-16438209
]
Peter Vary commented on HIVE-19212:
---
I have tried your patch locally without the {{--dirty-workspace}} and was
working like charm. Since {{patchprocess}} directory is in {{.gitignore}} it
should not matter anyway. I would prefer to stick to your original idea. Maybe
[~szita] could help us out. He knows more about the whole stuff, and might be
able to easily pinpoint the issue here.
> Fix findbugs yetus pre-commit checks
>
>
> Key: HIVE-19212
> URL: https://issues.apache.org/jira/browse/HIVE-19212
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19212.1.patch
>
>
> Follow up from HIVE-18883, the committed patch isn't working and Findbugs is
> still not working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438207#comment-16438207
]
Peter Vary commented on HIVE-18883:
---
[~stakiar]: Something should be amiss here. The {{patchprocess}} directory
should be in {{.gitignore}}, so it should not be deleted. It is not telling I
know the answer :), just that it is strange :(
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19106) Hive ZooKeeper Locking - Throw and Log
[
https://issues.apache.org/jira/browse/HIVE-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438184#comment-16438184
]
Hive QA commented on HIVE-19106:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 32 unchanged - 0 fixed
= 33 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10193/dev-support/hive-personality.sh
|
| git revision | master / 24969a8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10193/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10193/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10193/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive ZooKeeper Locking - Throw and Log
> --
>
> Key: HIVE-19106
> URL: https://issues.apache.org/jira/browse/HIVE-19106
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Igor Kryvenko
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-19106.01.patch, HIVE-19106.02.patch
>
>
> {code:java}
> ...
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> LOG.error("Node " + name + " can not be deleted after " +
> numRetriesForUnLock + " attempts.");
> throw new LockException(e);
> }
> }
> {code}
> Do not log and throw. Only throw and move the message into the
> {{LockException}}. There is already {{error}} level logging up the stack.
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L492-L495
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16144) CompactionInfo doesn't have equals/hashCode but used in Set
[
https://issues.apache.org/jira/browse/HIVE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438169#comment-16438169
]
Hive QA commented on HIVE-16144:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918795/HIVE-16144.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 163 failed/errored test(s), 14196 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=50)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_create_table_does_not_exist]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad_db1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tablestatus]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_s
[jira] [Updated] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19214: - Attachment: HIVE-19214.2.patch > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19214.1.patch, HIVE-19214.2.patch > > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438156#comment-16438156 ] Prasanth Jayachandran commented on HIVE-19214: -- [~gopalv] can you please take a look? small patch. > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19214.1.patch, HIVE-19214.2.patch > > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19086) Write notifications in bulk only when commitTransaction actually commits
[
https://issues.apache.org/jira/browse/HIVE-19086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438155#comment-16438155
]
Vihang Karajgaonkar commented on HIVE-19086:
The above mentioned approach is very hard to implement because the EVENT_ID is
passed to the non-transactional listener in an in-memory structure derived from
the actual {{ListenerEvent}} sub-classes. If we delay the EVENT_ID generation
to the actual commit time, then we cannot pass this EVENT_ID to using the
existing mechanism.
[~akolb] [~thejas] Do you know what is the motivation of using this way to pass
the EVENT_ID to non-transactional listeners? Why is it needed in the first
place. I seems to me that the getCurrentNotificationID and
getNextNotificationEvent APIs are sufficient to find out new events in the
system. Why do we need to pass the EVENT_IDs to non-transactional listeners
this way?
> Write notifications in bulk only when commitTransaction actually commits
>
>
> Key: HIVE-19086
> URL: https://issues.apache.org/jira/browse/HIVE-19086
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> This is an optimization that is targeting reducing the amount of time the
> global DB lock is held for notifications.
> The idea is to collect all notifications and only push them when
> commitTransaction() actually commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19214: - Attachment: HIVE-19214.1.patch > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19214.1.patch > > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438146#comment-16438146
]
Ashutosh Chauhan commented on HIVE-19155:
-
[~vgarg] It will be good to have this in 3.0 as well.
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> {code}
> You can reproduce this using the following DDL
> {code}
> create database druid_test;
> use druid_test;
> create table test_table(`timecolumn` timestamp, `userid` string, `num_l`
> float);
> insert into test_table values ('2015-03-08 00:00:00', 'i1-start', 4);
> insert into test_table values ('2015-
[jira] [Updated] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19155:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Slim!
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> {code}
> You can reproduce this using the following DDL
> {code}
> create database druid_test;
> use druid_test;
> create table test_table(`timecolumn` timestamp, `userid` string, `num_l`
> float);
> insert into test_table values ('2015-03-08 00:00:00', 'i1-start', 4);
> insert into te
[jira] [Commented] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438141#comment-16438141 ] Prasanth Jayachandran commented on HIVE-19205: -- bq. Will batch size still be configurable somehow? Yes. It will be configurable for advanced use cases. But it will default to 10. bq. One can set compaction specific properties on a table The current expectation is that the table already exists. So I would guess any compactor specific properties would directly go to the table via alter command or during creation. The streaming connection API does not currently have a way to propagate custom properties to the table. > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > index, compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward compatibility, new streaming API will be in its own hive > module) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438142#comment-16438142
]
Ashutosh Chauhan commented on HIVE-19157:
-
+1
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
[ https://issues.apache.org/jira/browse/HIVE-19202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19202: -- Component/s: CBO > CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() > - > > Key: HIVE-19202 > URL: https://issues.apache.org/jira/browse/HIVE-19202 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Critical > Fix For: 2.1.1 > > Attachments: HIVE-19202.1.patch > > > I ran a query with join and group by with below settings, COB failed due to > NullPointerException in HiveAggregate.isBucketedInput() > set hive.execution.engine=tez; > set hive.cbo.costmodel.extended=true; > > In class HiveRelMdDistribution, we implemented below functions: > public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery > mq) > public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) > > But in HiveAggregate.isBucketedInput, the argument passed to distribution is > "this.getInput()" > , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438137#comment-16438137 ] Prasanth Jayachandran commented on HIVE-19210: -- Removed references of hcatalog.streaming from TestCompactor. Also included streaming module to packaging. > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch, HIVE-19210.2.patch, > HIVE-19210.3.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438131#comment-16438131 ] Eugene Koifman commented on HIVE-19210: --- +1 pending tests > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch, HIVE-19210.2.patch, > HIVE-19210.3.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19210: - Attachment: HIVE-19210.3.patch > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch, HIVE-19210.2.patch, > HIVE-19210.3.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438127#comment-16438127 ] Eugene Koifman commented on HIVE-19205: --- Will batch size still be configurable somehow? I think that could be useful for advanced use cases. One can set compaction specific properties on a table (https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties) but this may need another task here to make sure it's adequate/works as expected. > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > index, compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward compatibility, new streaming API will be in its own hive > module) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16144) CompactionInfo doesn't have equals/hashCode but used in Set
[
https://issues.apache.org/jira/browse/HIVE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438123#comment-16438123
]
Hive QA commented on HIVE-16144:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10192/dev-support/hive-personality.sh
|
| git revision | master / b14113b |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10192/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10192/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hive-unit standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10192/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CompactionInfo doesn't have equals/hashCode but used in Set
> ---
>
> Key: HIVE-16144
> URL: https://issues.apache.org/jira/browse/HIVE-16144
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-16144.01.patch
>
>
> CompactionTxnHandler.findPotentialCompactions() uses a Set
> but CompactionInfo doesn't have equals/hashCode.
> should do the same as CompactionInfo.compareTo()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19205 started by Prasanth Jayachandran. > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > index, compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward compatibility, new streaming API will be in its own hive > module) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438114#comment-16438114 ] Prasanth Jayachandran commented on HIVE-19210: -- Not moving mutate package. Also removed the streaming integration tester standalone utility. > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch, HIVE-19210.2.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19210: - Attachment: HIVE-19210.2.patch > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch, HIVE-19210.2.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438108#comment-16438108 ] Jason Dere commented on HIVE-19200: --- +1 > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1) In the Vectorizer, detect if data type conversion is needed between the > partition and the desired table schema. If so and LLAP I/O is enabled that > does encoded catching, then do not vectorize. Why? When LLAP I/O is in > encoded catching mode, it delivers VectorizedRowBatch (VRB) to the > VectorMapOperator instead of (object) rows. We currently do not have logic > for converting VRBs. So, we either get Wrong Results or more likely > ClassCastException on the expected vs actual ColumnVector columns. > 2) Cleaned up error message logic.that was suppressing the new message from > the EXPLAIN VECTORIZATION display. > 3) NOTE: Some of the SELECT statements in the schema_evol_test*.q are > commented out because I bumped into a another bug. I'll file that one soon > and add comments to the Q files. > --- > The longer-term solution can be done later in steps: > 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert > columns to different data types. This is needed when LLAP is doing its > encoding / caching and feeds VRBs to VectorMapOperator instead of rows. > Similar to what MapOperator does today, VectorMapOperator would need to be > enhanced to convert partition VRBs into the table schema VRBs that the vector > operator tree expect. > 2) Today, vectorization logic is strictly positional based. It insists that > the partition columns have the same names as the table schema. The > MapOperator (and ORC) does more general conversion that uses column names > instead of column position. We'd need to enhance all 3 classes to handle > column name based conversion. The 3 classes are: the new VRB-to-VRB > conversion class, VectorDeserializeRow, and VectorAssignRow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438109#comment-16438109
]
Hive QA commented on HIVE-18910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918792/HIVE-18910.30.patch
{color:green}SUCCESS:{color} +1 due to 17 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 13796 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2
[jira] [Updated] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19008: Attachment: HIVE-19008.1.patch > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438107#comment-16438107
]
slim bouguerra commented on HIVE-19155:
---
[~ashutoshc] sure wanna file a Jira will tackle it when am looking at the
failing tests due to dynamic partitions.
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> {code}
> You can reproduce this using the following DDL
> {code}
> create database druid_test;
> use druid_test;
> create table test_table(`timecolumn` timestamp, `userid` string, `num_l`
> float);
> insert into test_table values ('2015-03-08 00:00:00', 'i1-start', 4);
> insert in
[jira] [Updated] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19008: Target Version/s: 3.1.0 (was: 3.0.0) Status: Patch Available (was: Open) > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-19008: --- Assignee: Sahil Takiar > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438099#comment-16438099
]
Ashutosh Chauhan commented on HIVE-19155:
-
I see. +1 for this patch.
Seems like granularity column can be avoided altogether even in dynamic
partition op. But thats for another jira.
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> {code}
> You can reproduce this using the following DDL
> {code}
> create database druid_test;
> use druid_test;
> create table test_table(`timecolumn` timestamp, `userid` string, `num_l`
> float);
> insert into test_table values ('2015-03-08 00:00:00',
[jira] [Commented] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438094#comment-16438094
]
Eugene Koifman commented on HIVE-19199:
---
+1
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19199.1.patch
>
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438093#comment-16438093 ] Prasanth Jayachandran commented on HIVE-19210: -- [~ashutoshc]/[~ekoifman] can some please review this patch? This creates a new top level streaming module and copied code from hcatalog streaming API. Also deprecated the org.apache.hive.hcatalog.streaming package altogether. This will be the base patch for other subtasks in HIVE-19205 > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18741) Add support for Import into Acid table
[ https://issues.apache.org/jira/browse/HIVE-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18741: -- Status: Open (was: Patch Available) > Add support for Import into Acid table > -- > > Key: HIVE-18741 > URL: https://issues.apache.org/jira/browse/HIVE-18741 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18741.01.patch > > > This should follow Load Data approach (or use load data directly) > Note that import supports partition spec > Does import support loading files not created by Export? If so, similarly to > HIVE-19029 - should check for Acid meta columns and reject -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438077#comment-16438077 ] Eugene Koifman edited comment on HIVE-18739 at 4/13/18 11:20 PM: - patch 23 makes use of true temp temp table (session scoped) - this addresses security concerns. [~jdere] could you look at temp table enhancements? In general, partitioned temp tables are still not available - only in the limited case required for Export. was (Author: ekoifman): patch 23 makes use of true temp temp table (session scoped) - this addresses security concerns. > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438077#comment-16438077 ] Eugene Koifman edited comment on HIVE-18739 at 4/13/18 11:19 PM: - patch 23 makes use of true temp temp table (session scoped) - this addresses security concerns. was (Author: ekoifman): patch 23 makes use of true temp temp table (session scoped) - this should address security concerns. > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438077#comment-16438077 ] Eugene Koifman commented on HIVE-18739: --- patch 23 makes use of true temp temp table (session scoped) - this should address security concerns. > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18739: -- Attachment: HIVE-18739.23.patch > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18739: -- Attachment: (was: HIVE-18739.22.patch) > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18739: -- Attachment: HIVE-18739.22.patch > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, > HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, > HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, > HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, > HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19079:
Attachment: Expanded-Truncated-Query.png
Collapsed-Truncated-Query.png
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: Collapsed-Truncated-Query.png,
> Expanded-Truncated-Query.png, HIVE-19079.1.patch
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19079:
Attachment: HIVE-19079.1.patch
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19079:
Target Version/s: 3.1.0 (was: 3.0.0)
Status: Patch Available (was: Open)
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438042#comment-16438042
]
slim bouguerra commented on HIVE-19157:
---
[~ashutoshc] Done
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19160) Insert data into decimal column fails with Null Pointer Exception
[ https://issues.apache.org/jira/browse/HIVE-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19160: --- Attachment: HIVE-19160.2.patch > Insert data into decimal column fails with Null Pointer Exception > - > > Key: HIVE-19160 > URL: https://issues.apache.org/jira/browse/HIVE-19160 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19160.1.patch, HIVE-19160.2.patch > > > drop table if exists testDecimal; > create table testDecimal > (cIdTINYINT, > cBigIntDECIMAL, > cInt DECIMAL, > cSmallInt DECIMAL, > cTinyint DECIMAL); > insert into testDecimal values > (1, > 1234567890123456789, > 1234567890, > 12345, > 123); > insert into testDecimal values > (2, > 1, > 2, > 3, > 4); > The second insert fails with null pointer exception. > 2018-04-10T15:23:23,080 ERROR [5dba40ef-be49-4187-8a72-afbb46c41ecc main] > metastore.RetryingHMSHandler: java.lang.NullPointerException > at > org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318) > at > org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35) > at > org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1040) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7166) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy40.set_aggr_stats_for(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1870) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.setPartitionColumnStatistics(SessionHiveMetaStoreClient.java:395) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) > at com.sun.proxy.$Proxy41.setPartitionColumnStatistics(Unknown Source) > at > org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4171) > at > org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:179) > at > org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:83) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19157:
--
Attachment: HIVE-19157.4.patch
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19210: - Attachment: HIVE-19210.1.patch > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19210: - Status: Patch Available (was: Open) > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19210.1.patch > > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438030#comment-16438030 ] Prasanth Jayachandran commented on HIVE-19205: -- [~vgarg] Can this be included in 3.0.0 release? > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > index, compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward compatibility, new streaming API will be in its own hive > module) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19161: -- Attachment: HIVE-19161.4.patch > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.2.patch, > HIVE-19161.3.patch, HIVE-19161.4.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19161: -- Attachment: (was: HIVE-19161.4.patch) > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.2.patch, > HIVE-19161.3.patch, HIVE-19161.4.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19214: - Reporter: Gopal V (was: Prasanth Jayachandran) > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19157:
--
Attachment: HIVE-19157.3.patch
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19214) High throughput ingest ORC format
[ https://issues.apache.org/jira/browse/HIVE-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19214: > High throughput ingest ORC format > - > > Key: HIVE-19214 > URL: https://issues.apache.org/jira/browse/HIVE-19214 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Create delta files with all ORC overhead disabled (no index, no compression, > no dictionary). Compactor will recreate the orc files with index, compression > and dictionary encoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438019#comment-16438019
]
Aihua Xu commented on HIVE-19204:
-
patch-1: made some changes some Task classes so the error will print out to the
client console from driver class. I think the general guidance should be: use
LOG.error() to print to the HS2 log file; call setExpection() to let driver
print the final task failures to the console.
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438018#comment-16438018
]
Vihang Karajgaonkar commented on HIVE-19164:
These are my diffs in case anyone is interested.
{noformat}
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
index 6015405..4d26f3e 100644
---
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
+++
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
@@ -142,6 +142,7 @@ public void testVersionMatching () throws Exception {
ObjectStore.setSchemaVerified(false);
hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION, true);
+hiveConf = new HiveConf(this.getClass());
setVersion(hiveConf, metastoreSchemaInfo.getHiveSchemaVersion());
driver = DriverFactory.newDriver(hiveConf);
CommandProcessorResponse proc = driver.run("show tables");
@@ -191,37 +192,34 @@ public void testVersionCompatibility () throws Exception {
}
// write the given version to metastore
- private String getVersion(HiveConf conf) throws HiveMetaException {
+ private String getVersion(HiveConf conf) throws Exception {
return getMetaStoreVersion();
}
// write the given version to metastore
- private void setVersion(HiveConf conf, String version) throws
HiveMetaException {
+ private void setVersion(HiveConf conf, String version) throws Exception {
setMetaStoreVersion(version, "setVersion test");
}
// Load the version stored in the metastore db
- public String getMetaStoreVersion() throws HiveMetaException {
-ObjectStore objStore = new ObjectStore();
-objStore.setConf(hiveConf);
+ public String getMetaStoreVersion() throws HiveMetaException, MetaException {
+RawStore ms = HiveMetaStore.HMSHandler.getMSForConf(hiveConf);
try {
- return objStore.getMetaStoreSchemaVersion();
+ return ms.getMetaStoreSchemaVersion();
} catch (MetaException e) {
throw new HiveMetaException("Failed to get version", e);
}
}
// Store the given version and comment in the metastore
- public void setMetaStoreVersion(String newVersion, String comment) throws
HiveMetaException {
-ObjectStore objStore = new ObjectStore();
-objStore.setConf(hiveConf);
+ public void setMetaStoreVersion(String newVersion, String comment)
+ throws HiveMetaException, MetaException {
+RawStore ms = HiveMetaStore.HMSHandler.getMSForConf(hiveConf);
try {
- objStore.setMetaStoreSchemaVersion(newVersion, comment);
+ ms.setMetaStoreSchemaVersion(newVersion, comment);
} catch (MetaException e) {
throw new HiveMetaException("Failed to set version", e);
}
}
-
-
}
{noformat}
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438017#comment-16438017 ] Daniel Dai commented on HIVE-19161: --- The 4th revision includes: 1. Fix against Oracle/Postgres/MsSql 2. New criteria for enable information schema restriction (GenericUDFRestrictInformationSchema) 3. Address Thejas' review comments Include thrift changes as well to run through ptest. > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.2.patch, > HIVE-19161.3.patch, HIVE-19161.4.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19161: -- Attachment: HIVE-19161.4.patch > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.2.patch, > HIVE-19161.3.patch, HIVE-19161.4.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438016#comment-16438016
]
Vihang Karajgaonkar commented on HIVE-19164:
I don't know what changed which started to break the test but it seems to me
that the {{setVersion}} method in this test directly initializes the
ObjectStore and then calls setConf. However, that would not update the version
table since that code path gets executed via {{checkSchema}} method which gets
called when its instantiated via
{{HiveMetaStore.HMSHandler.getMSForConf(hiveConf)}}. Once I make that change
the test starts working for me.
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19204:
Status: Patch Available (was: Open)
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19204:
Attachment: HIVE-19204.1.patch
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19167) Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL)
[ https://issues.apache.org/jira/browse/HIVE-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438012#comment-16438012 ] Matt McCline commented on HIVE-19167: - Seems to be no related failures but lets wait until Hive QA tests have been stabilized for branch-3 work and rerunthen to verify. > Map data type doesn't keep the order of the key/values pairs as read (Part 2, > The Sequel or SQL) > --- > > Key: HIVE-19167 > URL: https://issues.apache.org/jira/browse/HIVE-19167 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19167.01.patch, HIVE-19167.02.patch > > > HIVE-19116: "Vectorization: Vector Map data type doesn't keep the order of > the key/values pairs as read" didn't fix all the places where HashMap is used > instead of LinkedHashMap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Attachment: HIVE-19200.01.patch > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1) In the Vectorizer, detect if data type conversion is needed between the > partition and the desired table schema. If so and LLAP I/O is enabled that > does encoded catching, then do not vectorize. Why? When LLAP I/O is in > encoded catching mode, it delivers VectorizedRowBatch (VRB) to the > VectorMapOperator instead of (object) rows. We currently do not have logic > for converting VRBs. So, we either get Wrong Results or more likely > ClassCastException on the expected vs actual ColumnVector columns. > 2) Cleaned up error message logic.that was suppressing the new message from > the EXPLAIN VECTORIZATION display. > 3) NOTE: Some of the SELECT statements in the schema_evol_test*.q are > commented out because I bumped into a another bug. I'll file that one soon > and add comments to the Q files. > --- > The longer-term solution can be done later in steps: > 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert > columns to different data types. This is needed when LLAP is doing its > encoding / caching and feeds VRBs to VectorMapOperator instead of rows. > Similar to what MapOperator does today, VectorMapOperator would need to be > enhanced to convert partition VRBs into the table schema VRBs that the vector > operator tree expect. > 2) Today, vectorization logic is strictly positional based. It insists that > the partition columns have the same names as the table schema. The > MapOperator (and ORC) does more general conversion that uses column names > instead of column position. We'd need to enhance all 3 classes to handle > column name based conversion. The 3 classes are: the new VRB-to-VRB > conversion class, VectorDeserializeRow, and VectorAssignRow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19194) TestDruidStorageHandler fails
[
https://issues.apache.org/jira/browse/HIVE-19194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438009#comment-16438009
]
slim bouguerra commented on HIVE-19194:
---
okay will add some logging lines to help understand actual issue.
> TestDruidStorageHandler fails
> -
>
> Key: HIVE-19194
> URL: https://issues.apache.org/jira/browse/HIVE-19194
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: slim bouguerra
>Priority: Major
>
> This tests fails randomly. If its not reproducible locally consider improving
> its stability since it does fail once in a while on Hive QA.
> {code}
> java.lang.AssertionError: expected:<0> but was:<1> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:743) at
> org.junit.Assert.assertEquals(Assert.java:118) at
> org.junit.Assert.assertEquals(Assert.java:555) at
> org.junit.Assert.assertEquals(Assert.java:542) at
> org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable(TestDruidStorageHandler.java:414)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438008#comment-16438008
]
slim bouguerra commented on HIVE-19155:
---
[~ashutoshc] the Granularity Column is used by the dynamic partition operator
stage in order to sort and send rows to the underline partition, we can use
either but to keep code simple i am using only one which is the actual
timestamp field.
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> {code}
> You can reproduce this using the following DDL
> {code}
> create database druid_test;
> use druid_test;
> create table test_table(`timecolumn`
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Attachment: (was: HIVE-19200.01.patch) > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1) In the Vectorizer, detect if data type conversion is needed between the > partition and the desired table schema. If so and LLAP I/O is enabled that > does encoded catching, then do not vectorize. Why? When LLAP I/O is in > encoded catching mode, it delivers VectorizedRowBatch (VRB) to the > VectorMapOperator instead of (object) rows. We currently do not have logic > for converting VRBs. So, we either get Wrong Results or more likely > ClassCastException on the expected vs actual ColumnVector columns. > 2) Cleaned up error message logic.that was suppressing the new message from > the EXPLAIN VECTORIZATION display. > 3) NOTE: Some of the SELECT statements in the schema_evol_test*.q are > commented out because I bumped into a another bug. I'll file that one soon > and add comments to the Q files. > --- > The longer-term solution can be done later in steps: > 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert > columns to different data types. This is needed when LLAP is doing its > encoding / caching and feeds VRBs to VectorMapOperator instead of rows. > Similar to what MapOperator does today, VectorMapOperator would need to be > enhanced to convert partition VRBs into the table schema VRBs that the vector > operator tree expect. > 2) Today, vectorization logic is strictly positional based. It insists that > the partition columns have the same names as the table schema. The > MapOperator (and ORC) does more general conversion that uses column names > instead of column position. We'd need to enhance all 3 classes to handle > column name based conversion. The 3 classes are: the new VRB-to-VRB > conversion class, VectorDeserializeRow, and VectorAssignRow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438006#comment-16438006
]
Hive QA commented on HIVE-18910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 23s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-10191/patches/PreCommit-HIVE-Build-10191.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10191/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Migrate to Murmur hash for shuffle and bucketing
>
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch,
> HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch,
> HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch,
> HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch,
> HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch,
> HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch,
> HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.27.patch,
> HIVE-18910.28.patch, HIVE-18910.29.patch, HIVE-18910.3.patch,
> HIVE-18910.30.patch, HIVE-18910.4.patch, HIVE-18910.5.patch,
> HIVE-18910.6.patch, HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing
> users so that they dont have to reload the existing tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19167) Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL)
[
https://issues.apache.org/jira/browse/HIVE-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437999#comment-16437999
]
Hive QA commented on HIVE-19167:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918775/HIVE-19167.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 13797 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=96)
[udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,msck_repair_3.q,exim_19_external_over_existing.q,serde_
[jira] [Updated] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm
[ https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19126: Attachment: HIVE-19126.5.patch > CachedStore: Use memory estimation to limit cache size during prewarm > - > > Key: HIVE-19126 > URL: https://issues.apache.org/jira/browse/HIVE-19126 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19126.1.patch, HIVE-19126.2.patch, > HIVE-19126.3.patch, HIVE-19126.4.patch, HIVE-19126.5.patch, HIVE-19126.5.patch > > > We can rely on > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java > to estimate memory of SharedCache. This jira addresses the size estimation > during prewarm, so that we can stop when we hit the memory limit. In a > follow-up jira, we will work on estimation/eviction after prewarm is > complete, so that we can keep the frequently used tables and their partitions > in cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19155) Day time saving cause Druid inserts to fail with org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping segments
[
https://issues.apache.org/jira/browse/HIVE-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437974#comment-16437974
]
Ashutosh Chauhan commented on HIVE-19155:
-
Change to normalize segment interval to utc looks correct to me since at query
time we send query from Hive to Druid in UTC. However, you also had a change to
compute bucket start time from timestamp, which was earlier from granularity
column. Whats this granularity column contains? If it just has truncated
timestamp, do we need this column ever? We can always compute it from timestamp
column.
> Day time saving cause Druid inserts to fail with
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments
> -
>
> Key: HIVE-19155
> URL: https://issues.apache.org/jira/browse/HIVE-19155
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19155.patch
>
>
> If you try to insert data around the daylight saving time hour the query
> fails with following exception
> {code}
> 2018-04-10T11:24:58,836 ERROR [065fdaa2-85f9-4e49-adaf-3dc14d51be90 main]
> exec.DDLTask: Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hive.druid.io.druid.java.util.common.UOE: Cannot add overlapping
> segments [2015-03-08T05:00:00.000Z/2015-03-09T05:00:00.000Z and
> 2015-03-09T04:00:00.000Z/2015-03-10T04:00:00.000Z] with the same version
> [2018-04-10T11:24:48.388-07:00]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:914)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:919)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4831)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:394)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2443)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2114)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1797)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1532)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:204)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> [hive-cli-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> [hive-it-util-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437949#comment-16437949
]
Ashutosh Chauhan commented on HIVE-19157:
-
yes.. it tried to run those tests but those timed out.. likely a temp glitch in
tests.
Can you create a RB for this?
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-19079:
---
Assignee: Sahil Takiar
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437934#comment-16437934 ] Sahil Takiar commented on HIVE-19008: - Right now we are displaying the Spark session id in the web ui, we should fix that so that we display the Hive session id as that is probably more relevant. > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Priority: Major > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437917#comment-16437917
]
Ashutosh Chauhan commented on HIVE-19164:
-
Lets make a test only change for now so that we get closer to green build while
investigation continues for when/how this changed.
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437916#comment-16437916
]
Vihang Karajgaonkar commented on HIVE-19077:
You might want to try uploading the same version of the patch until we find a
better solution to this JIRA. That way you will not lose the position in the
queue and ptest will only once run with the latest patch attached. May be a
better solution for this patch could be to improve pre-commit admin job to not
add the patch to the queue if it already is pending. That way the precommit job
will run once with the latest version of the patch at the time when its ready
to run.
For instance in this case without the patch, jenkins would have run a
pre-commit job for all the times you re-attached but always with the same last
version of patch attached. That would have wasted lot of time and resources
testing essentially same patch.
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19167) Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL)
[
https://issues.apache.org/jira/browse/HIVE-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437910#comment-16437910
]
Hive QA commented on HIVE-19167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} serde: The patch generated 0 new + 248 unchanged - 2
fixed = 248 total (was 250) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch beeline passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hcatalog-pig-adapter passed checkstyle
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 15 new + 905 unchanged - 19
fixed = 920 total (was 924) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10190/dev-support/hive-personality.sh
|
| git revision | master / 3915980 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10190/yetus/diff-checkstyle-ql.txt
|
| modules | C: serde beeline hcatalog/core hcatalog/hcatalog-pig-adapter ql U:
. |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10190/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Map data type doesn't keep the order of the key/values pairs as read (Part 2,
> The Sequel or SQL)
> ---
>
> Key: HIVE-19167
> URL: https://issues.apache.org/jira/browse/HIVE-19167
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HIVE-19167.01.patch, HIVE-19167.02.patch
>
>
> HIVE-19116: "Vectorization: Vector Map data type doesn't keep the order of
> the key/valu
[jira] [Commented] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437893#comment-16437893
]
Deepak Jaiswal commented on HIVE-19077:
---
[~vihangk1] I never tried reattaching same version of patch but can look into
it. The problem arises when a patch is large like mine (HIVE-18910).
Practically, any commit would touch a file or more in it. Thus, by the time,
the turn for my patch comes, it is no good as it wont apply due to merge
conflicts.
I have already lost several runs due to it. To avoid this, what I do is I
refresh my patch every day in the evening to see if it applies, if it does not,
I fix the conflicts and upload it.
Once the queue finishes, hopefully over the weekend, I will try to see if a
patch with same version enqueues.
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
