[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14582: -- Labels: TODOC2.2 (was: ) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717647#comment-15717647
]
Lefty Leverenz commented on HIVE-15321:
---
Presumably the new default for *hive.metastore.server.max.message.size* doesn't
need to be documented in the wiki, because the change just appends L to
"100*1024*1024" -- right?
* [Configuration Properties -- hive.metastore.server.max.message.size |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.server.max.message.size]
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[
https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717629#comment-15717629
]
Hive QA commented on HIVE-12504:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841584/HIVE-12504.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2397/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2397/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841584 - PreCommit-HIVE-Build
> TxnHandler.abortTxn() should check if already aborted to improve message
>
>
> Key: HIVE-12504
> URL: https://issues.apache.org/jira/browse/HIVE-12504
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-12504.01.patch, HIVE-12504.patch
>
>
> If the update statement misses, it would be useful for abortTxn() to
> run another query to see if the requested txnid is in Aborted state to
> improve error msg.
> Note: get a separate read_committed connection for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15227) Optimize join + gby into semijoin
[ https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717574#comment-15717574 ] Lefty Leverenz commented on HIVE-15227: --- Doc note: *hive.enable.semijoin.conversion* needs to be documented in the wiki for release 2.2.0. * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] Added a TODOC2.2 label. > Optimize join + gby into semijoin > - > > Key: HIVE-15227 > URL: https://issues.apache.org/jira/browse/HIVE-15227 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch, > HIVE-15227.4.patch, HIVE-15227.patch > > > Calcite has a rule which can do this transformation. Lets take advantage of > this since Hive has native Left semi join operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15227) Optimize join + gby into semijoin
[ https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-15227: -- Labels: TODOC2.2 (was: ) > Optimize join + gby into semijoin > - > > Key: HIVE-15227 > URL: https://issues.apache.org/jira/browse/HIVE-15227 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch, > HIVE-15227.4.patch, HIVE-15227.patch > > > Calcite has a rule which can do this transformation. Lets take advantage of > this since Hive has native Left semi join operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[
https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717565#comment-15717565
]
Hive QA commented on HIVE-12504:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841584/HIVE-12504.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2396/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2396/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841584 - PreCommit-HIVE-Build
> TxnHandler.abortTxn() should check if already aborted to improve message
>
>
> Key: HIVE-12504
> URL: https://issues.apache.org/jira/browse/HIVE-12504
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-12504.01.patch, HIVE-12504.patch
>
>
> If the update statement misses, it would be useful for abortTxn() to
> run another query to see if the requested txnid is in Aborted state to
> improve error msg.
> Note: get a separate read_committed connection for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15335) Fast Decimal
[
https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717495#comment-15717495
]
Hive QA commented on HIVE-15335:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841574/HIVE-15335.01.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 56 failed/errored test(s), 10891 tests
executed
*Failed tests:*
{noformat}
TestHiveDecimalWritable - did not produce a TEST-*.xml file (likely timed out)
(batchId=274)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_native]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_decimal]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_2] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_3] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_4] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_5] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_6] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_serde]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_boolean]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_char] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[serde_regex] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_round] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_3]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_4]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_5]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_6]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_round]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_round_2]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_decimal]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[extrapolate_part_stats_partial_ndv]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_date]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_decimal]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_timestamp]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_varchar]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_all_primitive]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_adaptor_usage_mode]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_data_types]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_3]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_4]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_5]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_6]
(batchId=138)
[jira] [Commented] (HIVE-15333) Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as ResultSet
[
https://issues.apache.org/jira/browse/HIVE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717427#comment-15717427
]
Hive QA commented on HIVE-15333:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841570/HIVE-15333.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10740 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=159)
[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_import]
(batchId=84)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_00_unsupported_schema]
(batchId=84)
org.apache.hadoop.hive.ql.TestReplicationScenarios.testBasic (batchId=205)
org.apache.hive.hcatalog.api.TestHCatClientNotification.createTable
(batchId=218)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testReplDumpResultSet (batchId=214)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2394/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2394/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2394/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841570 - PreCommit-HIVE-Build
> Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as
> ResultSet
> -
>
> Key: HIVE-15333
> URL: https://issues.apache.org/jira/browse/HIVE-15333
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15333.1.patch, HIVE-15333.2.patch
>
>
> We're writing the return values to a file, but we don't add FetchTask while
> planning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15346) "values temp table" should not be an input source for the query
[
https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717317#comment-15717317
]
Hive QA commented on HIVE-15346:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841565/HIVE-15346.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10743 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=97)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_emit_interval]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_values]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_with_schema]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_emit_interval]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_emit_interval]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_insertpart_noinspriv]
(batchId=84)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2393/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2393/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841565 - PreCommit-HIVE-Build
> "values temp table" should not be an input source for the query
> ---
>
> Key: HIVE-15346
> URL: https://issues.apache.org/jira/browse/HIVE-15346
> Project: Hive
> Issue Type: Sub-task
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15346.1.patch
>
>
> For "insert values()" query, a values tmp table is created. Such table is
> treated as input source. That seems to be incorrect since that should be
> treated internal and temporary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15057) Support other types of operators (other than SELECT)
[
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717231#comment-15717231
]
Hive QA commented on HIVE-15057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841564/HIVE-15057.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10741 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=158)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2392/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2392/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2392/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841564 - PreCommit-HIVE-Build
> Support other types of operators (other than SELECT)
>
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch, HIVE-15057.2.patch,
> HIVE-15057.3.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We
> should add support for other types of operators so the optimization can work
> for complex queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15345) Spelling errors in logging and exceptions for query language code
[
https://issues.apache.org/jira/browse/HIVE-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717113#comment-15717113
]
Hive QA commented on HIVE-15345:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841561/HIVE-15345.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10728 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[str_to_map] (batchId=57)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exchange_partition_neg_incomplete_partition]
(batchId=84)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2391/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2391/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2391/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841561 - PreCommit-HIVE-Build
> Spelling errors in logging and exceptions for query language code
> -
>
> Key: HIVE-15345
> URL: https://issues.apache.org/jira/browse/HIVE-15345
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
> Attachments: HIVE-15345.1.patch
>
>
> Obvious typos and misspellings in the exceptions and messages.
> modifified -> modified
> commnad -> command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14895) CompactorMR.CompactorOutputCommitter race condition
[ https://issues.apache.org/jira/browse/HIVE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14895: -- Status: Patch Available (was: Open) > CompactorMR.CompactorOutputCommitter race condition > --- > > Key: HIVE-14895 > URL: https://issues.apache.org/jira/browse/HIVE-14895 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14895.01.patch > > > commitJob() method renames each file in TMP_LOCATION separately. so someone > may read FINAL_LOCATION and see only some of the files that should be there. > Use FileSystem.getFileStatus(TMP_LOCATION) and just rename the dir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14895) CompactorMR.CompactorOutputCommitter race condition
[ https://issues.apache.org/jira/browse/HIVE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14895: -- Attachment: HIVE-14895.01.patch > CompactorMR.CompactorOutputCommitter race condition > --- > > Key: HIVE-14895 > URL: https://issues.apache.org/jira/browse/HIVE-14895 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14895.01.patch > > > commitJob() method renames each file in TMP_LOCATION separately. so someone > may read FINAL_LOCATION and see only some of the files that should be there. > Use FileSystem.getFileStatus(TMP_LOCATION) and just rename the dir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15341) Get work path instead of attempted task path in HiveHFileOutputFormat
[ https://issues.apache.org/jira/browse/HIVE-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716983#comment-15716983 ] Chaoyu Tang commented on HIVE-15341: The failed tests are not related, they are aged. > Get work path instead of attempted task path in HiveHFileOutputFormat > - > > Key: HIVE-15341 > URL: https://issues.apache.org/jira/browse/HIVE-15341 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-15341.patch > > > It would be more robust to use FileOutputCommitter.getWorkPath instead of > FileOutputCommitter.getTaskAttemptPath. > The getTaskAttemptPath is same as getWorkPath in MR2 new APIs but is missing > in MR1 old APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15344) Spelling errors in logging and exceptions for metastore and service directories
[
https://issues.apache.org/jira/browse/HIVE-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716962#comment-15716962
]
Hive QA commented on HIVE-15344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841559/HIVE-15344.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10728 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_public_create]
(batchId=84)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorize_grant_public]
(batchId=84)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2390/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2390/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2390/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841559 - PreCommit-HIVE-Build
> Spelling errors in logging and exceptions for metastore and service
> directories
> ---
>
> Key: HIVE-15344
> URL: https://issues.apache.org/jira/browse/HIVE-15344
> Project: Hive
> Issue Type: Bug
> Components: CLI, Metastore, Server Infrastructure
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
> Attachments: HIVE-15344.1.patch
>
>
> More spelling errors in logging and exception messages.
> verififcation -> verification
> indexxes -> indexes
> resouce -> resource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: (was: HIVE-15337.01.patch) > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: (was: HIVE-15337.01.patch) > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: HIVE-15337.01.patch > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch, HIVE-15337.01.patch, > HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: HIVE-15337.01.patch > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch, HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716935#comment-15716935 ] Matt McCline commented on HIVE-15335: - Current patch is a Work-In-Progress so the performance can be checked that still has Q file wrong results. I'll post a Review Board on a later patch when things are ready for review. > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: HIVE-15337.01.patch > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Status: Patch Available (was: Open) > Need to specify starttime when putting Compaction job into "attempted" state > > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15337.01.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15351) Disable vectorized VectorUDFAdaptor usage with non-column or constant parameters
[ https://issues.apache.org/jira/browse/HIVE-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716920#comment-15716920 ] Prasanth Jayachandran commented on HIVE-15351: -- +1, pending tests. > Disable vectorized VectorUDFAdaptor usage with non-column or constant > parameters > > > Key: HIVE-15351 > URL: https://issues.apache.org/jira/browse/HIVE-15351 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-15351.01.patch > > > Vectorization using VectorUDFAdaptor is broken and produces wrong results > when the parameter(s) have vectorized expressions that allocate scratch > columns. So, for now, we restrict VectorUDFAdaptor usage to columns or > constant expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15351) Disable vectorized VectorUDFAdaptor usage with non-column or constant parameters
[ https://issues.apache.org/jira/browse/HIVE-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15351: Attachment: HIVE-15351.01.patch > Disable vectorized VectorUDFAdaptor usage with non-column or constant > parameters > > > Key: HIVE-15351 > URL: https://issues.apache.org/jira/browse/HIVE-15351 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-15351.01.patch > > > Vectorization using VectorUDFAdaptor is broken and produces wrong results > when the parameter(s) have vectorized expressions that allocate scratch > columns. So, for now, we restrict VectorUDFAdaptor usage to columns or > constant expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15351) Disable vectorized VectorUDFAdaptor usage with non-column or constant parameters
[ https://issues.apache.org/jira/browse/HIVE-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15351: Status: Patch Available (was: Open) > Disable vectorized VectorUDFAdaptor usage with non-column or constant > parameters > > > Key: HIVE-15351 > URL: https://issues.apache.org/jira/browse/HIVE-15351 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-15351.01.patch > > > Vectorization using VectorUDFAdaptor is broken and produces wrong results > when the parameter(s) have vectorized expressions that allocate scratch > columns. So, for now, we restrict VectorUDFAdaptor usage to columns or > constant expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15348) Add more specific error codes to ReplicationSemanticAnalyzer's SemanticExceptions
[ https://issues.apache.org/jira/browse/HIVE-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15348: Status: Patch Available (was: Open) > Add more specific error codes to ReplicationSemanticAnalyzer's > SemanticExceptions > - > > Key: HIVE-15348 > URL: https://issues.apache.org/jira/browse/HIVE-15348 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15348.1.patch > > > Also remove TODOS/FIXME in ReplicationSemanticAnalyzer and create > corresponding jiras or fix minor isues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15348) Add more specific error codes to ReplicationSemanticAnalyzer's SemanticExceptions
[ https://issues.apache.org/jira/browse/HIVE-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15348: Attachment: HIVE-15348.1.patch > Add more specific error codes to ReplicationSemanticAnalyzer's > SemanticExceptions > - > > Key: HIVE-15348 > URL: https://issues.apache.org/jira/browse/HIVE-15348 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15348.1.patch > > > Also remove TODOS/FIXME in ReplicationSemanticAnalyzer and create > corresponding jiras or fix minor isues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15348) Add more specific error codes to ReplicationSemanticAnalyzer's SemanticExceptions
[ https://issues.apache.org/jira/browse/HIVE-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15348: Description: Also remove TODOS/FIXME in ReplicationSemanticAnalyzer and create corresponding jiras or fix minor isues > Add more specific error codes to ReplicationSemanticAnalyzer's > SemanticExceptions > - > > Key: HIVE-15348 > URL: https://issues.apache.org/jira/browse/HIVE-15348 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Also remove TODOS/FIXME in ReplicationSemanticAnalyzer and create > corresponding jiras or fix minor isues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[ https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12504: -- Attachment: HIVE-12504.01.patch > TxnHandler.abortTxn() should check if already aborted to improve message > > > Key: HIVE-12504 > URL: https://issues.apache.org/jira/browse/HIVE-12504 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-12504.01.patch, HIVE-12504.patch > > > If the update statement misses, it would be useful for abortTxn() to > run another query to see if the requested txnid is in Aborted state to > improve error msg. > Note: get a separate read_committed connection for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15343) Spelling errors in logging and exceptions for beeline, common, hbase-handler, hcatalog, llap-server, orc, serde and shims
[
https://issues.apache.org/jira/browse/HIVE-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716829#comment-15716829
]
Hive QA commented on HIVE-15343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841558/HIVE-15343.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10754 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=91)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=227)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2389/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2389/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2389/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841558 - PreCommit-HIVE-Build
> Spelling errors in logging and exceptions for beeline, common, hbase-handler,
> hcatalog, llap-server, orc, serde and shims
> -
>
> Key: HIVE-15343
> URL: https://issues.apache.org/jira/browse/HIVE-15343
> Project: Hive
> Issue Type: Bug
> Components: Beeline, HBase Handler, HCatalog, llap, ORC,
> Serializers/Deserializers, Shims
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
> Attachments: HIVE-15343.1.patch
>
>
> There are a set of misspelled words in the logs and exceptions.
> Wtaited -> Waited
> prioroty -> priority
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716807#comment-15716807 ] Owen O'Malley commented on HIVE-15335: -- Matt, Can you post a pull request please? > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13452) StatsOptimizer should return no rows on empty table with group by
[
https://issues.apache.org/jira/browse/HIVE-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716771#comment-15716771
]
Ashutosh Chauhan commented on HIVE-13452:
-
+1 pending test
> StatsOptimizer should return no rows on empty table with group by
> -
>
> Key: HIVE-13452
> URL: https://issues.apache.org/jira/browse/HIVE-13452
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13452.01.patch
>
>
> {code}
> create table t1 (a int);
> analyze table t1 compute statistics;
> analyze table t1 compute statistics for columns;
> select count(1) from t1 group by 1;
> set hive.compute.query.using.stats=true;
> select count(1) from t1 group by 1;
> {code}
> In both cases result set should be empty. However, with statsoptimizer on
> Hive returns one row with value 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15335: Attachment: HIVE-15335.01.patch Rebase. > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15335: Attachment: (was: HIVE-15335.01.patch) > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15335: Status: Patch Available (was: In Progress) > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15335) Fast Decimal
[ https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15335: Status: In Progress (was: Patch Available) > Fast Decimal > > > Key: HIVE-15335 > URL: https://issues.apache.org/jira/browse/HIVE-15335 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15335.01.patch > > > Replace HiveDecimal implementation that currently represents the decimal > internally as a BigDecimal with a faster version that does not allocate extra > objects > Replace HiveDecimalWritable implementation with a faster version that has new > mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and > stores the result as a fast decimal instead of a slow byte array containing a > serialized BigInteger. > Provide faster ways to serialize/deserialize decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[ https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12504: -- Status: Patch Available (was: Open) > TxnHandler.abortTxn() should check if already aborted to improve message > > > Key: HIVE-12504 > URL: https://issues.apache.org/jira/browse/HIVE-12504 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-12504.patch > > > If the update statement misses, it would be useful for abortTxn() to > run another query to see if the requested txnid is in Aborted state to > improve error msg. > Note: get a separate read_committed connection for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[ https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12504: -- Target Version/s: 2.2.0 (was: 1.3.0) > TxnHandler.abortTxn() should check if already aborted to improve message > > > Key: HIVE-12504 > URL: https://issues.apache.org/jira/browse/HIVE-12504 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-12504.patch > > > If the update statement misses, it would be useful for abortTxn() to > run another query to see if the requested txnid is in Aborted state to > improve error msg. > Note: get a separate read_committed connection for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12504) TxnHandler.abortTxn() should check if already aborted to improve message
[ https://issues.apache.org/jira/browse/HIVE-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12504: -- Attachment: HIVE-12504.patch > TxnHandler.abortTxn() should check if already aborted to improve message > > > Key: HIVE-12504 > URL: https://issues.apache.org/jira/browse/HIVE-12504 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-12504.patch > > > If the update statement misses, it would be useful for abortTxn() to > run another query to see if the requested txnid is in Aborted state to > improve error msg. > Note: get a separate read_committed connection for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15342) Add support for primary/foreign keys in HBase metastore
[
https://issues.apache.org/jira/browse/HIVE-15342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716724#comment-15716724
]
Hive QA commented on HIVE-15342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841548/HIVE-15342.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10765 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841548 - PreCommit-HIVE-Build
> Add support for primary/foreign keys in HBase metastore
> ---
>
> Key: HIVE-15342
> URL: https://issues.apache.org/jira/browse/HIVE-15342
> Project: Hive
> Issue Type: Improvement
> Components: HBase Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 2.2.0
>
> Attachments: HIVE-15342.patch
>
>
> When HIVE-13076 was committed the calls into the HBase metastore were stubbed
> out. We need to implement support for constraints in the HBase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15333) Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as ResultSet
[ https://issues.apache.org/jira/browse/HIVE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15333: Attachment: HIVE-15333.2.patch > Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as > ResultSet > - > > Key: HIVE-15333 > URL: https://issues.apache.org/jira/browse/HIVE-15333 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15333.1.patch, HIVE-15333.2.patch > > > We're writing the return values to a file, but we don't add FetchTask while > planning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15346) "values temp table" should not be an input source for the query
[ https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15346: Status: Patch Available (was: Open) patch-1: Minor change to the code. The patch filters the input sources by table names. If it matches values_tmp_table_name, then we will skip. Many unit tests are affected. > "values temp table" should not be an input source for the query > --- > > Key: HIVE-15346 > URL: https://issues.apache.org/jira/browse/HIVE-15346 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15346.1.patch > > > For "insert values()" query, a values tmp table is created. Such table is > treated as input source. That seems to be incorrect since that should be > treated internal and temporary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15346) "values temp table" should not be an input source for the query
[ https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15346: Description: For "insert values()" query, a values tmp table is created. Such table is treated as input source. That seems to be incorrect since that should be treated internal and temporary. > "values temp table" should not be an input source for the query > --- > > Key: HIVE-15346 > URL: https://issues.apache.org/jira/browse/HIVE-15346 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15346.1.patch > > > For "insert values()" query, a values tmp table is created. Such table is > treated as input source. That seems to be incorrect since that should be > treated internal and temporary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15346) "values temp table" should not be an input source for the query
[ https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15346: Attachment: HIVE-15346.1.patch > "values temp table" should not be an input source for the query > --- > > Key: HIVE-15346 > URL: https://issues.apache.org/jira/browse/HIVE-15346 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15346.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15346) "values temp table" should not be an input source for the query
[ https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15346: Summary: "values temp table" should not be an input source for the query (was: "values temp table" should not be an input) > "values temp table" should not be an input source for the query > --- > > Key: HIVE-15346 > URL: https://issues.apache.org/jira/browse/HIVE-15346 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15346) "values temp table" should not be an input
[ https://issues.apache.org/jira/browse/HIVE-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15346: Summary: "values temp table" should not be an input (was: Remove "values temp table" from input list) > "values temp table" should not be an input > -- > > Key: HIVE-15346 > URL: https://issues.apache.org/jira/browse/HIVE-15346 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15341) Get work path instead of attempted task path in HiveHFileOutputFormat
[
https://issues.apache.org/jira/browse/HIVE-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716596#comment-15716596
]
Hive QA commented on HIVE-15341:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841543/HIVE-15341.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10723 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=158)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=112)
[bucketmapjoin3.q,union_date.q,cbo_gby.q,auto_join31.q,auto_sortmerge_join_1.q,join_cond_pushdown_unqual1.q,ppd_outer_join3.q,bucket_map_join_spark3.q,union28.q,statsfs.q,escape_sortby1.q,leftsemijoin.q,groupby_multi_single_reducer3.q,union_remove_6.q,join29.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2387/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2387/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2387/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841543 - PreCommit-HIVE-Build
> Get work path instead of attempted task path in HiveHFileOutputFormat
> -
>
> Key: HIVE-15341
> URL: https://issues.apache.org/jira/browse/HIVE-15341
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15341.patch
>
>
> It would be more robust to use FileOutputCommitter.getWorkPath instead of
> FileOutputCommitter.getTaskAttemptPath.
> The getTaskAttemptPath is same as getWorkPath in MR2 new APIs but is missing
> in MR1 old APIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13911) load inpath fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716593#comment-15716593
]
Sahil Takiar commented on HIVE-13911:
-
[~hsubramaniyan] this patch states:
{code}
// If we do a rename for a non-local file, we will be transfering the original
// file permissions from source to the destination. Else, in case of mvFile()
where we
// copy from source to destination, we will inherit the destination's parent
group ownership.
{code}
Could you explain this in more detail? When renaming a file on the same
filesystem, say from a staging directory to the final table location, why would
we want to preserve the permissions of the src file? Wouldn't we want to just
set the permissions to be the same as the parent directory (e.g. inherit parent
permissions)?
> load inpath fails throwing org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13911
> URL: https://issues.apache.org/jira/browse/HIVE-13911
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13911.1.patch, HIVE-13911.2.patch,
> HIVE-13911.3.patch, HIVE-13911.4.patch, HIVE-13911.5.patch
>
>
> Similar to HIVE-13857
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15312) reduce logging in certain places
[ https://issues.apache.org/jira/browse/HIVE-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15312: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > reduce logging in certain places > > > Key: HIVE-15312 > URL: https://issues.apache.org/jira/browse/HIVE-15312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15312.01.patch, HIVE-15312.02.patch, > HIVE-15312.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15323) allow the user to turn off reduce-side SMB join
[ https://issues.apache.org/jira/browse/HIVE-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15323: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > allow the user to turn off reduce-side SMB join > > > Key: HIVE-15323 > URL: https://issues.apache.org/jira/browse/HIVE-15323 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15323.01.patch, HIVE-15323.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15278) PTF+MergeJoin = NPE
[
https://issues.apache.org/jira/browse/HIVE-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15278:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master
> PTF+MergeJoin = NPE
> ---
>
> Key: HIVE-15278
> URL: https://issues.apache.org/jira/browse/HIVE-15278
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-15278.patch
>
>
> Manifests as
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115)
> at
> org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:340)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:114)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 29 more
> {noformat}
> It's actually a somewhat subtle ordering problem in sortmerge - as it stands,
> it calls different branches of the tree in closeOp after they themselves have
> already been closed. Other operators that clean stuff up in close may result
> in different errors. The common pattern is
> {noformat}
>1125 at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
>1126 at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
>1127 at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:404)
> ...
>1131 at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:428)
>1132 at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:388)
>1133 at
> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
> ...
>1139 at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:294)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)
[ https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15057: Attachment: HIVE-15057.3.patch > Support other types of operators (other than SELECT) > > > Key: HIVE-15057 > URL: https://issues.apache.org/jira/browse/HIVE-15057 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Physical Optimizer >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15057.1.patch, HIVE-15057.2.patch, > HIVE-15057.3.patch > > > Currently only SELECT operators are supported for nested column pruning. We > should add support for other types of operators so the optimization can work > for complex queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15130) Fixing shifting index issue related to object inspector caching
[
https://issues.apache.org/jira/browse/HIVE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15130:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~Ferd] for the review!
> Fixing shifting index issue related to object inspector caching
> ---
>
> Key: HIVE-15130
> URL: https://issues.apache.org/jira/browse/HIVE-15130
> Project: Hive
> Issue Type: Sub-task
> Components: Reader
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-15130.1.patch, HIVE-15130.2.patch
>
>
> Hive caches object inspector in memory to avoid recreating the same object.
> However, in {{ArrayWritableObjectInspector}}, the {{equals}} and {{hashCode}}
> doesn't take account on all attributes of the class, therefore, even though
> the internal state (e.g., {{adjustedIndex}}) is different, it could still be
> mapped to the same result. This could cause wrong field index being used when
> reading Parquet data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716556#comment-15716556
]
Sergey Shelukhin commented on HIVE-15296:
-
[~gopalv] can you take a look?
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.patch, HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=memory:4096, vCores:1, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=memory:59392, vCores:16,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=memory:59392, vCores:16, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> creationTime=1479500401929, allocationTime=1479500403426,
> startTime=1479500403427, finishTime=1479500743249, timeTaken=339822,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=TASK_HEARTBEAT_ERROR,
> diagnostics=AttemptID:attempt_1478967587833_2622_1_06_31_0 Timed out
> after 300 secs, nodeHttpAddress=http://3n01:15002, counters=Counters: 1,
> org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1
> 2016-11-18 20:25:43,255 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Processing de-allocate request for
> task=attempt_1478967587833_2622_1_06_31_0, state=ASSIGNED, endReason=OTHER
> 2016-11-18 20:25:43,259 [INFO] [Dispatcher thread {Central}]
> |node.AMNodeImpl|: Attempt failed on node: 3n01:15001 TA:
> attempt_1478967587833_2622_1_06_31_0 failed: true container:
> container_1_2622_01_56 numFailedTAs: 7
> 2016-11-18 20:25:43,262 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Source task attempt completed for vertex:
> vertex_1478967587833_2622_1_07 [Reducer 2] attempt:
> attempt_1478967587833_2622_1_06_31_0 with state: FAILED vertexState:
> RUNNING
> {noformat}
> Second attempt:
> {noformat}
> 2016-11-18 20:25:43,267 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_1, priority=64,
> capability=memory:4096, vCores:1, hosts=null
> 2016-11-18 20:25:43,297 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for
[jira] [Commented] (HIVE-15345) Spelling errors in logging and exceptions for query language code
[ https://issues.apache.org/jira/browse/HIVE-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716552#comment-15716552 ] Prasanth Jayachandran commented on HIVE-15345: -- lgtm, +1. > Spelling errors in logging and exceptions for query language code > - > > Key: HIVE-15345 > URL: https://issues.apache.org/jira/browse/HIVE-15345 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15345.1.patch > > > Obvious typos and misspellings in the exceptions and messages. > modifified -> modified > commnad -> command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15345) Spelling errors in logging and exceptions for query language code
[ https://issues.apache.org/jira/browse/HIVE-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15345: -- Status: Patch Available (was: Open) > Spelling errors in logging and exceptions for query language code > - > > Key: HIVE-15345 > URL: https://issues.apache.org/jira/browse/HIVE-15345 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15345.1.patch > > > Obvious typos and misspellings in the exceptions and messages. > modifified -> modified > commnad -> command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15345) Spelling errors in logging and exceptions for query language code
[ https://issues.apache.org/jira/browse/HIVE-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15345: -- Attachment: HIVE-15345.1.patch Corrections to spelling mistakes in exception messages and output. > Spelling errors in logging and exceptions for query language code > - > > Key: HIVE-15345 > URL: https://issues.apache.org/jira/browse/HIVE-15345 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15345.1.patch > > > Obvious typos and misspellings in the exceptions and messages. > modifified -> modified > commnad -> command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15343) Spelling errors in logging and exceptions for beeline, common, hbase-handler, hcatalog, llap-server, orc, serde and shims
[ https://issues.apache.org/jira/browse/HIVE-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15343: -- Attachment: HIVE-15343.1.patch Fixes for misspellings. > Spelling errors in logging and exceptions for beeline, common, hbase-handler, > hcatalog, llap-server, orc, serde and shims > - > > Key: HIVE-15343 > URL: https://issues.apache.org/jira/browse/HIVE-15343 > Project: Hive > Issue Type: Bug > Components: Beeline, HBase Handler, HCatalog, llap, ORC, > Serializers/Deserializers, Shims >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15343.1.patch > > > There are a set of misspelled words in the logs and exceptions. > Wtaited -> Waited > prioroty -> priority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15344) Spelling errors in logging and exceptions for metastore and service directories
[ https://issues.apache.org/jira/browse/HIVE-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15344: -- Status: Patch Available (was: Open) > Spelling errors in logging and exceptions for metastore and service > directories > --- > > Key: HIVE-15344 > URL: https://issues.apache.org/jira/browse/HIVE-15344 > Project: Hive > Issue Type: Bug > Components: CLI, Metastore, Server Infrastructure >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15344.1.patch > > > More spelling errors in logging and exception messages. > verififcation -> verification > indexxes -> indexes > resouce -> resource -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15344) Spelling errors in logging and exceptions for metastore and service directories
[ https://issues.apache.org/jira/browse/HIVE-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15344: -- Attachment: HIVE-15344.1.patch Fixes for spelling errors. > Spelling errors in logging and exceptions for metastore and service > directories > --- > > Key: HIVE-15344 > URL: https://issues.apache.org/jira/browse/HIVE-15344 > Project: Hive > Issue Type: Bug > Components: CLI, Metastore, Server Infrastructure >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15344.1.patch > > > More spelling errors in logging and exception messages. > verififcation -> verification > indexxes -> indexes > resouce -> resource -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15343) Spelling errors in logging and exceptions for beeline, common, hbase-handler, hcatalog, llap-server, orc, serde and shims
[ https://issues.apache.org/jira/browse/HIVE-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HIVE-15343: -- Status: Patch Available (was: Open) > Spelling errors in logging and exceptions for beeline, common, hbase-handler, > hcatalog, llap-server, orc, serde and shims > - > > Key: HIVE-15343 > URL: https://issues.apache.org/jira/browse/HIVE-15343 > Project: Hive > Issue Type: Bug > Components: Beeline, HBase Handler, HCatalog, llap, ORC, > Serializers/Deserializers, Shims >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Attachments: HIVE-15343.1.patch > > > There are a set of misspelled words in the logs and exceptions. > Wtaited -> Waited > prioroty -> priority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15334) HIVE-13945 changed scale rules for division
[ https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716220#comment-15716220 ] Sergey Shelukhin commented on HIVE-15334: - +1 > HIVE-13945 changed scale rules for division > --- > > Key: HIVE-15334 > URL: https://issues.apache.org/jira/browse/HIVE-15334 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch, > HIVE-15334.3.patch > > > Looks like HIVE-13945 change the decimal division precision/scale rules - the > explanation being "Changed the default decimal precision in division, not > sure why it was so low by default." > (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403) > As far as I can tell this causes decimal division to have a minimum scale of > 18. > cc [~sershe] - the rules that were in place were based on the SQL Server > precision/scale rules in > https://msdn.microsoft.com/en-us/library/ms190476.aspx > I'd like to revert this change to precision/scale rules -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15334) HIVE-13945 changed scale rules for division
[ https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716202#comment-15716202 ] Jason Dere edited comment on HIVE-15334 at 12/2/16 8:33 PM: Failures already have outstanding bugs for them. [~sershe] can you review? was (Author: jdere): Failures already have outstanding bugs for them. [~sershe] can you look? > HIVE-13945 changed scale rules for division > --- > > Key: HIVE-15334 > URL: https://issues.apache.org/jira/browse/HIVE-15334 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch, > HIVE-15334.3.patch > > > Looks like HIVE-13945 change the decimal division precision/scale rules - the > explanation being "Changed the default decimal precision in division, not > sure why it was so low by default." > (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403) > As far as I can tell this causes decimal division to have a minimum scale of > 18. > cc [~sershe] - the rules that were in place were based on the SQL Server > precision/scale rules in > https://msdn.microsoft.com/en-us/library/ms190476.aspx > I'd like to revert this change to precision/scale rules -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15334) HIVE-13945 changed scale rules for division
[ https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716202#comment-15716202 ] Jason Dere commented on HIVE-15334: --- Failures already have outstanding bugs for them. [~sershe] can you look? > HIVE-13945 changed scale rules for division > --- > > Key: HIVE-15334 > URL: https://issues.apache.org/jira/browse/HIVE-15334 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch, > HIVE-15334.3.patch > > > Looks like HIVE-13945 change the decimal division precision/scale rules - the > explanation being "Changed the default decimal precision in division, not > sure why it was so low by default." > (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403) > As far as I can tell this causes decimal division to have a minimum scale of > 18. > cc [~sershe] - the rules that were in place were based on the SQL Server > precision/scale rules in > https://msdn.microsoft.com/en-us/library/ms190476.aspx > I'd like to revert this change to precision/scale rules -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716180#comment-15716180
]
Hive QA commented on HIVE-15296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841541/HIVE-15296.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2385/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2385/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2385/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841541 - PreCommit-HIVE-Build
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.patch, HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=memory:4096, vCores:1, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=memory:59392, vCores:16,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=memory:59392, vCores:16, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1,
[jira] [Updated] (HIVE-15342) Add support for primary/foreign keys in HBase metastore
[ https://issues.apache.org/jira/browse/HIVE-15342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-15342: -- Status: Patch Available (was: Open) The large size of the patch is due to the protobuf code. The first diff in the patch can be ignored when reviewing the changes. > Add support for primary/foreign keys in HBase metastore > --- > > Key: HIVE-15342 > URL: https://issues.apache.org/jira/browse/HIVE-15342 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.2.0 > > Attachments: HIVE-15342.patch > > > When HIVE-13076 was committed the calls into the HBase metastore were stubbed > out. We need to implement support for constraints in the HBase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15342) Add support for primary/foreign keys in HBase metastore
[ https://issues.apache.org/jira/browse/HIVE-15342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-15342: -- Attachment: HIVE-15342.patch This patch adds two new columns in the catalog column column family in the tables_table, one for the primary key and one for all foreign keys. Putting it in the tables_table allows us to automatically drop the keys when the table is dropped, and in the future will allow a single call to fetch table and keys efficiently. > Add support for primary/foreign keys in HBase metastore > --- > > Key: HIVE-15342 > URL: https://issues.apache.org/jira/browse/HIVE-15342 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.2.0 > > Attachments: HIVE-15342.patch > > > When HIVE-13076 was committed the calls into the HBase metastore were stubbed > out. We need to implement support for constraints in the HBase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15340) Different result from vectorized vs. non-vectorized DATEDIFF with scalar parameter of type TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15340: Description: When "datediff(fl_date, timestamp "2007-03-14 08:21:59")" is added to existing vectorized_date_funcs.q Q file, it produces different results for vectorized and non-vectorized. (was: When "datediff(fl_date, timestamp "2007-03-14 08:21:59")" is added to existing vectorized_date_funcs.q Q file, it produces different results for vectorized and non-vectorized.) Summary: Different result from vectorized vs. non-vectorized DATEDIFF with scalar parameter of type TIMESTAMP (was: Wrong result from vectorized DATEDIFF with scalar parameter of type TIMESTAMP) > Different result from vectorized vs. non-vectorized DATEDIFF with scalar > parameter of type TIMESTAMP > > > Key: HIVE-15340 > URL: https://issues.apache.org/jira/browse/HIVE-15340 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > When "datediff(fl_date, timestamp "2007-03-14 08:21:59")" is added to > existing vectorized_date_funcs.q Q file, it produces different results for > vectorized and non-vectorized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15334) HIVE-13945 changed scale rules for division
[
https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716036#comment-15716036
]
Hive QA commented on HIVE-15334:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841533/HIVE-15334.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10740 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=115)
[join39.q,bucketsortoptimize_insert_7.q,vector_distinct_2.q,join11.q,union13.q,dynamic_rdd_cache.q,auto_sortmerge_join_16.q,windowing.q,union_remove_3.q,skewjoinopt7.q,stats7.q,annotate_stats_join.q,multi_insert_lateral_view.q,ptf_streaming.q,join_1to1.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2384/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2384/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2384/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841533 - PreCommit-HIVE-Build
> HIVE-13945 changed scale rules for division
> ---
>
> Key: HIVE-15334
> URL: https://issues.apache.org/jira/browse/HIVE-15334
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch,
> HIVE-15334.3.patch
>
>
> Looks like HIVE-13945 change the decimal division precision/scale rules - the
> explanation being "Changed the default decimal precision in division, not
> sure why it was so low by default."
> (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403)
> As far as I can tell this causes decimal division to have a minimum scale of
> 18.
> cc [~sershe] - the rules that were in place were based on the SQL Server
> precision/scale rules in
> https://msdn.microsoft.com/en-us/library/ms190476.aspx
> I'd like to revert this change to precision/scale rules
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15327:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for reviewing [~ashutoshc]!
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.03.patch, HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> For the *second* query, current (incorrect) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1001 Bob 104 3 Fli
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 103 2 Ema
> 1001 Bob 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 102 2 Del
> 1001 Bob 102 2 Del
> {noformat}
> Expected (correct) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 1001 Bob 104 3 Fli
> 1001 Bob 103 2 Ema
> 1001 Bob 102 2 Del
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15341) Get work path instead of attempted task path in HiveHFileOutputFormat
[ https://issues.apache.org/jira/browse/HIVE-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15341: --- Attachment: HIVE-15341.patch > Get work path instead of attempted task path in HiveHFileOutputFormat > - > > Key: HIVE-15341 > URL: https://issues.apache.org/jira/browse/HIVE-15341 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-15341.patch > > > It would be more robust to use FileOutputCommitter.getWorkPath instead of > FileOutputCommitter.getTaskAttemptPath. > The getTaskAttemptPath is same as getWorkPath in MR2 new APIs but is missing > in MR1 old APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4166) closeAllForUGI causes failure in hiveserver2 when fetching large amount of data
[ https://issues.apache.org/jira/browse/HIVE-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716009#comment-15716009 ] Mithun Radhakrishnan commented on HIVE-4166: Argh. This patch has gone stale. I'll get a rebased version of this shortly. > closeAllForUGI causes failure in hiveserver2 when fetching large amount of > data > --- > > Key: HIVE-4166 > URL: https://issues.apache.org/jira/browse/HIVE-4166 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security, Shims >Affects Versions: 0.10.0, 0.11.0 >Reporter: Chris Drome >Assignee: Chris Drome > Attachments: HIVE-4166-0.10.patch, HIVE-4166-trunk.patch > > > HiveServer2 configured to use Kerberos authentication with doAs enabled > throws an exception when fetching a large amount of data from a query. > The exception is caused because FileSystem.closeAllForUGI is always called at > the end of TUGIAssumingProcessor.process. This affects requests on the > ResultSet for data from a SELECT query when the amount of data exceeds a > certain size. At that point any subsequent calls to fetch more data throw an > exception because the underlying DFSClient has been closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15341) Get work path instead of attempted task path in HiveHFileOutputFormat
[ https://issues.apache.org/jira/browse/HIVE-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15341: --- Status: Patch Available (was: Open) > Get work path instead of attempted task path in HiveHFileOutputFormat > - > > Key: HIVE-15341 > URL: https://issues.apache.org/jira/browse/HIVE-15341 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-15341.patch > > > It would be more robust to use FileOutputCommitter.getWorkPath instead of > FileOutputCommitter.getTaskAttemptPath. > The getTaskAttemptPath is same as getWorkPath in MR2 new APIs but is missing > in MR1 old APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15296:
Attachment: HIVE-15296.patch
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.patch, HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=memory:4096, vCores:1, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=memory:59392, vCores:16,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=memory:59392, vCores:16, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> creationTime=1479500401929, allocationTime=1479500403426,
> startTime=1479500403427, finishTime=1479500743249, timeTaken=339822,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=TASK_HEARTBEAT_ERROR,
> diagnostics=AttemptID:attempt_1478967587833_2622_1_06_31_0 Timed out
> after 300 secs, nodeHttpAddress=http://3n01:15002, counters=Counters: 1,
> org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1
> 2016-11-18 20:25:43,255 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Processing de-allocate request for
> task=attempt_1478967587833_2622_1_06_31_0, state=ASSIGNED, endReason=OTHER
> 2016-11-18 20:25:43,259 [INFO] [Dispatcher thread {Central}]
> |node.AMNodeImpl|: Attempt failed on node: 3n01:15001 TA:
> attempt_1478967587833_2622_1_06_31_0 failed: true container:
> container_1_2622_01_56 numFailedTAs: 7
> 2016-11-18 20:25:43,262 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Source task attempt completed for vertex:
> vertex_1478967587833_2622_1_07 [Reducer 2] attempt:
> attempt_1478967587833_2622_1_06_31_0 with state: FAILED vertexState:
> RUNNING
> {noformat}
> Second attempt:
> {noformat}
> 2016-11-18 20:25:43,267 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_1, priority=64,
> capability=memory:4096, vCores:1, hosts=null
> 2016-11-18 20:25:43,297 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
>
[jira] [Updated] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks
[ https://issues.apache.org/jira/browse/HIVE-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15242: Assignee: Siddharth Seth (was: Sergey Shelukhin) > LLAP: Act on Node update notifications from registry, fix isAlive checks > > > Key: HIVE-15242 > URL: https://issues.apache.org/jira/browse/HIVE-15242 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-15242.01.patch, HIVE-15242.02.patch, > HIVE-15242.03.patch, HIVE-15242.04.patch > > > isAlive checks are currently completely broken, since the ZK registry does > not update existing ServiceIsntances. Instead it creates new instances each > time. > This causes non-existant nodes to be used for scheduling in case of node > failures. > Also, act on the notifications sent by the registry about nodes going down, > or new nodes being added. (new nodes already handled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15338) Wrong result from vectorized DATEDIFF with scalar parameter of type DATE
[
https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715924#comment-15715924
]
Matt McCline commented on HIVE-15338:
-
https://issues.apache.org/jira/browse/HIVE-15340
> Wrong result from vectorized DATEDIFF with scalar parameter of type DATE
>
>
> Key: HIVE-15338
> URL: https://issues.apache.org/jira/browse/HIVE-15338
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15338.01.patch
>
>
> Vectorization in vectorized DATEDIFF when parameter is (date '12-01-16',
> column). Jason noted that ('12-01-16', column) treats the first parameter as
> a string and is a different case.
> Current Q file test vectorized_date_funcs.q DOES NOT test the date scalar
> type case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15338) Wrong result from vectorized DATEDIFF with scalar parameter of type DATE
[
https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715919#comment-15715919
]
Matt McCline commented on HIVE-15338:
-
Dropping the new timestamp stuff from Q file:
{code}
datediff(fl_date, timestamp "2007-03-14 00:00:00"),
datediff(fl_date, timestamp "2007-03-14 08:21:59")
{code}
and creating a separate issue.
> Wrong result from vectorized DATEDIFF with scalar parameter of type DATE
>
>
> Key: HIVE-15338
> URL: https://issues.apache.org/jira/browse/HIVE-15338
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15338.01.patch
>
>
> Vectorization in vectorized DATEDIFF when parameter is (date '12-01-16',
> column). Jason noted that ('12-01-16', column) treats the first parameter as
> a string and is a different case.
> Current Q file test vectorized_date_funcs.q DOES NOT test the date scalar
> type case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715881#comment-15715881
]
Hive QA commented on HIVE-15327:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841528/HIVE-15327.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10728 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2383/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2383/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2383/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841528 - PreCommit-HIVE-Build
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.03.patch, HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
> {noformat}
> 99 0 Alice NULL
[jira] [Updated] (HIVE-15334) HIVE-13945 changed scale rules for division
[ https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-15334: -- Attachment: HIVE-15334.3.patch resubmit patch due to timed out MiniLlapLocal test > HIVE-13945 changed scale rules for division > --- > > Key: HIVE-15334 > URL: https://issues.apache.org/jira/browse/HIVE-15334 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch, > HIVE-15334.3.patch > > > Looks like HIVE-13945 change the decimal division precision/scale rules - the > explanation being "Changed the default decimal precision in division, not > sure why it was so low by default." > (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403) > As far as I can tell this causes decimal division to have a minimum scale of > 18. > cc [~sershe] - the rules that were in place were based on the SQL Server > precision/scale rules in > https://msdn.microsoft.com/en-us/library/ms190476.aspx > I'd like to revert this change to precision/scale rules -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715733#comment-15715733
]
Jesus Camacho Rodriguez commented on HIVE-15251:
[~ashutoshc], could you review it? Thanks
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15251.patch
>
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15251:
---
Comment: was deleted
(was:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841527/HIVE-15251.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2382/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2382/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2382/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:13.489
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2382/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:13.492
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4e07546 HIVE-15242: LLAP: Act on Node update notifications from
registry, fix isAlive checks (Siddharth Seth, Sergey Shelukhin reviewed by
Prasanth Jayachandran)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/join_emit_interval.q
Removing ql/src/test/queries/clientpositive/mapjoin_emit_interval.q
Removing ql/src/test/results/clientpositive/join_emit_interval.q.out
Removing ql/src/test/results/clientpositive/llap/join_emit_interval.q.out
Removing ql/src/test/results/clientpositive/llap/mapjoin_emit_interval.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4e07546 HIVE-15242: LLAP: Act on Node update notifications from
registry, fix isAlive checks (Siddharth Seth, Sergey Shelukhin reviewed by
Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:14.445
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: itests/src/test/resources/testconfiguration.properties:180
error: itests/src/test/resources/testconfiguration.properties: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841527 - PreCommit-HIVE-Build)
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15251.patch
>
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between
[jira] [Commented] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715723#comment-15715723
]
Hive QA commented on HIVE-15251:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841527/HIVE-15251.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2382/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2382/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2382/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:13.489
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2382/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:13.492
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4e07546 HIVE-15242: LLAP: Act on Node update notifications from
registry, fix isAlive checks (Siddharth Seth, Sergey Shelukhin reviewed by
Prasanth Jayachandran)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/join_emit_interval.q
Removing ql/src/test/queries/clientpositive/mapjoin_emit_interval.q
Removing ql/src/test/results/clientpositive/join_emit_interval.q.out
Removing ql/src/test/results/clientpositive/llap/join_emit_interval.q.out
Removing ql/src/test/results/clientpositive/llap/mapjoin_emit_interval.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4e07546 HIVE-15242: LLAP: Act on Node update notifications from
registry, fix isAlive checks (Siddharth Seth, Sergey Shelukhin reviewed by
Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-02 17:22:14.445
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: itests/src/test/resources/testconfiguration.properties:180
error: itests/src/test/resources/testconfiguration.properties: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841527 - PreCommit-HIVE-Build
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15251.patch
>
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between 100 and 102
> AND
[jira] [Issue Comment Deleted] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15327:
---
Comment: was deleted
(was:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841469/HIVE-15327.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10714 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=93)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2381/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2381/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2381/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841469 - PreCommit-HIVE-Build)
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.03.patch, HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
>
[jira] [Updated] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15327:
---
Attachment: HIVE-15327.03.patch
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.03.patch, HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> For the *second* query, current (incorrect) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1001 Bob 104 3 Fli
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 103 2 Ema
> 1001 Bob 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 102 2 Del
> 1001 Bob 102 2 Del
> {noformat}
> Expected (correct) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 1001 Bob 104 3 Fli
> 1001 Bob 103 2 Ema
> 1001 Bob 102 2 Del
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15251:
---
Attachment: HIVE-15251.patch
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15251.patch
>
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15251:
---
Status: Patch Available (was: In Progress)
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15251.patch
>
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-15251) Provide support for complex expressions in ON clauses for OUTER joins
[
https://issues.apache.org/jira/browse/HIVE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-15251 started by Jesus Camacho Rodriguez.
--
> Provide support for complex expressions in ON clauses for OUTER joins
> -
>
> Key: HIVE-15251
> URL: https://issues.apache.org/jira/browse/HIVE-15251
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> Follow-up of HIVE-15211.
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for outer joins.
> This patch is an extension to overcome these restrictions. Follow-up work
> will focus on identifying some cases, such as disjunctive predicates, that
> can be run more efficiently than with a cartesian product.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 LEFT OUTER JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 RIGHT OUTER JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 FULL OUTER JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715188#comment-15715188
]
Hive QA commented on HIVE-15327:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841469/HIVE-15327.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10714 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=93)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2381/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2381/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2381/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841469 - PreCommit-HIVE-Build
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
> {noformat}
> 99 0 Alice NULLNULL
[jira] [Updated] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value
[
https://issues.apache.org/jira/browse/HIVE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15327:
---
Attachment: HIVE-15327.02.patch
> Outerjoin might produce wrong result depending on joinEmitInterval value
>
>
> Key: HIVE-15327
> URL: https://issues.apache.org/jira/browse/HIVE-15327
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-15327.01.patch, HIVE-15327.02.patch,
> HIVE-15327.patch
>
>
> If joinEmitInterval is smaller than the group size, outerjoins might produce
> records with NULL appended values multiple times (once per group).
> HIVE-4689 targeted the same problem. However, the fix does not seem to cover
> all cases (in particular, it will not apply to left outer joins with filter
> conditions on the left input). The solution in HIVE-4689 was to disable
> (override) joinEmitInterval value for those cases. This fix follows the same
> approach.
> To reproduce the problem:
> {code}
> set hive.strict.checks.cartesian.product=false;
> set hive.join.emit.interval=1;
> CREATE TABLE test1 (key INT, value INT, col_1 STRING);
> INSERT INTO test1 VALUES (99, 0, 'Alice');
> INSERT INTO test1 VALUES (99, 2, 'Mat');
> INSERT INTO test1 VALUES (100, 1, 'Bob');
> INSERT INTO test1 VALUES (101, 2, 'Car');
> CREATE TABLE test2 (key INT, value INT, col_2 STRING);
> INSERT INTO test2 VALUES (102, 2, 'Del');
> INSERT INTO test2 VALUES (103, 2, 'Ema');
> INSERT INTO test2 VALUES (104, 3, 'Fli');
> -- Equi-condition and condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.value=test2.value AND test1.key between 100 and 102)
> LIMIT 10;
> -- Condition on one input (left outer join)
> SELECT *
> FROM test1 LEFT OUTER JOIN test2
> ON (test1.key between 100 and 102)
> LIMIT 10;
> {code}
> For the *first* query, current (incorrect) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> Expected (correct) result is:
> {noformat}
> 99 0 Alice NULLNULLNULL
> 1001 Bob NULLNULLNULL
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 99 2 Mat NULLNULLNULL
> {noformat}
> For the *second* query, current (incorrect) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1001 Bob 104 3 Fli
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 103 2 Ema
> 1001 Bob 103 2 Ema
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> 1012 Car 102 2 Del
> 1001 Bob 102 2 Del
> {noformat}
> Expected (correct) result is:
> {noformat}
> 1012 Car 104 3 Fli
> 1012 Car 103 2 Ema
> 1012 Car 102 2 Del
> 1001 Bob 104 3 Fli
> 1001 Bob 103 2 Ema
> 1001 Bob 102 2 Del
> 99 2 Mat NULLNULLNULL
> 99 0 Alice NULLNULLNULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15334) HIVE-13945 changed scale rules for division
[
https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715032#comment-15715032
]
Hive QA commented on HIVE-15334:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841457/HIVE-15334.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10725 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,delete_where_non_partitioned.q,explainuser_1.q,multi_insert.q,tez_dml.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_reduce2.q,vector_interval_mapjoin.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,tez_union.q,cbo_rp_subq_not_in.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2380/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2380/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2380/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841457 - PreCommit-HIVE-Build
> HIVE-13945 changed scale rules for division
> ---
>
> Key: HIVE-15334
> URL: https://issues.apache.org/jira/browse/HIVE-15334
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch
>
>
> Looks like HIVE-13945 change the decimal division precision/scale rules - the
> explanation being "Changed the default decimal precision in division, not
> sure why it was so low by default."
> (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403)
> As far as I can tell this causes decimal division to have a minimum scale of
> 18.
> cc [~sershe] - the rules that were in place were based on the SQL Server
> precision/scale rules in
> https://msdn.microsoft.com/en-us/library/ms190476.aspx
> I'd like to revert this change to precision/scale rules
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT
[
https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714928#comment-15714928
]
Steve Loughran commented on HIVE-15326:
---
HIVE-15016 includes a fix for that, simply by changing the case statement to
consider 3.x as needing the same shims as 2.x
> Hive shims report Unrecognized Hadoop major version number:
> 3.0.0-alpha2-SNAPSHOT
> -
>
> Key: HIVE-15326
> URL: https://issues.apache.org/jira/browse/HIVE-15326
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
> Environment: Hadoop trunk branch
>Reporter: Steve Loughran
>
> Hive built against Hadoop 2 fails to run against Hadoop 3.x,
> declaring:{{Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT}}
> Refusing to play on Hadoop 3.x may actually be the correct behaviour, though
> ideally we've retained API compatibility to everything works (maybe with some
> CP tweaking).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15339) Prefetch column stats for fields needed in FilterSelectivityEstimator
[
https://issues.apache.org/jira/browse/HIVE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714800#comment-15714800
]
Rajesh Balamohan edited comment on HIVE-15339 at 12/2/16 11:12 AM:
---
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions. Prefetch is a wrong term mentioned in the jira heading. Patch tries
to send all the needed columns in same call, and in other side these columns
stats get cached in AggregateColStats. Any col stats call fired later fetches
the data from the cache itself making it faster.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
was (Author: rajesh.balamohan):
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions. Prefetch is a wrong term in the jira. Patch tries to send all the
needed columns in same call, and in other side these columns stats get cached
in AggregateColStats. Any col stats call fired later fetches the data from the
cache itself making it faster.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
> Prefetch column stats for fields needed in FilterSelectivityEstimator
> -
>
> Key: HIVE-15339
> URL: https://issues.apache.org/jira/browse/HIVE-15339
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15339.1.patch
>
>
> Based on query pattern, {{FilterSelectivityEstimator}} gets column statistics
> from metastore in multiple calls. For instance, in the following query, it
> ends up getting individual column statistics for for flights multiple number
> of times.
> When the table has large number of partitions, getting statistics for columns
> via multiple calls can be very expensive. This would adversely impact the
> overall compilation time. The following query took 14 seconds to compile.
> {noformat}
> SELECT COUNT(`flights`.`flightnum`) AS `cnt_flightnum_ok`,
> YEAR(`flights`.`dateofflight`) AS `yr_flightdate_ok`
> FROM `flights` as `flights`
> JOIN `airlines` ON (`flights`.`uniquecarrier` = `airlines`.`code`)
> JOIN `airports` as `source_airport` ON (`flights`.`origin` =
> `source_airport`.`iata`)
> JOIN `airports` as `dest_airport` ON (`flights`.`dest` =
> `dest_airport`.`iata`)
> GROUP BY YEAR(`flights`.`dateofflight`);
> {noformat}
> It may be helpful to club all columns that need statistics and fetch these
> details in single remote call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15339) Prefetch column stats for fields needed in FilterSelectivityEstimator
[
https://issues.apache.org/jira/browse/HIVE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714800#comment-15714800
]
Rajesh Balamohan edited comment on HIVE-15339 at 12/2/16 11:12 AM:
---
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions. Prefetch is a wrong term in the jira. Patch tries to send all the
needed columns in same call, and in other side these columns stats get cached
in AggregateColStats. Any col stats call fired later fetches the data from the
cache itself making it faster.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
was (Author: rajesh.balamohan):
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
> Prefetch column stats for fields needed in FilterSelectivityEstimator
> -
>
> Key: HIVE-15339
> URL: https://issues.apache.org/jira/browse/HIVE-15339
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15339.1.patch
>
>
> Based on query pattern, {{FilterSelectivityEstimator}} gets column statistics
> from metastore in multiple calls. For instance, in the following query, it
> ends up getting individual column statistics for for flights multiple number
> of times.
> When the table has large number of partitions, getting statistics for columns
> via multiple calls can be very expensive. This would adversely impact the
> overall compilation time. The following query took 14 seconds to compile.
> {noformat}
> SELECT COUNT(`flights`.`flightnum`) AS `cnt_flightnum_ok`,
> YEAR(`flights`.`dateofflight`) AS `yr_flightdate_ok`
> FROM `flights` as `flights`
> JOIN `airlines` ON (`flights`.`uniquecarrier` = `airlines`.`code`)
> JOIN `airports` as `source_airport` ON (`flights`.`origin` =
> `source_airport`.`iata`)
> JOIN `airports` as `dest_airport` ON (`flights`.`dest` =
> `dest_airport`.`iata`)
> GROUP BY YEAR(`flights`.`dateofflight`);
> {noformat}
> It may be helpful to club all columns that need statistics and fetch these
> details in single remote call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15334) HIVE-13945 changed scale rules for division
[ https://issues.apache.org/jira/browse/HIVE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-15334: -- Attachment: HIVE-15334.2.patch fix golden file diffs > HIVE-13945 changed scale rules for division > --- > > Key: HIVE-15334 > URL: https://issues.apache.org/jira/browse/HIVE-15334 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15334.1.patch, HIVE-15334.2.patch > > > Looks like HIVE-13945 change the decimal division precision/scale rules - the > explanation being "Changed the default decimal precision in division, not > sure why it was so low by default." > (https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403) > As far as I can tell this causes decimal division to have a minimum scale of > 18. > cc [~sershe] - the rules that were in place were based on the SQL Server > precision/scale rules in > https://msdn.microsoft.com/en-us/library/ms190476.aspx > I'd like to revert this change to precision/scale rules -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15339) Prefetch column stats for fields needed in FilterSelectivityEstimator
[
https://issues.apache.org/jira/browse/HIVE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714800#comment-15714800
]
Rajesh Balamohan edited comment on HIVE-15339 at 12/2/16 11:00 AM:
---
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
was (Author: rajesh.balamohan):
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
> Prefetch column stats for fields needed in FilterSelectivityEstimator
> -
>
> Key: HIVE-15339
> URL: https://issues.apache.org/jira/browse/HIVE-15339
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15339.1.patch
>
>
> Based on query pattern, {{FilterSelectivityEstimator}} gets column statistics
> from metastore in multiple calls. For instance, in the following query, it
> ends up getting individual column statistics for for flights multiple number
> of times.
> When the table has large number of partitions, getting statistics for columns
> via multiple calls can be very expensive. This would adversely impact the
> overall compilation time. The following query took 14 seconds to compile.
> {noformat}
> SELECT COUNT(`flights`.`flightnum`) AS `cnt_flightnum_ok`,
> YEAR(`flights`.`dateofflight`) AS `yr_flightdate_ok`
> FROM `flights` as `flights`
> JOIN `airlines` ON (`flights`.`uniquecarrier` = `airlines`.`code`)
> JOIN `airports` as `source_airport` ON (`flights`.`origin` =
> `source_airport`.`iata`)
> JOIN `airports` as `dest_airport` ON (`flights`.`dest` =
> `dest_airport`.`iata`)
> GROUP BY YEAR(`flights`.`dateofflight`);
> {noformat}
> It may be helpful to club all columns that need statistics and fetch these
> details in single remote call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15339) Prefetch column stats for fields needed in FilterSelectivityEstimator
[
https://issues.apache.org/jira/browse/HIVE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15339:
Attachment: HIVE-15339.1.patch
Attaching .1 patch.
{noformat}
Without any patch: (compile time 14.2 seconds)
==
2016-12-02T05:20:34,867 DEBUG [cf8155ce-cf85-41b5-b0a3-4d4a6c75da5e main]
log.PerfLogger:
With Patch: (compile time 10.6 seconds)
=
2016-12-02T05:34:53,820 DEBUG [bfe87e40-4260-4f67-9e84-cd89694be1ad main]
log.PerfLogger:
{noformat}
metastore DB was hosted in postgres and flights table has around 7000
partitions.
\cc [~pxiong], [~ashutoshc], [~jcamachorodriguez]
> Prefetch column stats for fields needed in FilterSelectivityEstimator
> -
>
> Key: HIVE-15339
> URL: https://issues.apache.org/jira/browse/HIVE-15339
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15339.1.patch
>
>
> Based on query pattern, {{FilterSelectivityEstimator}} gets column statistics
> from metastore in multiple calls. For instance, in the following query, it
> ends up getting individual column statistics for for flights multiple number
> of times.
> When the table has large number of partitions, getting statistics for columns
> via multiple calls can be very expensive. This would adversely impact the
> overall compilation time. The following query took 14 seconds to compile.
> {noformat}
> SELECT COUNT(`flights`.`flightnum`) AS `cnt_flightnum_ok`,
> YEAR(`flights`.`dateofflight`) AS `yr_flightdate_ok`
> FROM `flights` as `flights`
> JOIN `airlines` ON (`flights`.`uniquecarrier` = `airlines`.`code`)
> JOIN `airports` as `source_airport` ON (`flights`.`origin` =
> `source_airport`.`iata`)
> JOIN `airports` as `dest_airport` ON (`flights`.`dest` =
> `dest_airport`.`iata`)
> GROUP BY YEAR(`flights`.`dateofflight`);
> {noformat}
> It may be helpful to club all columns that need statistics and fetch these
> details in single remote call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15303) Upgrade to Druid 0.9.2
[ https://issues.apache.org/jira/browse/HIVE-15303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15303: --- Assignee: slim bouguerra (was: Jesus Camacho Rodriguez) > Upgrade to Druid 0.9.2 > -- > > Key: HIVE-15303 > URL: https://issues.apache.org/jira/browse/HIVE-15303 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: slim bouguerra > > Upgrading to latest Druid release once it is done. HIVE-15277 has > dependencies on this new release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15303) Upgrade to Druid 0.9.2
[ https://issues.apache.org/jira/browse/HIVE-15303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15303: --- Component/s: Druid integration > Upgrade to Druid 0.9.2 > -- > > Key: HIVE-15303 > URL: https://issues.apache.org/jira/browse/HIVE-15303 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: slim bouguerra > > Upgrading to latest Druid release once it is done. HIVE-15277 has > dependencies on this new release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks
[ https://issues.apache.org/jira/browse/HIVE-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15242: - Component/s: llap > LLAP: Act on Node update notifications from registry, fix isAlive checks > > > Key: HIVE-15242 > URL: https://issues.apache.org/jira/browse/HIVE-15242 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15242.01.patch, HIVE-15242.02.patch, > HIVE-15242.03.patch, HIVE-15242.04.patch > > > isAlive checks are currently completely broken, since the ZK registry does > not update existing ServiceIsntances. Instead it creates new instances each > time. > This causes non-existant nodes to be used for scheduling in case of node > failures. > Also, act on the notifications sent by the registry about nodes going down, > or new nodes being added. (new nodes already handled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks
[ https://issues.apache.org/jira/browse/HIVE-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15242: - Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. > LLAP: Act on Node update notifications from registry, fix isAlive checks > > > Key: HIVE-15242 > URL: https://issues.apache.org/jira/browse/HIVE-15242 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15242.01.patch, HIVE-15242.02.patch, > HIVE-15242.03.patch, HIVE-15242.04.patch > > > isAlive checks are currently completely broken, since the ZK registry does > not update existing ServiceIsntances. Instead it creates new instances each > time. > This causes non-existant nodes to be used for scheduling in case of node > failures. > Also, act on the notifications sent by the registry about nodes going down, > or new nodes being added. (new nodes already handled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
