[jira] [Updated] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13667:

Attachment: HIVE-13667.3.patch

Attaching .3 patch.

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch, HIVE-13667.2.patch, 
> HIVE-13667.3.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15743) vectorized text parsing: speed up double parse

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849604#comment-15849604
 ] 

Teddy Choi commented on HIVE-15743:
---

[~gopalv], that's a practical idea!

> vectorized text parsing: speed up double parse
> --
>
> Key: HIVE-15743
> URL: https://issues.apache.org/jira/browse/HIVE-15743
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Teddy Choi
> Attachments: HIVE-15743.1.patch, HIVE-15743.2.patch, tpch-without.png
>
>
> {noformat}
> Double.parseDouble(
> new String(bytes, fieldStart, fieldLength, 
> StandardCharsets.UTF_8));{noformat}
> This takes ~25% of the query time in some cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15632) Hive/Druid integration: Incorrect result - Limit on timestamp disappears

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849602#comment-15849602
 ] 

Jesus Camacho Rodriguez commented on HIVE-15632:


I think we could keep it till we upgrade to Calcite 1.12.0 so we can keep track.

> Hive/Druid integration: Incorrect result - Limit on timestamp disappears
> 
>
> Key: HIVE-15632
> URL: https://issues.apache.org/jira/browse/HIVE-15632
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> This can be observed with the following query:
> {code:sql}
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset_hive
> ORDER BY `__time` ASC
> LIMIT 10;
> {code}
> Query is translated correctly to Druid _timeseries_, but _limit_ operator 
> disappears.
> {code}
> OK
> Plan optimized by CBO.
> Stage-0
>   Fetch Operator
> limit:-1
> Select Operator [SEL_1]
>   Output:["_col0"]
>   TableScan [TS_0]
> 
> Output:["__time"],properties:{"druid.query.json":"{\"queryType\":\"timeseries\",\"dataSource\":\"druid_tpcds_ss_sold_time_subset\",\"descending\":false,\"granularity\":\"NONE\",\"aggregations\":[],\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"]}","druid.query.type":"timeseries"}
> {code}
> Thus, result has more than 10 rows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849591#comment-15849591
 ] 

Hive QA commented on HIVE-13667:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850583/HIVE-13667.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11023 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
 (batchId=153)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testEscapedStrings (batchId=217)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=217)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testNonAsciiStrings (batchId=217)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3320/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3320/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3320/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850583 - PreCommit-HIVE-Build

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch, HIVE-13667.2.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849561#comment-15849561
 ] 

Siddharth Seth commented on HIVE-13667:
---

The hascode and equals can use the worker identity alone. Does not need to be 
all the fields. More efficient and, I'm not completely sure about the 
correctness of Objects.hashCode in this case (ServiceRecord does not implement 
a hashCode or equals)
{code}
public String getWorkerIdentity() {
  com.google.common.base.Objects.hashCode(srv, null);
  return srv.get(UNIQUE_IDENTIFIER);
}
{code}

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch, HIVE-13667.2.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15791) Remove unused ant files

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner reassigned HIVE-15791:
-


> Remove unused ant files
> ---
>
> Key: HIVE-15791
> URL: https://issues.apache.org/jira/browse/HIVE-15791
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849541#comment-15849541
 ] 

Lefty Leverenz commented on HIVE-15784:
---

[~mmccline], the parameter descriptions need to be updated -- currently they 
both say "The default value is false."

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15790) Remove unused beeline golden files

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-15790:
--
Status: Patch Available  (was: Open)

> Remove unused beeline golden files
> --
>
> Key: HIVE-15790
> URL: https://issues.apache.org/jira/browse/HIVE-15790
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15790.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15790) Remove unused beeline golden files

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-15790:
--
Attachment: HIVE-15790.1.patch

> Remove unused beeline golden files
> --
>
> Key: HIVE-15790
> URL: https://issues.apache.org/jira/browse/HIVE-15790
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15790.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15790) Remove unused beeline golden files

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner reassigned HIVE-15790:
-


> Remove unused beeline golden files
> --
>
> Key: HIVE-15790
> URL: https://issues.apache.org/jira/browse/HIVE-15790
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15765) Support bracketed comments

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-15765:
--
Status: Patch Available  (was: Open)

> Support bracketed comments
> --
>
> Key: HIVE-15765
> URL: https://issues.apache.org/jira/browse/HIVE-15765
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15765.1.patch, HIVE-15765.1.patch, 
> HIVE-15765.2.patch
>
>
> C-style comments are in the SQL spec as well as supported by all major DBs. 
> The are useful for inline annotation of the SQL. We should have them too.
> Example:
> {noformat}
> select
> /*+ MAPJOIN(a) */ /* mapjoin hint */
> a /* column */
> from foo join bar;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15765) Support bracketed comments

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-15765:
--
Status: Open  (was: Patch Available)

> Support bracketed comments
> --
>
> Key: HIVE-15765
> URL: https://issues.apache.org/jira/browse/HIVE-15765
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15765.1.patch, HIVE-15765.1.patch, 
> HIVE-15765.2.patch
>
>
> C-style comments are in the SQL spec as well as supported by all major DBs. 
> The are useful for inline annotation of the SQL. We should have them too.
> Example:
> {noformat}
> select
> /*+ MAPJOIN(a) */ /* mapjoin hint */
> a /* column */
> from foo join bar;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15765) Support bracketed comments

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-15765:
--
Attachment: HIVE-15765.2.patch

> Support bracketed comments
> --
>
> Key: HIVE-15765
> URL: https://issues.apache.org/jira/browse/HIVE-15765
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15765.1.patch, HIVE-15765.1.patch, 
> HIVE-15765.2.patch
>
>
> C-style comments are in the SQL spec as well as supported by all major DBs. 
> The are useful for inline annotation of the SQL. We should have them too.
> Example:
> {noformat}
> select
> /*+ MAPJOIN(a) */ /* mapjoin hint */
> a /* column */
> from foo join bar;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15473) Progress Bar on Beeline client

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-15473:
---
Attachment: HIVE-15473.7.patch

> Progress Bar on Beeline client
> --
>
> Key: HIVE-15473
> URL: https://issues.apache.org/jira/browse/HIVE-15473
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline, HiveServer2
>Affects Versions: 2.1.1
>Reporter: anishek
>Assignee: anishek
>Priority: Minor
> Attachments: HIVE-15473.2.patch, HIVE-15473.3.patch, 
> HIVE-15473.4.patch, HIVE-15473.5.patch, HIVE-15473.6.patch, 
> HIVE-15473.7.patch, screen_shot_beeline.jpg
>
>
> Hive Cli allows showing progress bar for tez execution engine as shown in 
> https://issues.apache.org/jira/secure/attachment/12678767/ux-demo.gif
> it would be great to have similar progress bar displayed when user is 
> connecting via beeline command line client as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849514#comment-15849514
 ] 

Hive QA commented on HIVE-15784:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850497/HIVE-15784.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 11023 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_text] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[structin] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_aggregate_9] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
 (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_bucket] 
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_cast_constant] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_2] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_4] 
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_round] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_distinct_2] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby4] 
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby6] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_3] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_mapjoin] 
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_mapjoin] 
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_mapjoin_reduce] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_mr_diff_schema_alias]
 (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_orderby_5] 
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce1] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce2] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce3] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_decimal]
 (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_string_concat] 
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_tablesample_rows] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_4] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_14] 
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_15] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_limit] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_date_funcs] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types]
 (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_shufflejoin] 
(batchId=67)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
 (batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
 (batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization2]
 (batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_2]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_aggregate_9]
 (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_bucket]
 (batchId=143)

[jira] [Updated] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15779:

Attachment: HIVE-15779.2.patch

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch, HIVE-15779.2.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15509) Add back the script + transform tests to minitez

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849494#comment-15849494
 ] 

Prasanth Jayachandran commented on HIVE-15509:
--

scriptfile1.q is taking 5 mins to run in minitez. Needs further investigation. 

> Add back the script + transform tests to minitez
> 
>
> Key: HIVE-15509
> URL: https://issues.apache.org/jira/browse/HIVE-15509
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15509.1.patch, HIVE-15509.1.patch, 
> HIVE-15509.2.patch
>
>
> Script operator cannot run in minillap and so was removed from the minillap 
> test suite. But tez supports script + transform. Add the removed tests back 
> to minitez test suite. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15779:

Status: Open  (was: Patch Available)

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15779:

Attachment: (was: HIVE-15779.2.patch)

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15779:

Attachment: HIVE-15779.2.patch

Attaching .2 patch to fix review comments and to address test case issue.

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch, HIVE-15779.2.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849489#comment-15849489
 ] 

Prasanth Jayachandran commented on HIVE-13667:
--

lgtm, +1

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch, HIVE-13667.2.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15388) HiveParser spends lots of time in parsing queries with lots of "("

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15388:
---
Status: Open  (was: Patch Available)

> HiveParser spends lots of time in parsing queries with lots of "("
> --
>
> Key: HIVE-15388
> URL: https://issues.apache.org/jira/browse/HIVE-15388
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15388.01.patch, HIVE-15388.02.patch, 
> HIVE-15388.03.patch, HIVE-15388.04.patch, hive-15388.stacktrace.txt
>
>
> Branch: apache-master (applicable with previous releases as well)
> Queries generated via tools can have lots of "(" for "AND/OR" conditions. 
> This causes huge delays in parsing phase when the number of expressions are 
> high.
> e.g
> {noformat}
> SELECT `iata`,
>`airport`,
>`city`,
>`state`,
>`country`,
>`lat`,
>`lon`
> FROM airports
> WHERE 
> ((`airports`.`airport`
>  = "Thigpen"
>   
>   OR `airports`.`airport` = "Astoria Regional")
>   
>  OR `airports`.`airport` = "Warsaw Municipal")
>   
> OR `airports`.`airport` = "John F Kennedy Memorial")
>  
> OR `airports`.`airport` = "Hall-Miller Municipal")
> 
> OR `airports`.`airport` = "Atqasuk")
>OR 
> `airports`.`airport` = "William B Hartsfield-Atlanta Intl")
>   OR 
> `airports`.`airport` = "Artesia Municipal")
>  OR 
> `airports`.`airport` = "Outagamie County Regional")
> OR 
> `airports`.`airport` = "Watertown Municipal")
>OR 
> `airports`.`airport` = "Augusta State")
>   OR 
> `airports`.`airport` = "Aurora Municipal")
>  OR 
> `airports`.`airport` = "Alakanuk")
> OR 
> `airports`.`airport` = "Austin Municipal")
>OR 
> `airports`.`airport` = "Auburn Municipal")
>   OR 
> `airports`.`airport` = "Auburn-Opelik")
>  OR 
> `airports`.`airport` = "Austin-Bergstrom International")
> OR 
> `airports`.`airport` = "Wausau Municipal")
>OR 
> `airports`.`airport` = "Mecklenburg-Brunswick Regional")
>   OR 
> `airports`.`airport` = "Alva Regional")
>  OR 
> `airports`.`airport` = "Asheville Regional")
> OR 
> `airports`.`airport` = "Avon Park Municipal")
>OR 
> `airports`.`airport` = "Wilkes-Barre/Scranton Intl")
>   OR 
> `airports`.`airport` = "Marana Northwest Regional")
>  OR 
> `airports`.`airport` = "Catalina")
> OR 
> `airports`.`airport` = "Washington Municipal")
>OR 
> `airports`.`airport` = "Wainwright")
>   OR `airports`.`airport` 
> = "West Memphis Municipal")
>  OR `airports`.`airport` 
> = "Arlington Municipal")
> OR `airports`.`airport` = 
> "Algona Municipal")
>OR `airports`.`airport` = 
> "Chandler")
>   OR `airports`.`airport` = 
> "Altus Municipal")
>  

[jira] [Updated] (HIVE-15388) HiveParser spends lots of time in parsing queries with lots of "("

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15388:
---
Status: Patch Available  (was: Open)

> HiveParser spends lots of time in parsing queries with lots of "("
> --
>
> Key: HIVE-15388
> URL: https://issues.apache.org/jira/browse/HIVE-15388
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15388.01.patch, HIVE-15388.02.patch, 
> HIVE-15388.03.patch, HIVE-15388.04.patch, hive-15388.stacktrace.txt
>
>
> Branch: apache-master (applicable with previous releases as well)
> Queries generated via tools can have lots of "(" for "AND/OR" conditions. 
> This causes huge delays in parsing phase when the number of expressions are 
> high.
> e.g
> {noformat}
> SELECT `iata`,
>`airport`,
>`city`,
>`state`,
>`country`,
>`lat`,
>`lon`
> FROM airports
> WHERE 
> ((`airports`.`airport`
>  = "Thigpen"
>   
>   OR `airports`.`airport` = "Astoria Regional")
>   
>  OR `airports`.`airport` = "Warsaw Municipal")
>   
> OR `airports`.`airport` = "John F Kennedy Memorial")
>  
> OR `airports`.`airport` = "Hall-Miller Municipal")
> 
> OR `airports`.`airport` = "Atqasuk")
>OR 
> `airports`.`airport` = "William B Hartsfield-Atlanta Intl")
>   OR 
> `airports`.`airport` = "Artesia Municipal")
>  OR 
> `airports`.`airport` = "Outagamie County Regional")
> OR 
> `airports`.`airport` = "Watertown Municipal")
>OR 
> `airports`.`airport` = "Augusta State")
>   OR 
> `airports`.`airport` = "Aurora Municipal")
>  OR 
> `airports`.`airport` = "Alakanuk")
> OR 
> `airports`.`airport` = "Austin Municipal")
>OR 
> `airports`.`airport` = "Auburn Municipal")
>   OR 
> `airports`.`airport` = "Auburn-Opelik")
>  OR 
> `airports`.`airport` = "Austin-Bergstrom International")
> OR 
> `airports`.`airport` = "Wausau Municipal")
>OR 
> `airports`.`airport` = "Mecklenburg-Brunswick Regional")
>   OR 
> `airports`.`airport` = "Alva Regional")
>  OR 
> `airports`.`airport` = "Asheville Regional")
> OR 
> `airports`.`airport` = "Avon Park Municipal")
>OR 
> `airports`.`airport` = "Wilkes-Barre/Scranton Intl")
>   OR 
> `airports`.`airport` = "Marana Northwest Regional")
>  OR 
> `airports`.`airport` = "Catalina")
> OR 
> `airports`.`airport` = "Washington Municipal")
>OR 
> `airports`.`airport` = "Wainwright")
>   OR `airports`.`airport` 
> = "West Memphis Municipal")
>  OR `airports`.`airport` 
> = "Arlington Municipal")
> OR `airports`.`airport` = 
> "Algona Municipal")
>OR `airports`.`airport` = 
> "Chandler")
>   OR `airports`.`airport` = 
> "Altus Municipal")
>  

[jira] [Updated] (HIVE-15388) HiveParser spends lots of time in parsing queries with lots of "("

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15388:
---
Attachment: HIVE-15388.04.patch

> HiveParser spends lots of time in parsing queries with lots of "("
> --
>
> Key: HIVE-15388
> URL: https://issues.apache.org/jira/browse/HIVE-15388
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15388.01.patch, HIVE-15388.02.patch, 
> HIVE-15388.03.patch, HIVE-15388.04.patch, hive-15388.stacktrace.txt
>
>
> Branch: apache-master (applicable with previous releases as well)
> Queries generated via tools can have lots of "(" for "AND/OR" conditions. 
> This causes huge delays in parsing phase when the number of expressions are 
> high.
> e.g
> {noformat}
> SELECT `iata`,
>`airport`,
>`city`,
>`state`,
>`country`,
>`lat`,
>`lon`
> FROM airports
> WHERE 
> ((`airports`.`airport`
>  = "Thigpen"
>   
>   OR `airports`.`airport` = "Astoria Regional")
>   
>  OR `airports`.`airport` = "Warsaw Municipal")
>   
> OR `airports`.`airport` = "John F Kennedy Memorial")
>  
> OR `airports`.`airport` = "Hall-Miller Municipal")
> 
> OR `airports`.`airport` = "Atqasuk")
>OR 
> `airports`.`airport` = "William B Hartsfield-Atlanta Intl")
>   OR 
> `airports`.`airport` = "Artesia Municipal")
>  OR 
> `airports`.`airport` = "Outagamie County Regional")
> OR 
> `airports`.`airport` = "Watertown Municipal")
>OR 
> `airports`.`airport` = "Augusta State")
>   OR 
> `airports`.`airport` = "Aurora Municipal")
>  OR 
> `airports`.`airport` = "Alakanuk")
> OR 
> `airports`.`airport` = "Austin Municipal")
>OR 
> `airports`.`airport` = "Auburn Municipal")
>   OR 
> `airports`.`airport` = "Auburn-Opelik")
>  OR 
> `airports`.`airport` = "Austin-Bergstrom International")
> OR 
> `airports`.`airport` = "Wausau Municipal")
>OR 
> `airports`.`airport` = "Mecklenburg-Brunswick Regional")
>   OR 
> `airports`.`airport` = "Alva Regional")
>  OR 
> `airports`.`airport` = "Asheville Regional")
> OR 
> `airports`.`airport` = "Avon Park Municipal")
>OR 
> `airports`.`airport` = "Wilkes-Barre/Scranton Intl")
>   OR 
> `airports`.`airport` = "Marana Northwest Regional")
>  OR 
> `airports`.`airport` = "Catalina")
> OR 
> `airports`.`airport` = "Washington Municipal")
>OR 
> `airports`.`airport` = "Wainwright")
>   OR `airports`.`airport` 
> = "West Memphis Municipal")
>  OR `airports`.`airport` 
> = "Arlington Municipal")
> OR `airports`.`airport` = 
> "Algona Municipal")
>OR `airports`.`airport` = 
> "Chandler")
>   OR `airports`.`airport` = 
> "Altus Municipal")
>   

[jira] [Commented] (HIVE-11394) Enhance EXPLAIN display for vectorization

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849472#comment-15849472
 ] 

Hive QA commented on HIVE-11394:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850582/HIVE-11394.098.patch

{color:green}SUCCESS:{color} +1 due to 160 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 10993 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=147)

[dynamic_semijoin_reduction.q,load_dyn_part5.q,vector_complex_join.q,orc_llap.q,vectorization_7.q,vectorization_pushdown.q,cbo_gby.q,mapjoin3.q,auto_sortmerge_join_1.q,lineage3.q,cross_product_check_1.q,cbo_join.q,vector_struct_in.q,bucketmapjoin3.q,current_date_timestamp.q,orc_ppd_schema_evol_2a.q,groupby2.q,schema_evol_text_vec_table.q,vectorized_join46.q,orc_ppd_date.q,create_merge_compressed.q,multiMapJoin1.q,vector_outer_join1.q,vector_char_simple.q,dynpart_sort_optimization_acid.q,having.q,leftsemijoin.q,special_character_in_tabnames_1.q,cte_mat_2.q,vectorization_8.q]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
 (batchId=139)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_inner_join]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join0]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join3]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join4]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_div0]
 (batchId=94)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_limit]
 (batchId=93)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in] 
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant]
 (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
 (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
 (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin]
 (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce]
 (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_14] 
(batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_16] 
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17] 
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_9] 
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_div0] 
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case] 
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_mapjoin] 
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_math_funcs]
 (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_nested_mapjoin]
 (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
 (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_string_funcs]
 (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs]
 (batchId=108)
org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testExprNodeBetweenWithDynamicValue
 

[jira] [Updated] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13667:

Attachment: HIVE-13667.2.patch

Attaching .2 patch to address review comments

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch, HIVE-13667.2.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14990) run all tests for MM tables and fix the issues that are found

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849404#comment-15849404
 ] 

Hive QA commented on HIVE-14990:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850538/HIVE-14990.12.patch

{color:green}SUCCESS:{color} +1 due to 15 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 937 failed/errored test(s), 10361 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=221)
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas] 
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
 (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_concatenate_indexed_table]
 (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2] 
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2_orc] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_3] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats] 
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype] 
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_6] 
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_1] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_1]
 (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
 (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_10] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_11] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_13] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_14] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_15] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_1] 
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_3] 
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_4] 
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_5] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_7] 
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
 (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1] 
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2] 
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3] 
(batchId=41)

[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-11394:

Status: Patch Available  (was: In Progress)

> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch, 
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch, 
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch, 
> HIVE-11394.091.patch, HIVE-11394.092.patch, HIVE-11394.093.patch, 
> HIVE-11394.094.patch, HIVE-11394.095.patch, HIVE-11394.096.patch, 
> HIVE-11394.097.patch, HIVE-11394.098.patch, HIVE-11394.09.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization 
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators.  E.g. Filter 
> Vectorization.  It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions.  E.g. 
> predicateExpression.  It includes all information of SUMMARY and OPERATOR, 
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization 
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION 
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for 
> GROUPBY operator: Data type struct of 
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:": 
> "false" which says a node has a GROUP BY with an AVG or some other aggregator 
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators 
> are row-mode.  I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at 
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.  
> Today, GROUP BY and FILE SINK are not native.  MAP JOIN and REDUCE SINK are 
> conditionally native.  FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
> Tez
> ...
>   Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: alltypesorc
>   Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
>   Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
> Group By Operator
>   keys: cint (type: int)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
>   Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false

[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-11394:

Attachment: HIVE-11394.098.patch

> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch, 
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch, 
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch, 
> HIVE-11394.091.patch, HIVE-11394.092.patch, HIVE-11394.093.patch, 
> HIVE-11394.094.patch, HIVE-11394.095.patch, HIVE-11394.096.patch, 
> HIVE-11394.097.patch, HIVE-11394.098.patch, HIVE-11394.09.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization 
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators.  E.g. Filter 
> Vectorization.  It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions.  E.g. 
> predicateExpression.  It includes all information of SUMMARY and OPERATOR, 
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization 
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION 
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for 
> GROUPBY operator: Data type struct of 
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:": 
> "false" which says a node has a GROUP BY with an AVG or some other aggregator 
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators 
> are row-mode.  I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at 
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.  
> Today, GROUP BY and FILE SINK are not native.  MAP JOIN and REDUCE SINK are 
> conditionally native.  FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
> Tez
> ...
>   Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: alltypesorc
>   Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
>   Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
> Group By Operator
>   keys: cint (type: int)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
>   Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false
>  

[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-11394:

Status: In Progress  (was: Patch Available)

> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch, 
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch, 
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch, 
> HIVE-11394.091.patch, HIVE-11394.092.patch, HIVE-11394.093.patch, 
> HIVE-11394.094.patch, HIVE-11394.095.patch, HIVE-11394.096.patch, 
> HIVE-11394.097.patch, HIVE-11394.098.patch, HIVE-11394.09.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization 
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators.  E.g. Filter 
> Vectorization.  It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions.  E.g. 
> predicateExpression.  It includes all information of SUMMARY and OPERATOR, 
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization 
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION 
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for 
> GROUPBY operator: Data type struct of 
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:": 
> "false" which says a node has a GROUP BY with an AVG or some other aggregator 
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators 
> are row-mode.  I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at 
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.  
> Today, GROUP BY and FILE SINK are not native.  MAP JOIN and REDUCE SINK are 
> conditionally native.  FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
> Tez
> ...
>   Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: alltypesorc
>   Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
>   Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
> Group By Operator
>   keys: cint (type: int)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
>   Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false

[jira] [Updated] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15784:

Status: Patch Available  (was: Open)

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15509) Add back the script + transform tests to minitez

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-15509:
-
Attachment: HIVE-15509.2.patch

> Add back the script + transform tests to minitez
> 
>
> Key: HIVE-15509
> URL: https://issues.apache.org/jira/browse/HIVE-15509
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15509.1.patch, HIVE-15509.1.patch, 
> HIVE-15509.2.patch
>
>
> Script operator cannot run in minillap and so was removed from the minillap 
> test suite. But tez supports script + transform. Add the removed tests back 
> to minitez test suite. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15688) LlapServiceDriver - an option to get rid of run.sh

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849355#comment-15849355
 ] 

Siddharth Seth commented on HIVE-15688:
---

I don't think start should be the default. That breaks existing behaviour which 
a lot of people are likely used to.
Thought we'd be doing this by continuing to generate a run.sh - which invokes a 
single java binary (instead of the 4 slider commands), accepting parameters for 
whether a destroy is required etc. That still saves most of the time, and 
retains compatibility. Having a package sit around which can be invoked 
multiple times is really useful while debugging. Also serves as a history for 
what was started (think this may still be the case - except without the 
run.sh). Most of the code would remain unchanged - except for the run.sh 
generation which points to the slider command or the new Java launch.

> LlapServiceDriver - an option to get rid of run.sh
> --
>
> Key: HIVE-15688
> URL: https://issues.apache.org/jira/browse/HIVE-15688
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15688.patch
>
>
> run.sh is very slow because it's 4 calls to slider, which means 4 JVMs, 4 
> connections to RM and other crap, for   2-5sec. of overhead per call, 
> depending on the machine/cluster.
> What we need is a mode for llapservicedriver that would not generate run.sh, 
> but would rather run the cluster immediately by calling the corresponding 4 
> slider APIs. Should probably be the default, too. For compat with scripts we 
> might generate blank run.sh for now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14420) Fix orc_llap_counters.q test failure in master

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-14420:
-
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review!

> Fix orc_llap_counters.q test failure in master
> --
>
> Key: HIVE-14420
> URL: https://issues.apache.org/jira/browse/HIVE-14420
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Affects Versions: 2.2.0, 2.1.1
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0
>
> Attachments: HIVE-14420.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-14990:

Attachment: HIVE-14990.12.patch

After a huge merge. Will probably have a lot of failures, I only verified the 
build.

> run all tests for MM tables and fix the issues that are found
> -
>
> Key: HIVE-14990
> URL: https://issues.apache.org/jira/browse/HIVE-14990
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, 
> HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, 
> HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, 
> HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, 
> HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, 
> HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.patch
>
>
> Expected failures 
> 1) All HCat tests (cannot write MM tables via the HCat writer)
> 2) Almost all merge tests (alter .. concat is not supported).
> 3) Tests that run dfs commands with specific paths (path changes).
> 4) Truncate column (not supported).
> 5) Describe formatted will have the new table fields in the output (before 
> merging MM with ACID).
> 6) Many tests w/explain extended - diff in partition "base file name" (path 
> changes).
> 7) TestTxnCommands - all the conversion tests, as they check for bucket count 
> using file lists (path changes).
> 8) HBase metastore tests cause methods are not implemented.
> 9) Some load and ExIm tests that export a table and then rely on specific 
> path for load (path changes).
> 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due 
> to how it accounts for buckets



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15458) Fix semi-join conversion rule for subquery

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849294#comment-15849294
 ] 

Vineet Garg commented on HIVE-15458:


cc [~ashutoshc]
I tried with cbo off for following query
{code:SQL} select p1.p_name from part p1 left semi join (select p.* from part 
p, part pp where p.p_size = pp.p_size) p2 on p1.p_type = p2.p_type; {code}

to produce similar AST to observe how we deal with this in non-cbo, but HIVE 
doesn't produce another JOIN underneath LEFT SEMI JOIN node. Probably we never 
get into this situation with cbo off.

> Fix semi-join conversion rule for subquery
> --
>
> Key: HIVE-15458
> URL: https://issues.apache.org/jira/browse/HIVE-15458
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>
> Subquery code in *CalcitePlanner* turns off *hive.enable.semijoin.conversion* 
> since it doesn't work for subqueries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15458) Fix semi-join conversion rule for subquery

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849292#comment-15849292
 ] 

Vineet Garg commented on HIVE-15458:


This is not really subquery issue. This is also reproducible with following 
query:
{code:SQL} select part.p_type from part join (select p1.p_name from part p1, 
part p2 group by p1.p_name) pp where pp.p_name = part.p_name; {code}

This will throw following exception in hive log
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid table 
alias or column reference '$hdt$_0': (possible column names are: 
$hdt$_1.p_name, $hdt$_2.dummy)
{noformat}

Note that after throwing this exception HIVE falls back to non-cbo path to 
execute this query successfully, so beeline/hivecli won't see this error.

Issue is during conversion of calcite plan to AST, specifically following code 
in {{ASTConverter.java}}

{code}
else if (r instanceof Join) {
  Join join = (Join) r;
  QueryBlockInfo left = convertSource(join.getLeft());
  QueryBlockInfo right = convertSource(join.getRight());
  s = new Schema(left.schema, right.schema);
  ASTNode cond = join.getCondition().accept(new RexVisitor(s));
  boolean semiJoin = join instanceof SemiJoin;
  if (join.getRight() instanceof Join) {
// Invert join inputs; this is done because otherwise the 
SemanticAnalyzer
// methods to merge joins will not kick in
JoinRelType type;
if (join.getJoinType() == JoinRelType.LEFT) {
  type = JoinRelType.RIGHT;
} else if (join.getJoinType() == JoinRelType.RIGHT) {
  type = JoinRelType.LEFT;
} else {
  type = join.getJoinType();
}
ast = ASTBuilder.join(right.ast, left.ast, type, cond, semiJoin);
  } else {
ast = ASTBuilder.join(left.ast, right.ast, join.getJoinType(), cond, 
semiJoin);
  }
  if (semiJoin) {
s = left.schema;
  }
{code}

We should not be inverting join inputs for SEMI join since it change the 
semantics.

Bypassing this for semi-join produces correct AST but further throws an 
exception while generating joinTree  from AST in 
{{SemanticAnalyzer::genJoinTree()}}

Plan after semi-join optimization looks like as follow:

{code}
HiveProject(p_type=[$1])
  HiveSemiJoin(condition=[=($0, $2)], joinType=[inner])
HiveProject(p_name=[$1], p_type=[$4])
  HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[default.part]], table:alias=[part])
HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[not 
available])
  HiveProject(p_name=[$1])
HiveFilter(condition=[IS NOT NULL($1)])
  HiveTableScan(table=[[default.part]], table:alias=[p1])
  HiveProject(DUMMY=[0])
HiveTableScan(table=[[default.part]], table:alias=[p2])
{code}


Since {{HiveSemiJoin}} has {{HiveJoin}} as it's right input following code in 
{{SemanticAnalyzer::genJoinTree()}} throws an error

{code}
ASTNode left = (ASTNode) joinParseTree.getChild(0);
ASTNode right = (ASTNode) joinParseTree.getChild(1);
boolean isValidLeftToken = isValidJoinSide(left);
boolean isJoinLeftToken = !isValidLeftToken && isJoinToken(left);
boolean isValidRightToken = isValidJoinSide(right);
boolean isJoinRightToken = !isValidRightToken && isJoinToken(right);
// TODO: if we didn't care about the column order, we could switch join 
sides here
//   for TOK_JOIN and TOK_FULLOUTERJOIN.
if (!isValidLeftToken && !isJoinLeftToken) {
  throw new SemanticException("Invalid token on the left side of the join: "
  + left.getToken().getText() + "; please rewrite your query");
} else if (!isValidRightToken) {
  String advice= "";
  if (isJoinRightToken && !isJoinLeftToken) {
advice = "; for example, put the nested join on the left side, or nest 
joins differently";
  } else if (isJoinRightToken) {
advice = "; for example, nest joins differently";
  }
  throw new SemanticException("Invalid token on the right side of the join: 
"
  + right.getToken().getText() + "; please rewrite your query" + advice);
}
{code}

{{genJoinTree}} does not expect it's right input to be another join

{code}
private static boolean isValidJoinSide(ASTNode right) {
return (right.getToken().getType() == HiveParser.TOK_TABREF)
|| (right.getToken().getType() == HiveParser.TOK_SUBQUERY)
|| (right.getToken().getType() == HiveParser.TOK_PTBLFUNCTION);
  }
{code}

> Fix semi-join conversion rule for subquery
> --
>
> Key: HIVE-15458
> URL: https://issues.apache.org/jira/browse/HIVE-15458
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>
> Subquery code in *CalcitePlanner* turns 

[jira] [Commented] (HIVE-15786) Provide additional information from the llapstatus command

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849287#comment-15849287
 ] 

Siddharth Seth commented on HIVE-15786:
---

PreCommit won't work because slider doesn't seem to publish artifacts... also 
dependence on a snapshot will cause failure.

> Provide additional information from the llapstatus command
> --
>
> Key: HIVE-15786
> URL: https://issues.apache.org/jira/browse/HIVE-15786
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15786.01.patch
>
>
> Slider is making enhancements to provide additional information like 
> completed containers, pending containers etc.
> Integrate with this to provide additional details in llapstatus.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849281#comment-15849281
 ] 

Siddharth Seth commented on HIVE-15779:
---

+1

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15787) Make druid jars available for llap

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15787:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master.

> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-15787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15787) Make druid jars available for llap

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849275#comment-15849275
 ] 

Hive QA commented on HIVE-15787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850514/HIVE-15787.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11022 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
 (batchId=153)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3315/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3315/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3315/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850514 - PreCommit-HIVE-Build

> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-15787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15763) Subquery in both LHS and RHS of IN/NOT IN throws misleading error

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15763:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Vineet!

> Subquery in both LHS and RHS of IN/NOT IN throws misleading error
> -
>
> Key: HIVE-15763
> URL: https://issues.apache.org/jira/browse/HIVE-15763
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>  Labels: sub-query
> Fix For: 2.2.0
>
> Attachments: HIVE-15763.1.patch, HIVE-15763.2.patch
>
>
> Following query throws an error
> {code}select * from part where (select max(p_size) from part) IN (select 
> p_size from part);{code}
> Error
> {noformat}
> SemanticException [Error 10249]: Line 1:79 Unsupported SubQuery Expression 
> 'p_size': Only 1 SubQuery expression is supported.
> {noformat}
> Such queries should either be supported or should be detected and an 
> appropriate error message should be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14420) Fix orc_llap_counters.q test failure in master

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849274#comment-15849274
 ] 

Siddharth Seth commented on HIVE-14420:
---

+1

> Fix orc_llap_counters.q test failure in master
> --
>
> Key: HIVE-14420
> URL: https://issues.apache.org/jira/browse/HIVE-14420
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Affects Versions: 2.2.0, 2.1.1
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14420.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15632) Hive/Druid integration: Incorrect result - Limit on timestamp disappears

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849265#comment-15849265
 ] 

Ashutosh Chauhan commented on HIVE-15632:
-

[~jcamachorodriguez] Can this be closed , since CALCITE-1577 is resolved ?

> Hive/Druid integration: Incorrect result - Limit on timestamp disappears
> 
>
> Key: HIVE-15632
> URL: https://issues.apache.org/jira/browse/HIVE-15632
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> This can be observed with the following query:
> {code:sql}
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset_hive
> ORDER BY `__time` ASC
> LIMIT 10;
> {code}
> Query is translated correctly to Druid _timeseries_, but _limit_ operator 
> disappears.
> {code}
> OK
> Plan optimized by CBO.
> Stage-0
>   Fetch Operator
> limit:-1
> Select Operator [SEL_1]
>   Output:["_col0"]
>   TableScan [TS_0]
> 
> Output:["__time"],properties:{"druid.query.json":"{\"queryType\":\"timeseries\",\"dataSource\":\"druid_tpcds_ss_sold_time_subset\",\"descending\":false,\"granularity\":\"NONE\",\"aggregations\":[],\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"]}","druid.query.type":"timeseries"}
> {code}
> Thus, result has more than 10 rows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15765) Support bracketed comments

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849234#comment-15849234
 ] 

Pengcheng Xiong commented on HIVE-15765:


The patch LGTM. Please take a look at the failing tests. 

> Support bracketed comments
> --
>
> Key: HIVE-15765
> URL: https://issues.apache.org/jira/browse/HIVE-15765
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-15765.1.patch, HIVE-15765.1.patch
>
>
> C-style comments are in the SQL spec as well as supported by all major DBs. 
> The are useful for inline annotation of the SQL. We should have them too.
> Example:
> {noformat}
> select
> /*+ MAPJOIN(a) */ /* mapjoin hint */
> a /* column */
> from foo join bar;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15786) Provide additional information from the llapstatus command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849205#comment-15849205
 ] 

Hive QA commented on HIVE-15786:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850512/HIVE-15786.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3314/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3314/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3314/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-02 01:01:25.472
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3314/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-02 01:01:25.475
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a67eb03 HIVE-15783 : small glitch in LlapServiceDriver on small 
VMs (Sergey Shelukhin, reviewed by Gopal V)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a67eb03 HIVE-15783 : small glitch in LlapServiceDriver on small 
VMs (Sergey Shelukhin, reviewed by Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-02 01:01:26.502
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file 
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java
patching file 
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
patching file 
llap-server/src/java/org/apache/hadoop/hive/llap/cli/SliderUtils.java
patching file 
llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusHelpers.java
patching file llap-server/src/main/resources/llap-cli-log4j2.properties
patching file pom.xml
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.6) for API "JDO"
DataNucleus Enhancer : Classpath
>>  /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) : 
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList
ENHANCED (Persistable) : 
org.apache.hadoop.hive.metastore.model.MStorageDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartition
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MIndex
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRole
ENHANCED 

[jira] [Commented] (HIVE-15777) propagate LLAP app ID to ATS and log it

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849198#comment-15849198
 ] 

Hive QA commented on HIVE-15777:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850486/HIVE-15777.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11007 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=103)

[vector_decimal_aggregate.q,ppd_join3.q,auto_join23.q,join10.q,union_remove_11.q,union_ppr.q,join32.q,groupby_multi_single_reducer2.q,input18.q,stats3.q,cbo_simple_select.q,parquet_join.q,join26.q,groupby1.q,join_reorder2.q]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
 (batchId=153)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3313/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3313/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3313/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850486 - PreCommit-HIVE-Build

> propagate LLAP app ID to ATS and log it 
> 
>
> Key: HIVE-15777
> URL: https://issues.apache.org/jira/browse/HIVE-15777
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15777.01.patch, HIVE-15777.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15730) JDBC should use SQLFeatureNotSupportedException where appropriate instead of SQLException

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849182#comment-15849182
 ] 

Thejas M Nair commented on HIVE-15730:
--

[~vgumashta] Can you please review this patch ?


> JDBC should use SQLFeatureNotSupportedException where appropriate instead of  
> SQLException
> --
>
> Key: HIVE-15730
> URL: https://issues.apache.org/jira/browse/HIVE-15730
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Thejas M Nair
>Assignee: Sankar Hariappan
> Attachments: HIVE_15730_1.patch
>
>
> An example is HiveBaseResultSet.rowDeleted. It throws SQLException("Method 
> not supported") instead of SQLFeatureNotSupportedException.
> For that optional method, the use of SQLFeatureNotSupportedException is more 
> appropriate.
> See 
> http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html#rowDeleted()
> http://docs.oracle.com/javase/7/docs/api/java/sql/SQLFeatureNotSupportedException.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849163#comment-15849163
 ] 

Eugene Koifman commented on HIVE-15774:
---

compBuilder.setIsAcid() should not depend on strict/non strict mode- this is 
just a property of the table itself

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15787) Make druid jars available for llap

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849149#comment-15849149
 ] 

Prasanth Jayachandran commented on HIVE-15787:
--

+1

> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-15787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15787) Make druid jars available for llap

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15787:

Status: Patch Available  (was: Open)

> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-15787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15787) Make druid jars available for llap

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15787:

Attachment: HIVE-15787.patch

Verified druid-storage handler  jar is present in 
llap-slider/package/files/lib/ after executing bin/hive --service llap 
--instances 1

[~prasanth_j] Can you take a look?

> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-15787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15787) Make druid jars available for llap

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reassigned HIVE-15787:
---


> Make druid jars available for llap
> --
>
> Key: HIVE-15787
> URL: https://issues.apache.org/jira/browse/HIVE-15787
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15786) Provide additional information from the llapstatus command

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-15786:
--
Attachment: HIVE-15786.01.patch

Patch generates a human readable summary, while looping. The final JSON output 
includes completed containers.

Needs a new version of slider, can only commit once that is available.

Adds a check for an annoying ZK ACL issue which causes confusion when running 
this script.

The POJOs have been re-factored into a separate class. (I'll rename the class 
in the next patch).
SliderUtils will conflict with a change that [~sershe] is making. Will re-base 
once that goes in.


Sample output (human readable)
{code}
INFO LlapStatusServiceDriverConsole: 

INFO LlapStatusServiceDriverConsole: LLAP Starting up with 
ApplicationId=application_1484282558103_4898 Started 0/2 instances


INFO LlapStatusServiceDriverConsole: 

INFO LlapStatusServiceDriverConsole: 

INFO LlapStatusServiceDriverConsole: LLAP Starting up with 
ApplicationId=application_1484282558103_4898 Started 0/2 instances
container_e08_1484282558103_4898_01_02 failed. Logs at: 
http://cn105-10.l42scl.hortonworks.com:19888/jobhistory/logs/cn108-10.l42scl.hortonworks.com:45454/container_e08_1484282558103_4898_01_02/ctx/sseth


INFO LlapStatusServiceDriverConsole: 

{code}



[~prasanth_j] - could you please review.

> Provide additional information from the llapstatus command
> --
>
> Key: HIVE-15786
> URL: https://issues.apache.org/jira/browse/HIVE-15786
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15786.01.patch
>
>
> Slider is making enhancements to provide additional information like 
> completed containers, pending containers etc.
> Integrate with this to provide additional details in llapstatus.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15786) Provide additional information from the llapstatus command

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-15786:
--
Status: Patch Available  (was: Open)

> Provide additional information from the llapstatus command
> --
>
> Key: HIVE-15786
> URL: https://issues.apache.org/jira/browse/HIVE-15786
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15786.01.patch
>
>
> Slider is making enhancements to provide additional information like 
> completed containers, pending containers etc.
> Integrate with this to provide additional details in llapstatus.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14901) HiveServer2: Use user supplied fetch size to determine #rows serialized in tasks

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849140#comment-15849140
 ] 

Hive QA commented on HIVE-14901:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850490/HIVE-14901.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11021 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
 (batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth
 (batchId=218)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthUDFBlacklist.testBlackListedUdfUsage
 (batchId=217)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testAllowedCommands
 (batchId=218)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testAuthorization1
 (batchId=218)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testBlackListedUdfUsage
 (batchId=218)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testConfigWhiteList
 (batchId=218)
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1 
(batchId=229)
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.testAuthorization1 
(batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3311/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3311/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3311/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850490 - PreCommit-HIVE-Build

> HiveServer2: Use user supplied fetch size to determine #rows serialized in 
> tasks
> 
>
> Key: HIVE-14901
> URL: https://issues.apache.org/jira/browse/HIVE-14901
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, ODBC
>Affects Versions: 2.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Norris Lee
> Attachments: HIVE-14901.1.patch, HIVE-14901.patch
>
>
> Currently, we use {{hive.server2.thrift.resultset.max.fetch.size}} to decide 
> the max number of rows that we write in tasks. However, we should ideally use 
> the user supplied value (which can be extracted from the 
> ThriftCLIService.FetchResults' request parameter) to decide how many rows to 
> serialize in a blob in the tasks. We should however use 
> {{hive.server2.thrift.resultset.max.fetch.size}} to have an upper bound on 
> it, so that we don't go OOM in tasks and HS2. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15786) Provide additional information from the llapstatus command

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth reassigned HIVE-15786:
-


> Provide additional information from the llapstatus command
> --
>
> Key: HIVE-15786
> URL: https://issues.apache.org/jira/browse/HIVE-15786
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>
> Slider is making enhancements to provide additional information like 
> completed containers, pending containers etc.
> Integrate with this to provide additional details in llapstatus.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (HIVE-15634) Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-15634 started by slim bouguerra.
-
> Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization
> 
>
> Key: HIVE-15634
> URL: https://issues.apache.org/jira/browse/HIVE-15634
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: slim bouguerra
>Priority: Critical
>
> {{SET hive.tez.java.opts=-Duser.timezone="UTC";}} can be used to change 
> timezone for Tez tasks. However, when Fetch optimizer kicks in because we can 
> push the full query to Druid, we obtain different values for the timestamp 
> than when jobs are executed. This probably has to do with the timezone on the 
> client side. How should we handle this issue?
> For instance, this can be observed with the following query:
> {code:sql}
> set hive.fetch.task.conversion=more;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-10-31 19:00:00
> 1999-11-01 19:00:00
> 1999-11-02 19:00:00
> 1999-11-03 19:00:00
> 1999-11-04 19:00:00
> 1999-11-05 19:00:00
> 1999-11-06 19:00:00
> 1999-11-07 19:00:00
> 1999-11-08 19:00:00
> set hive.fetch.task.conversion=none;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-11-01 00:00:00
> 1999-11-02 00:00:00
> 1999-11-03 00:00:00
> 1999-11-04 00:00:00
> 1999-11-05 00:00:00
> 1999-11-06 00:00:00
> 1999-11-07 00:00:00
> 1999-11-08 00:00:00
> 1999-11-09 00:00:00
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15634) Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra reassigned HIVE-15634:
-

Assignee: slim bouguerra  (was: Jesus Camacho Rodriguez)

> Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization
> 
>
> Key: HIVE-15634
> URL: https://issues.apache.org/jira/browse/HIVE-15634
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: slim bouguerra
>Priority: Critical
>
> {{SET hive.tez.java.opts=-Duser.timezone="UTC";}} can be used to change 
> timezone for Tez tasks. However, when Fetch optimizer kicks in because we can 
> push the full query to Druid, we obtain different values for the timestamp 
> than when jobs are executed. This probably has to do with the timezone on the 
> client side. How should we handle this issue?
> For instance, this can be observed with the following query:
> {code:sql}
> set hive.fetch.task.conversion=more;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-10-31 19:00:00
> 1999-11-01 19:00:00
> 1999-11-02 19:00:00
> 1999-11-03 19:00:00
> 1999-11-04 19:00:00
> 1999-11-05 19:00:00
> 1999-11-06 19:00:00
> 1999-11-07 19:00:00
> 1999-11-08 19:00:00
> set hive.fetch.task.conversion=none;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-11-01 00:00:00
> 1999-11-02 00:00:00
> 1999-11-03 00:00:00
> 1999-11-04 00:00:00
> 1999-11-05 00:00:00
> 1999-11-06 00:00:00
> 1999-11-07 00:00:00
> 1999-11-08 00:00:00
> 1999-11-09 00:00:00
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15634) Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization

[slim bouguerra (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849116#comment-15849116
 ] 

slim bouguerra commented on HIVE-15634:
---

making sure that the timezone is the same as the hive server will solve the 
issue.
Hence avoid doing {code} SET hive.tez.java.opts=-Duser.timezone="UTC" {code}

> Hive/Druid integration: Timestamp column inconsistent w/o Fetch optimization
> 
>
> Key: HIVE-15634
> URL: https://issues.apache.org/jira/browse/HIVE-15634
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> {{SET hive.tez.java.opts=-Duser.timezone="UTC";}} can be used to change 
> timezone for Tez tasks. However, when Fetch optimizer kicks in because we can 
> push the full query to Druid, we obtain different values for the timestamp 
> than when jobs are executed. This probably has to do with the timezone on the 
> client side. How should we handle this issue?
> For instance, this can be observed with the following query:
> {code:sql}
> set hive.fetch.task.conversion=more;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-10-31 19:00:00
> 1999-11-01 19:00:00
> 1999-11-02 19:00:00
> 1999-11-03 19:00:00
> 1999-11-04 19:00:00
> 1999-11-05 19:00:00
> 1999-11-06 19:00:00
> 1999-11-07 19:00:00
> 1999-11-08 19:00:00
> set hive.fetch.task.conversion=none;
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset
> WHERE `__time` < '1999-11-10 00:00:00';
> OK
> 1999-11-01 00:00:00
> 1999-11-02 00:00:00
> 1999-11-03 00:00:00
> 1999-11-04 00:00:00
> 1999-11-05 00:00:00
> 1999-11-06 00:00:00
> 1999-11-07 00:00:00
> 1999-11-08 00:00:00
> 1999-11-09 00:00:00
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15632) Hive/Druid integration: Incorrect result - Limit on timestamp disappears

[slim bouguerra (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849115#comment-15849115
 ] 

slim bouguerra commented on HIVE-15632:
---

there is no limit on timeseries query. 

> Hive/Druid integration: Incorrect result - Limit on timestamp disappears
> 
>
> Key: HIVE-15632
> URL: https://issues.apache.org/jira/browse/HIVE-15632
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> This can be observed with the following query:
> {code:sql}
> SELECT DISTINCT `__time`
> FROM store_sales_sold_time_subset_hive
> ORDER BY `__time` ASC
> LIMIT 10;
> {code}
> Query is translated correctly to Druid _timeseries_, but _limit_ operator 
> disappears.
> {code}
> OK
> Plan optimized by CBO.
> Stage-0
>   Fetch Operator
> limit:-1
> Select Operator [SEL_1]
>   Output:["_col0"]
>   TableScan [TS_0]
> 
> Output:["__time"],properties:{"druid.query.json":"{\"queryType\":\"timeseries\",\"dataSource\":\"druid_tpcds_ss_sold_time_subset\",\"descending\":false,\"granularity\":\"NONE\",\"aggregations\":[],\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"]}","druid.query.type":"timeseries"}
> {code}
> Thus, result has more than 10 rows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849085#comment-15849085
 ] 

Wei Zheng commented on HIVE-15774:
--

[~ekoifman] Can you take a look?

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849072#comment-15849072
 ] 

Hive QA commented on HIVE-15774:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850496/HIVE-15774.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 11022 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3310/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3310/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3310/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850496 - PreCommit-HIVE-Build

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14870) OracleStore: RawStore implementation optimized for Oracle

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849065#comment-15849065
 ] 

Sergey Shelukhin commented on HIVE-14870:
-

Skimmed thru the patch; sorry for the delay.
It looks like most of the SQL in the patch is very generic, with a few 
easy-to-isolate features like ROWNUM (and quotes to be added for 
postgres/etc.). The things like PartitionFilterGenerator, that use DB-specific 
approach, already fairly well isolated.
Some TODOs may need follow-up (e.g. implement filters where it looks like 
something is already implemented).

The SDS table changes should be doable separately from this change, by changing 
datanucleus schema and making a corresponding upgrade script, it sounds like a 
good idea.

It doesn't address the questions above wrt upgrade scripts, conversion scripts, 
etc. 
Scripts aside, Hybrid...Proxy doesn't look like it's a viable 
upgrade/conversion path for an average user.

I think it should be possible to do it in 3.5 steps.
1) Make low-hanging-fruit schema changes in a regular manner, if desired.
2) Introduce SqlObjectStore, which is basically already there in this patch. 
Code similar to one in ACID sql query helper and directSQL code might help make 
the less-trivial of the queries compatible across DBs.
3) Deprecate/drop ObjectStore.
3.5) Make schema changes that datanucleus makes impossible, if any (or all of 
them if (1) is not done).

> OracleStore: RawStore implementation optimized for Oracle
> -
>
> Key: HIVE-14870
> URL: https://issues.apache.org/jira/browse/HIVE-14870
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-14870.patch, OracleStoreDesignProposal.pdf, 
> schema-oraclestore.sql
>
>
> The attached document is a proposal for a RawStore implementation which is 
> optimized for Oracle and replaces DataNucleus. The document outlines schema 
> changes, OracleStore implementation details, and performance tests against 
> ObjectStore, ObjectStore+DirectSQL, and OracleStore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15783) small glitch in LlapServiceDriver on small VMs

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15783:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master

> small glitch in LlapServiceDriver on small VMs
> --
>
> Key: HIVE-15783
> URL: https://issues.apache.org/jira/browse/HIVE-15783
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-15783.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849027#comment-15849027
 ] 

Sergey Shelukhin commented on HIVE-15784:
-

+1 pending tests

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15509) Add back the script + transform tests to minitez

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849024#comment-15849024
 ] 

Prasanth Jayachandran commented on HIVE-15509:
--

Rebase isn't sufficient. Need some golden file updates. Will update shortly


> Add back the script + transform tests to minitez
> 
>
> Key: HIVE-15509
> URL: https://issues.apache.org/jira/browse/HIVE-15509
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15509.1.patch, HIVE-15509.1.patch
>
>
> Script operator cannot run in minillap and so was removed from the minillap 
> test suite. But tez supports script + transform. Add the removed tests back 
> to minitez test suite. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15784:

Attachment: HIVE-15784.01.patch

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15760) TezCompiler throws ConcurrentModificationException during cycle detection

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-15760:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to master

> TezCompiler throws ConcurrentModificationException during cycle detection
> -
>
> Key: HIVE-15760
> URL: https://issues.apache.org/jira/browse/HIVE-15760
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Deepak Jaiswal
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE.15760.1.patch
>
>
> HIVE-15269 is causing the following exception when we run explain on query72 
> (TPC-DS).
> {code}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler$SemiJoinCycleRemovalDueToMapsideJoins.process(TezCompiler.java:689)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemiJoinCyclesDueToMapsideJoins(TezCompiler.java:759)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:112)
> at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11122)
> at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:275)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:129)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:513)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1305)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1445)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline reassigned HIVE-15784:
---


> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15774:
-
Status: Patch Available  (was: Open)

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15774:
-
Status: Open  (was: Patch Available)

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15774:
-
Attachment: HIVE-15774.2.patch

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch, HIVE-15774.2.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15783) small glitch in LlapServiceDriver on small VMs

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849006#comment-15849006
 ] 

Gopal V commented on HIVE-15783:


Heh, I want half a thread please.

LGTM - +1.

> small glitch in LlapServiceDriver on small VMs
> --
>
> Key: HIVE-15783
> URL: https://issues.apache.org/jira/browse/HIVE-15783
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15783.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15748) Remove cycles created due to semi join branch and map join Op on same operator pipeline

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-15748:
--
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master

> Remove cycles created due to semi join branch and map join Op on same 
> operator pipeline
> ---
>
> Key: HIVE-15748
> URL: https://issues.apache.org/jira/browse/HIVE-15748
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Fix For: 2.2.0
>
> Attachments: HIVE-15748.1.patch, HIVE-15748.2.patch, 
> HIVE-15748.3.patch
>
>
> If a semi join branch and map join operator are on same operator pipeline, 
> then there could be a cycle created. Where the other map feeding into the 
> mapjoin operator is waiting for the semi join branch to finish causing a 
> cycle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15782) query on parquet table returns incorrect result when hive.optimize.index.filter is set to true

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849002#comment-15849002
 ] 

Hive QA commented on HIVE-15782:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850463/HIVE-15782.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11023 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
 (batchId=153)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.ql.io.parquet.TestParquetRecordReaderWrapper.testBuilderComplexTypes
 (batchId=253)
org.apache.hadoop.hive.ql.io.parquet.TestParquetRecordReaderWrapper.testBuilderComplexTypes2
 (batchId=253)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3309/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3309/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3309/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850463 - PreCommit-HIVE-Build

> query on parquet table returns incorrect result when 
> hive.optimize.index.filter is set to true 
> ---
>
> Key: HIVE-15782
> URL: https://issues.apache.org/jira/browse/HIVE-15782
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15782.1.patch
>
>
> When hive.optimize.index.filter is set to true, the parquet table is filtered 
> using the parquet column index. 
> {noformat}
> set hive.optimize.index.filter=true;
> CREATE TABLE t1 (
>   name string,
>   dec decimal(5,0)
> ) stored as parquet;
> insert into table t1 values('Jim', 3);
> insert into table t1 values('Tom', 5);
> select * from t1 where (name = 'Jim' or dec = 5);
> {noformat}
> Only one row {{Jim, 3}} is returned, but both should be returned. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15783) small glitch in LlapServiceDriver on small VMs

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15783:

Attachment: HIVE-15783.patch

[~sseth] can you take a look? tiny patch

> small glitch in LlapServiceDriver on small VMs
> --
>
> Key: HIVE-15783
> URL: https://issues.apache.org/jira/browse/HIVE-15783
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15783.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15783) small glitch in LlapServiceDriver on small VMs

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15783:

Status: Patch Available  (was: Open)

> small glitch in LlapServiceDriver on small VMs
> --
>
> Key: HIVE-15783
> URL: https://issues.apache.org/jira/browse/HIVE-15783
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15783.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15783) small glitch in LlapServiceDriver on small VMs

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-15783:
---


> small glitch in LlapServiceDriver on small VMs
> --
>
> Key: HIVE-15783
> URL: https://issues.apache.org/jira/browse/HIVE-15783
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15749) Add missing ASF headers

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-15749:
---
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks, Peter.

> Add missing ASF headers
> ---
>
> Key: HIVE-15749
> URL: https://issues.apache.org/jira/browse/HIVE-15749
> Project: Hive
>  Issue Type: Task
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15749.patch
>
>
> There are missing ASF headers in the following files:
> - 
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
> - 
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidRecordWriter.java
> - jdbc/src/test/org/apache/hive/jdbc/TestHivePreparedStatement.java
> - 
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/LineRrOffsetReader.java
> - 
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/PassThruOffsetReader.java
> - ql/src/test/org/apache/hadoop/hive/ql/parse/TestMergeStatement.java
> - ql/src/test/org/apache/hadoop/hive/ql/plan/TestMapWork.java
> As discussed on the dev list, creating a jira, and adding a patch with the 
> appropriate headers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14901) HiveServer2: Use user supplied fetch size to determine #rows serialized in tasks

[Norris Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Norris Lee updated HIVE-14901:
--
Status: Patch Available  (was: In Progress)

> HiveServer2: Use user supplied fetch size to determine #rows serialized in 
> tasks
> 
>
> Key: HIVE-14901
> URL: https://issues.apache.org/jira/browse/HIVE-14901
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, ODBC
>Affects Versions: 2.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Norris Lee
> Attachments: HIVE-14901.1.patch, HIVE-14901.patch
>
>
> Currently, we use {{hive.server2.thrift.resultset.max.fetch.size}} to decide 
> the max number of rows that we write in tasks. However, we should ideally use 
> the user supplied value (which can be extracted from the 
> ThriftCLIService.FetchResults' request parameter) to decide how many rows to 
> serialize in a blob in the tasks. We should however use 
> {{hive.server2.thrift.resultset.max.fetch.size}} to have an upper bound on 
> it, so that we don't go OOM in tasks and HS2. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14901) HiveServer2: Use user supplied fetch size to determine #rows serialized in tasks

[Norris Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Norris Lee updated HIVE-14901:
--
Attachment: HIVE-14901.1.patch

> HiveServer2: Use user supplied fetch size to determine #rows serialized in 
> tasks
> 
>
> Key: HIVE-14901
> URL: https://issues.apache.org/jira/browse/HIVE-14901
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, ODBC
>Affects Versions: 2.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Norris Lee
> Attachments: HIVE-14901.1.patch, HIVE-14901.patch
>
>
> Currently, we use {{hive.server2.thrift.resultset.max.fetch.size}} to decide 
> the max number of rows that we write in tasks. However, we should ideally use 
> the user supplied value (which can be extracted from the 
> ThriftCLIService.FetchResults' request parameter) to decide how many rows to 
> serialize in a blob in the tasks. We should however use 
> {{hive.server2.thrift.resultset.max.fetch.size}} to have an upper bound on 
> it, so that we don't go OOM in tasks and HS2. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15517) NOT (x <=> y) returns NULL if x or y is NULL

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15517:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> NOT (x <=> y) returns NULL if x or y is NULL
> 
>
> Key: HIVE-15517
> URL: https://issues.apache.org/jira/browse/HIVE-15517
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Operators, Query Processor, SQL
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Alexey Bedrintsev
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15517.01.patch
>
>
> I created a table as following:
> create table test(x string, y string);
> insert into test values ('q', 'q'), ('q', 'w'), (NULL, 'q'), ('q', NULL), 
> (NULL, NULL);
> Then I try to compare values taking NULLs into account:
> select *, x<=>y, not (x<=> y), (x <=> y) = false from test;
> OK
> q   q   truefalse   false
> q   w   false   truetrue
> q   NULLfalse   NULLtrue
> NULLq   false   NULLtrue
> NULLNULLtrueNULLfalse
> I expected that 4th column will be the same as 5th one but actually got NULL 
> as result of "not false" and "not true" expressions.
> Hive 1.2.1000.2.5.0.0-1245
> Subversion 
> git://c66-slave-20176e25-3/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
>  -r da6c690d384d1666f5a5f450be5cbc54e2fe4bd6
> Compiled by jenkins on Fri Aug 26 01:39:52 UTC 2016
> From source with checksum c30648316a632f7a753f4359e5c8f4d6



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15517) NOT (x <=> y) returns NULL if x or y is NULL

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848945#comment-15848945
 ] 

Sergey Shelukhin commented on HIVE-15517:
-

Should the be resolved?

> NOT (x <=> y) returns NULL if x or y is NULL
> 
>
> Key: HIVE-15517
> URL: https://issues.apache.org/jira/browse/HIVE-15517
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Operators, Query Processor, SQL
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Alexey Bedrintsev
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15517.01.patch
>
>
> I created a table as following:
> create table test(x string, y string);
> insert into test values ('q', 'q'), ('q', 'w'), (NULL, 'q'), ('q', NULL), 
> (NULL, NULL);
> Then I try to compare values taking NULLs into account:
> select *, x<=>y, not (x<=> y), (x <=> y) = false from test;
> OK
> q   q   truefalse   false
> q   w   false   truetrue
> q   NULLfalse   NULLtrue
> NULLq   false   NULLtrue
> NULLNULLtrueNULLfalse
> I expected that 4th column will be the same as 5th one but actually got NULL 
> as result of "not false" and "not true" expressions.
> Hive 1.2.1000.2.5.0.0-1245
> Subversion 
> git://c66-slave-20176e25-3/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
>  -r da6c690d384d1666f5a5f450be5cbc54e2fe4bd6
> Compiled by jenkins on Fri Aug 26 01:39:52 UTC 2016
> From source with checksum c30648316a632f7a753f4359e5c8f4d6



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15700) BytesColumnVector can get stuck trying to resize byte buffer

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848934#comment-15848934
 ] 

Matt McCline commented on HIVE-15700:
-

+1 LGTM

> BytesColumnVector can get stuck trying to resize byte buffer
> 
>
> Key: HIVE-15700
> URL: https://issues.apache.org/jira/browse/HIVE-15700
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15700.1.patch, HIVE-15700.2.patch, 
> HIVE-15700.3.patch, HIVE-15700.4.patch
>
>
> While looking at HIVE-15698, hit an issue where one of the reducers was stuck 
> in the following stack trace:
> {noformat}
> Thread 12735: (state = IN_JAVA)
>  - 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.increaseBufferSpace(int)
>  @bci=22, line=245 (Compiled frame; information may be imprecise)
>  - org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(int, 
> byte[], int, int) @bci=18, line=150 (Interpreted frame)
>  - 
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storeRowColumn(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,
>  int, int, boolean) @bci=536, line=442 (Compiled frame)
>  - 
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,
>  int) @bci=110, line=761 (Interpreted frame)
>  - 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(org.apache.hadoop.io.BytesWritable,
>  java.lang.Iterable, byte) @bci=184, line=444 (Interpreted frame)
>  - org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector() 
> @bci=119, line=388 (Interpreted frame)
>  - org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord() @bci=8, 
> line=239 (Interpreted frame)
>  - org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run() @bci=124, 
> line=319 (Interpreted frame)
>  - 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(java.util.Map,
>  java.util.Map) @bci=30, line=185 (Interpreted frame)
>  - org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(java.util.Map, 
> java.util.Map) @bci=159, line=168 (Interpreted frame)
>  - org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run() @bci=65, 
> line=370 (Interpreted frame)
>  - org.apache.tez.runtime.task.TaskRunner2Callable$1.run() @bci=133, line=73 
> (Interpreted frame)
>  - org.apache.tez.runtime.task.TaskRunner2Callable$1.run() @bci=1, line=61 
> (Interpreted frame)
>  - 
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
>  java.security.AccessControlContext) @bci=0 (Compiled frame)
>  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
> java.security.PrivilegedExceptionAction) @bci=42, line=422 (Interpreted frame)
>  - 
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
>  @bci=14, line=1724 (Interpreted frame)
>  - org.apache.tez.runtime.task.TaskRunner2Callable.callInternal() @bci=38, 
> line=61 (Interpreted frame)
>  - org.apache.tez.runtime.task.TaskRunner2Callable.callInternal() @bci=1, 
> line=37 (Interpreted frame)
>  - org.apache.tez.common.CallableWithNdc.call() @bci=8, line=36 (Interpreted 
> frame)
>  - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=95, line=1142 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
> The reducer's input was 167 9MB binary values coming from the previous map 
> job. Per [~gopalv] the BytesColumnVector is stuck trying to reallocate/copy 
> all of these values into the same memory buffer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15777) propagate LLAP app ID to ATS and log it

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15777:

Attachment: HIVE-15777.01.patch

Updated the patch; moved the code to ATSHook, fixed the NPEs in tests where the 
servicerecord doesn't have the property.

> propagate LLAP app ID to ATS and log it 
> 
>
> Key: HIVE-15777
> URL: https://issues.apache.org/jira/browse/HIVE-15777
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15777.01.patch, HIVE-15777.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15774) Ensure DbLockManager backward compatibility for non-ACID resources

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848931#comment-15848931
 ] 

Hive QA commented on HIVE-15774:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850462/HIVE-15774.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11022 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge 
(batchId=195)
org.apache.hadoop.hive.ql.io.orc.TestNewInputOutputFormat.testNewOutputFormatComplex
 (batchId=255)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks 
(batchId=274)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testShowTablesLock 
(batchId=274)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3308/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3308/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3308/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850462 - PreCommit-HIVE-Build

> Ensure DbLockManager backward compatibility for non-ACID resources
> --
>
> Key: HIVE-15774
> URL: https://issues.apache.org/jira/browse/HIVE-15774
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive, Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15774.1.patch
>
>
> In pre-ACID days, users perform operations such as INSERT with either 
> ZooKeeperHiveLockManager or no lock manager at all. If their workflow is 
> designed to take advantage of no locking and they take care of the control of 
> concurrency, this works well with good performance.
> With ACID, if users enable transactions (i.e. using DbTxnManager & 
> DbLockManager), then for all the operations, different types of locks will be 
> acquired accordingly by DbLockManager, even for non-ACID resources. This may 
> impact the performance of some workflows designed for pre-ACID use cases.
> A viable solution would be to differentiate the locking mode for ACID and 
> non-ACID resources, so that DbLockManager will continue its current behavior 
> for ACID tables, but will be able to acquire a less strict lock type for 
> non-ACID resources, thus avoiding the performance loss for those workflows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15723) Hive should report a warning about missing table/column statistics to user.

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15723:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed addendum patch

> Hive should report a warning about missing table/column statistics to user.
> ---
>
> Key: HIVE-15723
> URL: https://issues.apache.org/jira/browse/HIVE-15723
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15723.01.patch, HIVE-15723.02.patch, 
> HIVE-15723.03.patch, HIVE-15723.04.patch, HIVE-15723.05.patch
>
>
> Many Hive performance issues are due to missing statistics. Either all, table 
> or column statistics are missing. Potentially a new partition has been added 
> and customer forgot to gather stats for that partition.
> A simple warning about a table or column missing statistics can be very 
> helpful and makes hive more user friendly. Hive already has this information, 
> its a matter of printing it out.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15649) LLAP IO may NPE on all-column read

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15649:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed the extra patch. Thanks for the review!

> LLAP IO may NPE on all-column read
> --
>
> Key: HIVE-15649
> URL: https://issues.apache.org/jira/browse/HIVE-15649
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-15649.01.patch, HIVE-15649.02.patch, 
> HIVE-15649.03.patch, HIVE-15649.04.patch, HIVE-15649.patch
>
>
> It seems like very few paths use READ_ALL_COLUMNS config, but some do. LLAP 
> IO doesn't account for that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15777) propagate LLAP app ID to ATS and log it

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848901#comment-15848901
 ] 

Jason Dere commented on HIVE-15777:
---

Maybe move the changes in Driver into ATSHook of possible? Just to avoid 
cluttering Driver.execute() more than it already is, and avoid yet another 
field to HookContext, if we can retrieve the information within the ATSHook 
like we're doing for ExecutionMode.

Is this patch responsible for the test failures?

> propagate LLAP app ID to ATS and log it 
> 
>
> Key: HIVE-15777
> URL: https://issues.apache.org/jira/browse/HIVE-15777
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15777.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15763) Subquery in both LHS and RHS of IN/NOT IN throws misleading error

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848889#comment-15848889
 ] 

Hive QA commented on HIVE-15763:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850461/HIVE-15763.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11023 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3307/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3307/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3307/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850461 - PreCommit-HIVE-Build

> Subquery in both LHS and RHS of IN/NOT IN throws misleading error
> -
>
> Key: HIVE-15763
> URL: https://issues.apache.org/jira/browse/HIVE-15763
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>  Labels: sub-query
> Attachments: HIVE-15763.1.patch, HIVE-15763.2.patch
>
>
> Following query throws an error
> {code}select * from part where (select max(p_size) from part) IN (select 
> p_size from part);{code}
> Error
> {noformat}
> SemanticException [Error 10249]: Line 1:79 Unsupported SubQuery Expression 
> 'p_size': Only 1 SubQuery expression is supported.
> {noformat}
> Such queries should either be supported or should be detected and an 
> appropriate error message should be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of same priority

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-15779:
---
Summary: LLAP: WaitQueue comparators should return 0 when tasks of the same 
DAG are of same priority  (was: LLAP: WaitQueue comparators should return 0 
when tasks are of same priority)

> LLAP: WaitQueue comparators should return 0 when tasks of the same DAG are of 
> same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848849#comment-15848849
 ] 

Thejas M Nair commented on HIVE-15766:
--

Moved this to a top level issue as this is seen independent of the replication 
v2 work.

> DBNotificationlistener leaks JDOPersistenceManager
> --
>
> Key: HIVE-15766
> URL: https://issues.apache.org/jira/browse/HIVE-15766
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15766.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848847#comment-15848847
 ] 

Thejas M Nair commented on HIVE-15766:
--

+1
Thanks for adding the test case as well.


> DBNotificationlistener leaks JDOPersistenceManager
> --
>
> Key: HIVE-15766
> URL: https://issues.apache.org/jira/browse/HIVE-15766
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15766.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14901) HiveServer2: Use user supplied fetch size to determine #rows serialized in tasks

[Norris Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Norris Lee updated HIVE-14901:
--
Status: In Progress  (was: Patch Available)

> HiveServer2: Use user supplied fetch size to determine #rows serialized in 
> tasks
> 
>
> Key: HIVE-14901
> URL: https://issues.apache.org/jira/browse/HIVE-14901
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, ODBC
>Affects Versions: 2.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Norris Lee
> Attachments: HIVE-14901.patch
>
>
> Currently, we use {{hive.server2.thrift.resultset.max.fetch.size}} to decide 
> the max number of rows that we write in tasks. However, we should ideally use 
> the user supplied value (which can be extracted from the 
> ThriftCLIService.FetchResults' request parameter) to decide how many rows to 
> serialize in a blob in the tasks. We should however use 
> {{hive.server2.thrift.resultset.max.fetch.size}} to have an upper bound on 
> it, so that we don't go OOM in tasks and HS2. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-15766:
-
Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-14841)

> DBNotificationlistener leaks JDOPersistenceManager
> --
>
> Key: HIVE-15766
> URL: https://issues.apache.org/jira/browse/HIVE-15766
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15766.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks are of same priority

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848843#comment-15848843
 ] 

Sergey Shelukhin commented on HIVE-15779:
-

+1; integer compare could also be used.

> LLAP: WaitQueue comparators should return 0 when tasks are of same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15779) LLAP: WaitQueue comparators should return 0 when tasks are of same priority

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848843#comment-15848843
 ] 

Sergey Shelukhin edited comment on HIVE-15779 at 2/1/17 7:40 PM:
-

+1; Integer.compare could also be used.
I wonder why java uses ?: and not minus in its implementation. Overflow 
concerns?


was (Author: sershe):
+1; integer compare could also be used.

> LLAP: WaitQueue comparators should return 0 when tasks are of same priority
> ---
>
> Key: HIVE-15779
> URL: https://issues.apache.org/jira/browse/HIVE-15779
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15779.1.patch
>
>
> Observed cases, where in tasks within same vertex were competing with each 
> and getting killed
> {noformat}
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003877_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003179_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003877_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_002959_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003832_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002959_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003723_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003254_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003723_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003560_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003076_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003560_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003775_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004011_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003775_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003842_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_004045_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003842_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003953_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003915_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003953_7 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003819_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003919_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003819_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_002074_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003790_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_002074_8 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_003670_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003736_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003670_7 because 
> of lower priority
> [IPC Server handler 1 on 44598 (1484282558103_4855_1_00_003153_8)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003877_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003153_8 because 
> of lower priority
> [IPC Server handler 0 on 44598 (1484282558103_4855_1_00_003328_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003775_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003328_7 because 
> of lower priority
> [IPC Server handler 4 on 44598 (1484282558103_4855_1_00_003817_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003842_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003817_7 because 
> of lower priority
> [IPC Server handler 2 on 44598 (1484282558103_4855_1_00_004065_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003723_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_004065_7 because 
> of lower priority
> [IPC Server handler 3 on 44598 (1484282558103_4855_1_00_003902_7)] 
> impl.TaskExecutorService: attempt_1484282558103_4855_1_00_003560_7 evicted 
> from wait queue in favor of attempt_1484282558103_4855_1_00_003902_7 because 
> of lower priority
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13667) Improve performance for ServiceInstanceSet.getByHost

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848836#comment-15848836
 ] 

Sergey Shelukhin commented on HIVE-13667:
-

It would be nice to run cluster tests on this, and flex/kill LLAP containers 
while some long query is running.

> Improve performance for ServiceInstanceSet.getByHost
> 
>
> Key: HIVE-13667
> URL: https://issues.apache.org/jira/browse/HIVE-13667
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Attachments: HIVE-13667.1.patch
>
>
> ServiceInstanceSet.getByHost is used for scheduling local tasks as well as 
> constructing the log URL.
> It ends up traversing all hosts on each lookup. This should be avoided.
> cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15782) query on parquet table returns incorrect result when hive.optimize.index.filter is set to true

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848830#comment-15848830
 ] 

Aihua Xu commented on HIVE-15782:
-

patch-1: decimal, date and timestamp currently are supported for filtering for 
parquet. Currently if there is such type in the filter expression, such 
subexpression with that type is incorrectly ignored. 

With the patch, if we can't convert search argument into filter expression, 
then no filtering will be applied on parquet files.

> query on parquet table returns incorrect result when 
> hive.optimize.index.filter is set to true 
> ---
>
> Key: HIVE-15782
> URL: https://issues.apache.org/jira/browse/HIVE-15782
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15782.1.patch
>
>
> When hive.optimize.index.filter is set to true, the parquet table is filtered 
> using the parquet column index. 
> {noformat}
> set hive.optimize.index.filter=true;
> CREATE TABLE t1 (
>   name string,
>   dec decimal(5,0)
> ) stored as parquet;
> insert into table t1 values('Jim', 3);
> insert into table t1 values('Tom', 5);
> select * from t1 where (name = 'Jim' or dec = 5);
> {noformat}
> Only one row {{Jim, 3}} is returned, but both should be returned. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)