[jira] [Updated] (HIVE-17036) Lineage: Minor CPU/Mem optimization for lineage transform
[ https://issues.apache.org/jira/browse/HIVE-17036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17036: Attachment: prof_1.png prof_2.png > Lineage: Minor CPU/Mem optimization for lineage transform > - > > Key: HIVE-17036 > URL: https://issues.apache.org/jira/browse/HIVE-17036 > Project: Hive > Issue Type: Improvement > Components: lineage >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: prof_1.png, prof_2.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17036) Lineage: Minor CPU/Mem optimization for lineage transform
[ https://issues.apache.org/jira/browse/HIVE-17036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17036: Issue Type: Improvement (was: Bug) > Lineage: Minor CPU/Mem optimization for lineage transform > - > > Key: HIVE-17036 > URL: https://issues.apache.org/jira/browse/HIVE-17036 > Project: Hive > Issue Type: Improvement > Components: lineage >Reporter: Rajesh Balamohan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17012) ACID Table: Number of reduce tasks should be computed correctly when sort.dynamic.partition is enabled
[
https://issues.apache.org/jira/browse/HIVE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071990#comment-16071990
]
Rajesh Balamohan edited comment on HIVE-17012 at 7/5/17 8:55 AM:
-
{{ReducerTraits}} would be {{FIXED}} for ACID tables with buckets.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L102
prevents from computing reducer tasks for Reducer 3.
was (Author: rajesh.balamohan):
ReducerTraits would be {{FIXED}} for ACID tables with buckets.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L102
prevents from computing reducer tasks for Reducer 3.
> ACID Table: Number of reduce tasks should be computed correctly when
> sort.dynamic.partition is enabled
> --
>
> Key: HIVE-17012
> URL: https://issues.apache.org/jira/browse/HIVE-17012
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Rajesh Balamohan
> Labels: performance
> Attachments: plan.txt
>
>
> {code}
> Map 1: 446/446 Reducer 2: 2/2 Reducer 3: 2/2
> --
> Compile Query 0.24s
> Prepare Plan0.35s
> Submit Plan 0.18s
> Start DAG 0.21s
> Run DAG 32332.27s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11390343.00 0 0 2,879,987,999
> 2,879,987,999
> Reducer 2 31281225.00 0 0 2,750,387,156
> 0
> Reducer 3 751498.00 0 0 129,600,843
> 0
> --
> {code}
> Time taken: 32438.42 seconds to insert <3B rows with
> {code}
> create table store_sales
> (
> ss_sold_time_sk bigint,
> ss_item_skbigint,
> ss_customer_skbigint,
> ss_cdemo_sk bigint,
> ss_hdemo_sk bigint,
> ss_addr_skbigint,
> ss_store_sk bigint,
> ss_promo_sk bigint,
> ss_ticket_number bigint,
> ss_quantity int,
> ss_wholesale_cost double,
> ss_list_price double,
> ss_sales_pricedouble,
> ss_ext_discount_amt double,
> ss_ext_sales_pricedouble,
> ss_ext_wholesale_cost double,
> ss_ext_list_price double,
> ss_ext_taxdouble,
> ss_coupon_amt double,
> ss_net_paid double,
> ss_net_paid_inc_tax double,
> ss_net_profit double
> )
> partitioned by (ss_sold_date_sk bigint)
> CLUSTERED BY (ss_ticket_number) INTO 2 BUCKETS
> STORED AS ORC
> TBLPROPERTIES ('transactional'='true', 'transactional_properties'='default')
> ;
> from tpcds_text_1000.store_sales ss
> insert into table store_sales partition (ss_sold_date_sk)
> select
> ss.ss_sold_time_sk,
> ss.ss_item_sk,
> ss.ss_customer_sk,
> ss.ss_cdemo_sk,
> ss.ss_hdemo_sk,
> ss.ss_addr_sk,
> ss.ss_store_sk,
> ss.ss_promo_sk,
> ss.ss_ticket_number,
> ss.ss_quantity,
> ss.ss_wholesale_cost,
> ss.ss_list_price,
> ss.ss_sales_price,
> ss.ss_ext_discount_amt,
> ss.ss_ext_sales_price,
> ss.ss_ext_wholesale_cost,
> ss.ss_ext_list_price,
> ss.ss_ext_tax,
> ss.ss_coupon_amt,
> ss.ss_net_paid,
> ss.ss_net_paid_inc_tax,
> ss.ss_net_profit,
> ss.ss_sold_date_sk
> where ss.ss_sold_date_sk is not null
> insert into table store_sales partition (ss_sold_date_sk)
> select
> ss.ss_sold_time_sk,
> ss.ss_item_sk,
> ss.ss_customer_sk,
> ss.ss_cdemo_sk,
> ss.ss_hdemo_sk,
> ss.ss_addr_sk,
> ss.ss_store_sk,
> ss.ss_promo_sk,
> ss.ss_ticket_number,
> ss.ss_quantity,
> ss.ss_wholesale_cost,
> ss.ss_list_price,
> ss.ss_sales_price,
> ss.ss_ext_discount_amt,
>
[jira] [Comment Edited] (HIVE-17012) ACID Table: Number of reduce tasks should be computed correctly when sort.dynamic.partition is enabled
[
https://issues.apache.org/jira/browse/HIVE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071990#comment-16071990
]
Rajesh Balamohan edited comment on HIVE-17012 at 7/5/17 8:54 AM:
-
ReducerTraits would be {{FIXED}} for ACID tables with buckets.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L102
prevents from computing reducer tasks for Reducer 3.
was (Author: rajesh.balamohan):
ReducerTraits would be FIXED for ACID tables with buckets.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L102
prevents from computing reducer tasks for Reducer 3.
> ACID Table: Number of reduce tasks should be computed correctly when
> sort.dynamic.partition is enabled
> --
>
> Key: HIVE-17012
> URL: https://issues.apache.org/jira/browse/HIVE-17012
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Rajesh Balamohan
> Labels: performance
> Attachments: plan.txt
>
>
> {code}
> Map 1: 446/446 Reducer 2: 2/2 Reducer 3: 2/2
> --
> Compile Query 0.24s
> Prepare Plan0.35s
> Submit Plan 0.18s
> Start DAG 0.21s
> Run DAG 32332.27s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11390343.00 0 0 2,879,987,999
> 2,879,987,999
> Reducer 2 31281225.00 0 0 2,750,387,156
> 0
> Reducer 3 751498.00 0 0 129,600,843
> 0
> --
> {code}
> Time taken: 32438.42 seconds to insert <3B rows with
> {code}
> create table store_sales
> (
> ss_sold_time_sk bigint,
> ss_item_skbigint,
> ss_customer_skbigint,
> ss_cdemo_sk bigint,
> ss_hdemo_sk bigint,
> ss_addr_skbigint,
> ss_store_sk bigint,
> ss_promo_sk bigint,
> ss_ticket_number bigint,
> ss_quantity int,
> ss_wholesale_cost double,
> ss_list_price double,
> ss_sales_pricedouble,
> ss_ext_discount_amt double,
> ss_ext_sales_pricedouble,
> ss_ext_wholesale_cost double,
> ss_ext_list_price double,
> ss_ext_taxdouble,
> ss_coupon_amt double,
> ss_net_paid double,
> ss_net_paid_inc_tax double,
> ss_net_profit double
> )
> partitioned by (ss_sold_date_sk bigint)
> CLUSTERED BY (ss_ticket_number) INTO 2 BUCKETS
> STORED AS ORC
> TBLPROPERTIES ('transactional'='true', 'transactional_properties'='default')
> ;
> from tpcds_text_1000.store_sales ss
> insert into table store_sales partition (ss_sold_date_sk)
> select
> ss.ss_sold_time_sk,
> ss.ss_item_sk,
> ss.ss_customer_sk,
> ss.ss_cdemo_sk,
> ss.ss_hdemo_sk,
> ss.ss_addr_sk,
> ss.ss_store_sk,
> ss.ss_promo_sk,
> ss.ss_ticket_number,
> ss.ss_quantity,
> ss.ss_wholesale_cost,
> ss.ss_list_price,
> ss.ss_sales_price,
> ss.ss_ext_discount_amt,
> ss.ss_ext_sales_price,
> ss.ss_ext_wholesale_cost,
> ss.ss_ext_list_price,
> ss.ss_ext_tax,
> ss.ss_coupon_amt,
> ss.ss_net_paid,
> ss.ss_net_paid_inc_tax,
> ss.ss_net_profit,
> ss.ss_sold_date_sk
> where ss.ss_sold_date_sk is not null
> insert into table store_sales partition (ss_sold_date_sk)
> select
> ss.ss_sold_time_sk,
> ss.ss_item_sk,
> ss.ss_customer_sk,
> ss.ss_cdemo_sk,
> ss.ss_hdemo_sk,
> ss.ss_addr_sk,
> ss.ss_store_sk,
> ss.ss_promo_sk,
> ss.ss_ticket_number,
> ss.ss_quantity,
> ss.ss_wholesale_cost,
> ss.ss_list_price,
> ss.ss_sales_price,
> ss.ss_ext_discount_amt,
>
[jira] [Updated] (HIVE-17035) Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked
[
https://issues.apache.org/jira/browse/HIVE-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-17035:
Attachment: HIVE-17035.1.patch
\cc [~jcamachorodriguez], [~ashutoshc]
> Optimizer: Lineage transform() should be invoked after rest of the optimizers
> are invoked
> -
>
> Key: HIVE-17035
> URL: https://issues.apache.org/jira/browse/HIVE-17035
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-17035.1.patch
>
>
> In a fairly large query which had tens of left join, time taken to create
> linageInfo itself took 1500+ seconds. This is due to the fact that the table
> had lots of columns and in some processing, it ended up having 7000+ value
> columns in {{ReduceSinkLineage}}.
> It would be good to invoke lineage transform when rest of the optimizers in
> {{Optimizer}} are invoked. This would avoid help in improving the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17035) Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked
[
https://issues.apache.org/jira/browse/HIVE-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-17035:
Assignee: Rajesh Balamohan
Status: Patch Available (was: Open)
> Optimizer: Lineage transform() should be invoked after rest of the optimizers
> are invoked
> -
>
> Key: HIVE-17035
> URL: https://issues.apache.org/jira/browse/HIVE-17035
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-17035.1.patch
>
>
> In a fairly large query which had tens of left join, time taken to create
> linageInfo itself took 1500+ seconds. This is due to the fact that the table
> had lots of columns and in some processing, it ended up having 7000+ value
> columns in {{ReduceSinkLineage}}.
> It would be good to invoke lineage transform when rest of the optimizers in
> {{Optimizer}} are invoked. This would avoid help in improving the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[
https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-17011:
Attachment: HIVE-17011.3.patch
Thanks [~jcamachorodriguez]. Attaching .2 patch with additional method
{{getOperatorSignature}}. This patch resets the signature on setters.
> PlanUtils.getExprList() causes memory pressure when invoked from
> SharedWorkOptimizer on queries with lots of union operators
>
>
> Key: HIVE-17011
> URL: https://issues.apache.org/jira/browse/HIVE-17011
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: CPU_usage.png, HIVE-17011.1.patch, HIVE-17011.3.patch,
> Memory_usage.png
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[
https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072357#comment-16072357
]
Rajesh Balamohan edited comment on HIVE-17011 at 7/3/17 12:37 PM:
--
Thanks [~jcamachorodriguez]. Attaching .3 patch with additional method
{{getOperatorSignature}}. This patch resets the signature on setters.
was (Author: rajesh.balamohan):
Thanks [~jcamachorodriguez]. Attaching .2 patch with additional method
{{getOperatorSignature}}. This patch resets the signature on setters.
> PlanUtils.getExprList() causes memory pressure when invoked from
> SharedWorkOptimizer on queries with lots of union operators
>
>
> Key: HIVE-17011
> URL: https://issues.apache.org/jira/browse/HIVE-17011
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: CPU_usage.png, HIVE-17011.1.patch, HIVE-17011.3.patch,
> Memory_usage.png
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17012) ACID Table: Number of reduce tasks should be computed correctly when sort.dynamic.partition is enabled
[ https://issues.apache.org/jira/browse/HIVE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071990#comment-16071990 ] Rajesh Balamohan commented on HIVE-17012: - ReducerTraits would be FIXED for ACID tables with buckets. https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L102 prevents from computing reducer tasks for Reducer 3. > ACID Table: Number of reduce tasks should be computed correctly when > sort.dynamic.partition is enabled > -- > > Key: HIVE-17012 > URL: https://issues.apache.org/jira/browse/HIVE-17012 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: plan.txt > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17012) ACID Table: Number of reduce tasks should be computed correctly when sort.dynamic.partition is enabled
[ https://issues.apache.org/jira/browse/HIVE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17012: Attachment: plan.txt > ACID Table: Number of reduce tasks should be computed correctly when > sort.dynamic.partition is enabled > -- > > Key: HIVE-17012 > URL: https://issues.apache.org/jira/browse/HIVE-17012 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: plan.txt > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[ https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17011: Priority: Minor (was: Major) > PlanUtils.getExprList() causes memory pressure when invoked from > SharedWorkOptimizer on queries with lots of union operators > > > Key: HIVE-17011 > URL: https://issues.apache.org/jira/browse/HIVE-17011 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: CPU_usage.png, HIVE-17011.1.patch, Memory_usage.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[
https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-17011:
Attachment: HIVE-17011.1.patch
Attaching .1 patch.
Tested on a query which had 100s of unionall on simple table.
Without patch, SharedWorkOptimizer::transform took {{323.667 seconds}}
With Patch, SharedWorkOptimizer::transform {{116.395 seconds}}
\cc [~jcamachorodriguez]
> PlanUtils.getExprList() causes memory pressure when invoked from
> SharedWorkOptimizer on queries with lots of union operators
>
>
> Key: HIVE-17011
> URL: https://issues.apache.org/jira/browse/HIVE-17011
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: CPU_usage.png, HIVE-17011.1.patch, Memory_usage.png
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[ https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17011: Status: Patch Available (was: Open) > PlanUtils.getExprList() causes memory pressure when invoked from > SharedWorkOptimizer on queries with lots of union operators > > > Key: HIVE-17011 > URL: https://issues.apache.org/jira/browse/HIVE-17011 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: CPU_usage.png, HIVE-17011.1.patch, Memory_usage.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17011) PlanUtils.getExprList() causes memory pressure when invoked from SharedWorkOptimizer on queries with lots of union operators
[ https://issues.apache.org/jira/browse/HIVE-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-17011: Attachment: CPU_usage.png Memory_usage.png > PlanUtils.getExprList() causes memory pressure when invoked from > SharedWorkOptimizer on queries with lots of union operators > > > Key: HIVE-17011 > URL: https://issues.apache.org/jira/browse/HIVE-17011 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: CPU_usage.png, Memory_usage.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16987) Llap: StackOverFlow in EvictionDispatcher:notifyEvicted
[
https://issues.apache.org/jira/browse/HIVE-16987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067817#comment-16067817
]
Rajesh Balamohan commented on HIVE-16987:
-
LGTM. +1
> Llap: StackOverFlow in EvictionDispatcher:notifyEvicted
> ---
>
> Key: HIVE-16987
> URL: https://issues.apache.org/jira/browse/HIVE-16987
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16987.patch
>
>
> Env: hive master (commit:6de5d1d4ba40269a0bf5ad6ee42797021541fe97) + tez
> master
> Query: insert overwrite with dynamic partitions
> {noformat}
> errorMessage=Cannot recover from this
> error:org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.StackOverflowError
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.StackOverflowError
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 17 more
> Caused by: java.io.IOException: java.lang.StackOverflowError
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.rethrowErrorIfAny(LlapRecordReader.java:307)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.nextCvb(LlapRecordReader.java:262)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:202)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:65)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ... 23 more
> Caused by: java.lang.StackOverflowError
> at
> org.apache.hadoop.hive.llap.cache.EvictionDispatcher.notifyEvicted(EvictionDispatcher.java:44)
> at
> org.apache.hadoop.hive.llap.cache.SerDeLowLevelCacheImpl$LlapSerDeDataBuffer.notifyEvicted(SerDeLowLevelCacheImpl.java:60)
> at
> org.apache.hadoop.hive.llap.cache.EvictionDispatcher.notifyEvicted(EvictionDispatcher.java:44)
> at
>
[jira] [Updated] (HIVE-16903) LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE
[ https://issues.apache.org/jira/browse/HIVE-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16903: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks [~gopalv]. Committed to master. > LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE > -- > > Key: HIVE-16903 > URL: https://issues.apache.org/jira/browse/HIVE-16903 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-16903.1.patch > > > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java#L130 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-16903) LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE
[ https://issues.apache.org/jira/browse/HIVE-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-16903: --- Assignee: Rajesh Balamohan > LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE > -- > > Key: HIVE-16903 > URL: https://issues.apache.org/jira/browse/HIVE-16903 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-16903.1.patch > > > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java#L130 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16903) LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE
[ https://issues.apache.org/jira/browse/HIVE-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16903: Attachment: HIVE-16903.1.patch \cc [~sseth]. > LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE > -- > > Key: HIVE-16903 > URL: https://issues.apache.org/jira/browse/HIVE-16903 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-16903.1.patch > > > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java#L130 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16903) LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE
[ https://issues.apache.org/jira/browse/HIVE-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16903: Status: Patch Available (was: Open) > LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE > -- > > Key: HIVE-16903 > URL: https://issues.apache.org/jira/browse/HIVE-16903 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-16903.1.patch > > > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java#L130 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16903) LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE
[ https://issues.apache.org/jira/browse/HIVE-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16903: Component/s: llap > LLAP: Fix config name issue in SHUFFLE_MANAGE_OS_CACHE > -- > > Key: HIVE-16903 > URL: https://issues.apache.org/jira/browse/HIVE-16903 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Priority: Trivial > > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java#L130 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[
https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037800#comment-16037800
]
Rajesh Balamohan commented on HIVE-16323:
-
SynchronizedMetaStoreClient is used only in load-dynamic-partition threads.
Should {{ObjectStore.shutdodown()}} set {{pm}} to null as this can be invoked
lots of times?. Also getPartition() called via loadPartition() should be using
"getSychronizedMSC()::getPartitionWithAuthInfo()" to be on safer side.
> HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
> ---
>
> Key: HIVE-16323
> URL: https://issues.apache.org/jira/browse/HIVE-16323
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16323.1.patch, HIVE-16323.2.patch,
> HIVE-16323.3.patch, PM_leak.png
>
>
> Hive.loadDynamicPartitions creates threads with new embedded rawstore, but
> never close them, thus we leak PersistenceManager one per such thread.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time
[ https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15300: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the review [~ashutoshc]. Committed to master. > Reuse table information in SemanticAnalyzer::getMetaData to reduce > compilation time > --- > > Key: HIVE-15300 > URL: https://issues.apache.org/jira/browse/HIVE-15300 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch, > HIVE-15300.3.patch, HIVE-15300.4.patch, HIVE-15300.5.patch > > > E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the > table details for the same table repeatedly. It took 20+seconds to compile > the query. > It would be good to reuse the table information in > SemanticAnalyzer::getMetadata. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time
[ https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15300: Attachment: HIVE-15300.5.patch Reuploading the patch with minor fix (i.e adding to cache only table is not null). "cte_3, cte_4" tests are passing locally. > Reuse table information in SemanticAnalyzer::getMetaData to reduce > compilation time > --- > > Key: HIVE-15300 > URL: https://issues.apache.org/jira/browse/HIVE-15300 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch, > HIVE-15300.3.patch, HIVE-15300.4.patch, HIVE-15300.5.patch > > > E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the > table details for the same table repeatedly. It took 20+seconds to compile > the query. > It would be good to reuse the table information in > SemanticAnalyzer::getMetadata. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16746:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the review [~ashutoshc]. Committed to master.
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022552#comment-16022552
]
Rajesh Balamohan commented on HIVE-16746:
-
RB: https://reviews.apache.org/r/59517/
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022540#comment-16022540
]
Rajesh Balamohan commented on HIVE-16746:
-
test failures are not related to this patch.
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-16746:
---
Assignee: Rajesh Balamohan
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16746:
Attachment: HIVE-16746.1.patch
With patch, response time came down from 185 seconds to 140-145 seconds. This
benefits irrespective of store implementation. Currently CachedStore does not
yet implement index caching.
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16746:
Status: Patch Available (was: Open)
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time
[ https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15300: Attachment: HIVE-15300.4.patch Rebasing the patch with minor fixes. This is mainly for ObjectStore. If CachedStore is used, getTable wouldn't be expensive. > Reuse table information in SemanticAnalyzer::getMetaData to reduce > compilation time > --- > > Key: HIVE-15300 > URL: https://issues.apache.org/jira/browse/HIVE-15300 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch, > HIVE-15300.3.patch, HIVE-15300.4.patch > > > E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the > table details for the same table repeatedly. It took 20+seconds to compile > the query. > It would be good to reuse the table information in > SemanticAnalyzer::getMetadata. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows
[
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022142#comment-16022142
]
Rajesh Balamohan commented on HIVE-16737:
-
+1. LGTM.
> LLAP: Shuffle handler TCP listen queue overflows
>
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737.2.patch,
> HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0 50 *:15551*:*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows
[
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022086#comment-16022086
]
Rajesh Balamohan commented on HIVE-16737:
-
Minor: would be nice to add backlog detail in the log
{{LOG.info("LlapShuffleHandler" + " listening on port " + port);}}
> LLAP: Shuffle handler TCP listen queue overflows
>
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0 50 *:15551*:*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16692) LLAP: Keep alive connection in shuffle handler should not be closed until entire data is flushed out
[
https://issues.apache.org/jira/browse/HIVE-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16692:
Attachment: HIVE-16692.addendum.patch
Thanks [~sseth]. Realized that there is a minor corner case that needs to be
handled which is added in the addendum patch. Plz let me know if it is fine to
add it in this jira. Otherwise it can be a separate jira.
> LLAP: Keep alive connection in shuffle handler should not be closed until
> entire data is flushed out
>
>
> Key: HIVE-16692
> URL: https://issues.apache.org/jira/browse/HIVE-16692
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16692.1.patch, HIVE-16692.addendum.patch
>
>
> In corner cases with keep-alive enabled, it is possible that the headers are
> written out in the response and downstream was able to read the headers.
> But possible that the mapOutput construction took a lot longer time (due to
> disk or any other issue) in server side. In the mean time, keep alive timeout
> can kick in and close the connection from server side. In such cases, there
> is a possibility that downstream can get "connection reset". Ideally keep
> alive should kick in only after flushing entire response downstream.
> e.g error msg in client side
> {noformat}
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:209)
> ~[?:1.8.0_112]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:675)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> ~[?:1.8.0_112]
> at
> org.apache.tez.http.HttpConnection.getInputStream(HttpConnection.java:260)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupConnection(Fetcher.java:460)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:492)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:417)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:215)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:73)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> This corner case handling was not pulled in earlier from MR handler fixes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16692) LLAP: Keep alive connection in shuffle handler should not be closed until entire data is flushed out
[
https://issues.apache.org/jira/browse/HIVE-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16692:
Status: Patch Available (was: Open)
> LLAP: Keep alive connection in shuffle handler should not be closed until
> entire data is flushed out
>
>
> Key: HIVE-16692
> URL: https://issues.apache.org/jira/browse/HIVE-16692
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16692.1.patch
>
>
> In corner cases with keep-alive enabled, it is possible that the headers are
> written out in the response and downstream was able to read the headers.
> But possible that the mapOutput construction took a lot longer time (due to
> disk or any other issue) in server side. In the mean time, keep alive timeout
> can kick in and close the connection from server side. In such cases, there
> is a possibility that downstream can get "connection reset". Ideally keep
> alive should kick in only after flushing entire response downstream.
> e.g error msg in client side
> {noformat}
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:209)
> ~[?:1.8.0_112]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:675)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> ~[?:1.8.0_112]
> at
> org.apache.tez.http.HttpConnection.getInputStream(HttpConnection.java:260)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupConnection(Fetcher.java:460)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:492)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:417)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:215)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:73)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> This corner case handling was not pulled in earlier from MR handler fixes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16692) LLAP: Keep alive connection in shuffle handler should not be closed until entire data is flushed out
[
https://issues.apache.org/jira/browse/HIVE-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-16692:
---
Assignee: Rajesh Balamohan
> LLAP: Keep alive connection in shuffle handler should not be closed until
> entire data is flushed out
>
>
> Key: HIVE-16692
> URL: https://issues.apache.org/jira/browse/HIVE-16692
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16692.1.patch
>
>
> In corner cases with keep-alive enabled, it is possible that the headers are
> written out in the response and downstream was able to read the headers.
> But possible that the mapOutput construction took a lot longer time (due to
> disk or any other issue) in server side. In the mean time, keep alive timeout
> can kick in and close the connection from server side. In such cases, there
> is a possibility that downstream can get "connection reset". Ideally keep
> alive should kick in only after flushing entire response downstream.
> e.g error msg in client side
> {noformat}
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:209)
> ~[?:1.8.0_112]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:675)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> ~[?:1.8.0_112]
> at
> org.apache.tez.http.HttpConnection.getInputStream(HttpConnection.java:260)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupConnection(Fetcher.java:460)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:492)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:417)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:215)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:73)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> This corner case handling was not pulled in earlier from MR handler fixes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16692) LLAP: Keep alive connection in shuffle handler should not be closed until entire data is flushed out
[
https://issues.apache.org/jira/browse/HIVE-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16692:
Attachment: HIVE-16692.1.patch
> LLAP: Keep alive connection in shuffle handler should not be closed until
> entire data is flushed out
>
>
> Key: HIVE-16692
> URL: https://issues.apache.org/jira/browse/HIVE-16692
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16692.1.patch
>
>
> In corner cases with keep-alive enabled, it is possible that the headers are
> written out in the response and downstream was able to read the headers.
> But possible that the mapOutput construction took a lot longer time (due to
> disk or any other issue) in server side. In the mean time, keep alive timeout
> can kick in and close the connection from server side. In such cases, there
> is a possibility that downstream can get "connection reset". Ideally keep
> alive should kick in only after flushing entire response downstream.
> e.g error msg in client side
> {noformat}
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:209)
> ~[?:1.8.0_112]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_112]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
> ~[?:1.8.0_112]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:675)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
> ~[?:1.8.0_112]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> ~[?:1.8.0_112]
> at
> org.apache.tez.http.HttpConnection.getInputStream(HttpConnection.java:260)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupConnection(Fetcher.java:460)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:492)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:417)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:215)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:73)
> ~[tez-runtime-library-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at
> org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.8.4.2.6.1.0-11.jar:0.8.4.2.6.1.0-11]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> This corner case handling was not pulled in earlier from MR handler fixes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-15942) Q22 does not get vectorized due to grouping set evaluations
[
https://issues.apache.org/jira/browse/HIVE-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan resolved HIVE-15942.
-
Resolution: Fixed
Checked with apache master. {{Map 1}} gets vectorized now with Q22. Marking
this ticket as fixed.
> Q22 does not get vectorized due to grouping set evaluations
> ---
>
> Key: HIVE-15942
> URL: https://issues.apache.org/jira/browse/HIVE-15942
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: query_plan_q22_HIVE-15942.txt
>
>
> Env: apache hive master with LLAP + tez master
> Query: q22 @ tpcds 10 TB scale
> Map-1 does not get vectorized. Hive logs point out that group set evaluation
> is preventing it from vectorization
> {noformat}
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [VALUE._col0, VALUE._col1, VALUE._col2]
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> GBY using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=VALUE._col0, 1=VALUE._col1, 2=VALUE._col2},
> scratchColumnTypeNames []
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process going to
> walk the operator stack to get vectorization context for RS
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: walkStackToFindVectorizationContext GBY has new
> vectorization context Context name GBY, level 0, sorted projectionColumnMap
> {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator RS
> using vectorization contextContext name GBY, level 0, sorted
> projectionColumnMap {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating MapWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Grouping sets not
> supported
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Pruning grouping set
> id not supported
> 2017-02-16T07:10:06,085 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Vectorizing ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: vectorizeReduceWork reducer Operator: SEL...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [KEY.reducesinkkey0, KEY.reducesinkkey1,
> KEY.reducesinkkey2, KEY.reducesinkkey3, KEY.reducesinkkey4]
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> SEL using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=KEY.reducesinkkey0, 1=KEY.reducesinkkey1,
> 2=KEY.reducesinkkey2, 3=KEY.reducesinkkey3, 4=KEY.reducesinkkey4},
> scratchColumnTypeNames []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16548) LLAP: EncodedReaderImpl.addOneCompressionBuffer throws NPE
[
https://issues.apache.org/jira/browse/HIVE-16548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988194#comment-15988194
]
Rajesh Balamohan commented on HIVE-16548:
-
This happened as a part of regular data population (create table blah ..select)
script. Only one query was executed and this happened in couple of queries. Log
file is really large to be uploaded here. Will check on trimming it.
> LLAP: EncodedReaderImpl.addOneCompressionBuffer throws NPE
> --
>
> Key: HIVE-16548
> URL: https://issues.apache.org/jira/browse/HIVE-16548
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>
> Env: Based on apr-25 apache master codebase.
> {noformat}
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Buffer
> size too small. size = 65536 needed = 3762509
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:695)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:454)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:420)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:242)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:239)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: Buffer size too small. size =
> 65536 needed = 3762509
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.addOneCompressionBuffer(EncodedReaderImpl.java:1223)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.prepareRangesForCompressedRead(EncodedReaderImpl.java:813)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:685)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 17 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:695)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:454)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:420)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:242)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:239)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 6 more
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Commented] (HIVE-16353) Jetty 9 upgrade breaks hive master which is 2.3.0-snapshot
[
https://issues.apache.org/jira/browse/HIVE-16353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982369#comment-15982369
]
Rajesh Balamohan commented on HIVE-16353:
-
Thanks [~gopalv]. Patch lgtm. +1.
> Jetty 9 upgrade breaks hive master which is 2.3.0-snapshot
> --
>
> Key: HIVE-16353
> URL: https://issues.apache.org/jira/browse/HIVE-16353
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16353.patch
>
>
> HIVE-16049 upgraded to jetty 9. It is committed to apache master which is
> still 2.3.0-snapshot. This breaks couple of other components like LLAP and
> ends up throwing the following error during runtime.
> {noformat}
> 2017-04-02T20:17:45,435 WARN [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP
> Daemon with exception
> java.lang.NoClassDefFoundError: org/eclipse/jetty/rewrite/handler/Rule
> at org.apache.hive.http.HttpServer$Builder.build(HttpServer.java:125)
> ~[hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.daemon.services.impl.LlapWebServices.serviceInit(LlapWebServices.java:102)
> ~[hive-llap-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ~[hadoop-common-2.7.3.2.5.0.0-1245.jar:?]
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> ~[hadoop-common-2.7.3.2.5.0.0-1245.jar:?]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:385)
> ~[hive-llap-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ~[hadoop-common-2.7.3.2.5.0.0-1245.jar:?]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:528)
> [hive-llap-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> Caused by: java.lang.ClassNotFoundException:
> org.eclipse.jetty.rewrite.handler.Rule
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_77]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_77]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_77]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_77]
> ... 7 more
> 2017-04-02T20:17:45,441 INFO [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown
> invoked
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968393#comment-15968393
]
Rajesh Balamohan commented on HIVE-16436:
-
[~prasanth_j] - I have not seen negative values in the response times. I tried
with .2 patch and it works fine. I am fine with either of the patches. :)
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch, HIVE-16436.2.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967338#comment-15967338
]
Rajesh Balamohan edited comment on HIVE-16436 at 4/13/17 9:28 AM:
--
{{perfLogger.PerfLogEnd}} was getting invoked for every report causing this
issue.
\cc [~anishek], [~gopalv]
was (Author: rajesh.balamohan):
{{perfLogger.PerfLogEnd}} was getting invoked for every report causing this
issue.
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16436:
Status: Patch Available (was: Open)
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967338#comment-15967338
]
Rajesh Balamohan commented on HIVE-16436:
-
{{perfLogger.PerfLogEnd}} was getting invoked for every report causing this
issue.
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-16436:
---
Assignee: Rajesh Balamohan
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16436:
Attachment: HIVE-16436.1.patch
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16044: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks [~gopalv]. Committed to master. > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16044.1.patch, HIVE-16044.2.patch, > wihtoutPatch_Llap_shuffleHandler.png, withPatch_llap_shuffleHanlder.png > > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.profiler.png
HIVE-16406.2.patch
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.2.patch,
> HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: (was: HIVE-16406.profiler.png)
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Status: Patch Available (was: Open)
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.1.patch
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.profiler.png
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16290) Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when minValue == filterValue
[
https://issues.apache.org/jira/browse/HIVE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16290:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.3.0
Status: Resolved (was: Patch Available)
Thanks [~gopalv]. Committed to master.
> Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when
> minValue == filterValue
> -
>
> Key: HIVE-16290
> URL: https://issues.apache.org/jira/browse/HIVE-16290
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.3.0
>
> Attachments: HIVE-16290.1.patch, HIVE-16290.2.patch
>
>
> Issue:
> =
> In {{StatsRulesProcFactory::evaluateCompator}}, when {{minValue}} is >=
> filtered {{value}}, it should return all rows. Currently, it returns
> {{numRows/3}}. This causes lesser number of reducers to be spun up in
> queries. E.g Q79 in TPC-DS.
> E.g: TPC-DS store table stats:
> =
> {noformat}
> hive --orcfiledump
> hdfs://nn:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/store/00_0
> Stripe Statistics:
> Stripe 1:
> Column 0: count: 1002 hasNull: false
> Column 1: count: 1002 hasNull: false min: 1 max: 1002 sum: 502503
> Column 2: count: 1002 hasNull: false min: AABA max:
> PPBA sum: 16032
> Column 3: count: 1002 hasNull: false min: max: 2001-03-13 sum: 9950
> Column 4: count: 1002 hasNull: false min: max: 2001-03-12 sum: 5010
> Column 5: count: 273 hasNull: true min: 2450820 max: 2451313 sum:
> 669141525
> Column 6: count: 1002 hasNull: false min: max: pri sum: 3916
> Column 7: count: 994 hasNull: true min: 200 max: 300 sum: 249970
> Column 8: count: 996 hasNull: true min: 5002549 max: 9997773 sum:
> 7382689071
> Column 9: count: 1002 hasNull: false min: max: 8AM-8AM sum: 7088
> select compute_stats(s_employee_count, 16) from store;
> {"columntype":"Long","min":200,"max":300,"countnulls":8,"numdistinctvalues":63,"ndvbitvector":"{0,
> 1, 2, 3, 4, 5, 11, 12}{0, 1, 2, 3, 6}{0, 1, 2, 3, 4, 5, 7, 11}{0, 1, 2, 3,
> 4, 5, 7}{0, 1, 2, 3, 4, 5, 6}{0, 1, 2, 3, 4, 5, 8}{0, 1, 2, 3, 4}{0, 1, 2, 3,
> 4, 5, 7, 9}{0, 1, 2, 3, 4}{0}{0, 1, 2, 3, 4, 5, 7}{0, 1, 2, 3, 4, 5, 6, 7}{0,
> 1, 2, 3, 4, 8, 9, 14}{0, 1, 2, 3, 5}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1, 2, 3, 4,
> 5, 6, 8}"}
> {noformat}
> {noformat}
> explain select count(s_store_sk) from store where s_number_employees > 200
> and s_number_employees < 295;
> {noformat}
> Above query would first apply 1002/3 = 334 for {{s_number_employees > 200}}
> and then 334 / 3 = 111 for {{s_number_employees < 295}}. Ideally it should
> return all 1002 rows for filter {{s_number_employees > 200}}.
> In TPC-DS Q79, this causes too less reduce tasks to be spun up causing
> runtime delays.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16290) Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when minValue == filterValue
[
https://issues.apache.org/jira/browse/HIVE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16290:
Status: Patch Available (was: Open)
> Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when
> minValue == filterValue
> -
>
> Key: HIVE-16290
> URL: https://issues.apache.org/jira/browse/HIVE-16290
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16290.1.patch
>
>
> Issue:
> =
> In {{StatsRulesProcFactory::evaluateCompator}}, when {{minValue}} is >=
> filtered {{value}}, it should return all rows. Currently, it returns
> {{numRows/3}}. This causes lesser number of reducers to be spun up in
> queries. E.g Q79 in TPC-DS.
> E.g: TPC-DS store table stats:
> =
> {noformat}
> hive --orcfiledump
> hdfs://nn:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/store/00_0
> Stripe Statistics:
> Stripe 1:
> Column 0: count: 1002 hasNull: false
> Column 1: count: 1002 hasNull: false min: 1 max: 1002 sum: 502503
> Column 2: count: 1002 hasNull: false min: AABA max:
> PPBA sum: 16032
> Column 3: count: 1002 hasNull: false min: max: 2001-03-13 sum: 9950
> Column 4: count: 1002 hasNull: false min: max: 2001-03-12 sum: 5010
> Column 5: count: 273 hasNull: true min: 2450820 max: 2451313 sum:
> 669141525
> Column 6: count: 1002 hasNull: false min: max: pri sum: 3916
> Column 7: count: 994 hasNull: true min: 200 max: 300 sum: 249970
> Column 8: count: 996 hasNull: true min: 5002549 max: 9997773 sum:
> 7382689071
> Column 9: count: 1002 hasNull: false min: max: 8AM-8AM sum: 7088
> select compute_stats(s_employee_count, 16) from store;
> {"columntype":"Long","min":200,"max":300,"countnulls":8,"numdistinctvalues":63,"ndvbitvector":"{0,
> 1, 2, 3, 4, 5, 11, 12}{0, 1, 2, 3, 6}{0, 1, 2, 3, 4, 5, 7, 11}{0, 1, 2, 3,
> 4, 5, 7}{0, 1, 2, 3, 4, 5, 6}{0, 1, 2, 3, 4, 5, 8}{0, 1, 2, 3, 4}{0, 1, 2, 3,
> 4, 5, 7, 9}{0, 1, 2, 3, 4}{0}{0, 1, 2, 3, 4, 5, 7}{0, 1, 2, 3, 4, 5, 6, 7}{0,
> 1, 2, 3, 4, 8, 9, 14}{0, 1, 2, 3, 5}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1, 2, 3, 4,
> 5, 6, 8}"}
> {noformat}
> {noformat}
> explain select count(s_store_sk) from store where s_number_employees > 200
> and s_number_employees < 295;
> {noformat}
> Above query would first apply 1002/3 = 334 for {{s_number_employees > 200}}
> and then 334 / 3 = 111 for {{s_number_employees < 295}}. Ideally it should
> return all 1002 rows for filter {{s_number_employees > 200}}.
> In TPC-DS Q79, this causes too less reduce tasks to be spun up causing
> runtime delays.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16290) Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when minValue == filterValue
[
https://issues.apache.org/jira/browse/HIVE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16290:
Description:
Issue:
=
In {{StatsRulesProcFactory::evaluateCompator}}, when {{minValue}} is >=
filtered {{value}}, it should return all rows. Currently, it returns
{{numRows/3}}. This causes lesser number of reducers to be spun up in queries.
E.g Q79 in TPC-DS.
E.g: TPC-DS store table stats:
=
{noformat}
hive --orcfiledump
hdfs://nn:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/store/00_0
Stripe Statistics:
Stripe 1:
Column 0: count: 1002 hasNull: false
Column 1: count: 1002 hasNull: false min: 1 max: 1002 sum: 502503
Column 2: count: 1002 hasNull: false min: AABA max:
PPBA sum: 16032
Column 3: count: 1002 hasNull: false min: max: 2001-03-13 sum: 9950
Column 4: count: 1002 hasNull: false min: max: 2001-03-12 sum: 5010
Column 5: count: 273 hasNull: true min: 2450820 max: 2451313 sum: 669141525
Column 6: count: 1002 hasNull: false min: max: pri sum: 3916
Column 7: count: 994 hasNull: true min: 200 max: 300 sum: 249970
Column 8: count: 996 hasNull: true min: 5002549 max: 9997773 sum: 7382689071
Column 9: count: 1002 hasNull: false min: max: 8AM-8AM sum: 7088
select compute_stats(s_employee_count, 16) from store;
{"columntype":"Long","min":200,"max":300,"countnulls":8,"numdistinctvalues":63,"ndvbitvector":"{0,
1, 2, 3, 4, 5, 11, 12}{0, 1, 2, 3, 6}{0, 1, 2, 3, 4, 5, 7, 11}{0, 1, 2, 3, 4,
5, 7}{0, 1, 2, 3, 4, 5, 6}{0, 1, 2, 3, 4, 5, 8}{0, 1, 2, 3, 4}{0, 1, 2, 3, 4,
5, 7, 9}{0, 1, 2, 3, 4}{0}{0, 1, 2, 3, 4, 5, 7}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1,
2, 3, 4, 8, 9, 14}{0, 1, 2, 3, 5}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1, 2, 3, 4, 5, 6,
8}"}
{noformat}
{noformat}
explain select count(s_store_sk) from store where s_number_employees > 200 and
s_number_employees < 295;
{noformat}
Above query would first apply 1002/3 = 334 for {{s_number_employees > 200}} and
then 334 / 3 = 111 for {{s_number_employees < 295}}. Ideally it should return
all 1002 rows for filter {{s_number_employees > 200}}.
In TPC-DS Q79, this causes too less reduce tasks to be spun up causing runtime
delays.
was:
Issue:
=
In {{StatsRulesProcFactory::evaluateCompator}}, when {{minValue}} is >=
filtered {{value}}, it should return all rows. Currently, it returns
{{numRows/3}}. This lesser number of reducers to be spun up in queries. E.g Q79
in TPC-DS.
E.g: TPC-DS store table stats:
=
{noformat}
hive --orcfiledump
hdfs://nn:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/store/00_0
Stripe Statistics:
Stripe 1:
Column 0: count: 1002 hasNull: false
Column 1: count: 1002 hasNull: false min: 1 max: 1002 sum: 502503
Column 2: count: 1002 hasNull: false min: AABA max:
PPBA sum: 16032
Column 3: count: 1002 hasNull: false min: max: 2001-03-13 sum: 9950
Column 4: count: 1002 hasNull: false min: max: 2001-03-12 sum: 5010
Column 5: count: 273 hasNull: true min: 2450820 max: 2451313 sum: 669141525
Column 6: count: 1002 hasNull: false min: max: pri sum: 3916
Column 7: count: 994 hasNull: true min: 200 max: 300 sum: 249970
Column 8: count: 996 hasNull: true min: 5002549 max: 9997773 sum: 7382689071
Column 9: count: 1002 hasNull: false min: max: 8AM-8AM sum: 7088
select compute_stats(s_employee_count, 16) from store;
{"columntype":"Long","min":200,"max":300,"countnulls":8,"numdistinctvalues":63,"ndvbitvector":"{0,
1, 2, 3, 4, 5, 11, 12}{0, 1, 2, 3, 6}{0, 1, 2, 3, 4, 5, 7, 11}{0, 1, 2, 3, 4,
5, 7}{0, 1, 2, 3, 4, 5, 6}{0, 1, 2, 3, 4, 5, 8}{0, 1, 2, 3, 4}{0, 1, 2, 3, 4,
5, 7, 9}{0, 1, 2, 3, 4}{0}{0, 1, 2, 3, 4, 5, 7}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1,
2, 3, 4, 8, 9, 14}{0, 1, 2, 3, 5}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1, 2, 3, 4, 5, 6,
8}"}
{noformat}
{noformat}
explain select count(s_store_sk) from store where s_number_employees > 200 and
s_number_employees < 295;
{noformat}
Above query would first apply 1002/3 = 334 for {{s_number_employees > 200}} and
then 334 / 3 = 111 for {{s_number_employees < 295}}. Ideally it should return
all 1002 rows for filter {{s_number_employees > 200}}.
In TPC-DS Q79, this causes too less reduce tasks to be spun up causing runtime
delays.
> Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when
> minValue == filterValue
> -
>
> Key: HIVE-16290
> URL: https://issues.apache.org/jira/browse/HIVE-16290
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16290.1.patch
>
>
> Issue:
> =
> In
[jira] [Updated] (HIVE-16290) Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when minValue == filterValue
[
https://issues.apache.org/jira/browse/HIVE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16290:
Attachment: HIVE-16290.1.patch
> Stats: StatsRulesProcFactory::evaluateComparator estimates are wrong when
> minValue == filterValue
> -
>
> Key: HIVE-16290
> URL: https://issues.apache.org/jira/browse/HIVE-16290
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16290.1.patch
>
>
> Issue:
> =
> In {{StatsRulesProcFactory::evaluateCompator}}, when {{minValue}} is >=
> filtered {{value}}, it should return all rows. Currently, it returns
> {{numRows/3}}. This lesser number of reducers to be spun up in queries. E.g
> Q79 in TPC-DS.
> E.g: TPC-DS store table stats:
> =
> {noformat}
> hive --orcfiledump
> hdfs://nn:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/store/00_0
> Stripe Statistics:
> Stripe 1:
> Column 0: count: 1002 hasNull: false
> Column 1: count: 1002 hasNull: false min: 1 max: 1002 sum: 502503
> Column 2: count: 1002 hasNull: false min: AABA max:
> PPBA sum: 16032
> Column 3: count: 1002 hasNull: false min: max: 2001-03-13 sum: 9950
> Column 4: count: 1002 hasNull: false min: max: 2001-03-12 sum: 5010
> Column 5: count: 273 hasNull: true min: 2450820 max: 2451313 sum:
> 669141525
> Column 6: count: 1002 hasNull: false min: max: pri sum: 3916
> Column 7: count: 994 hasNull: true min: 200 max: 300 sum: 249970
> Column 8: count: 996 hasNull: true min: 5002549 max: 9997773 sum:
> 7382689071
> Column 9: count: 1002 hasNull: false min: max: 8AM-8AM sum: 7088
> select compute_stats(s_employee_count, 16) from store;
> {"columntype":"Long","min":200,"max":300,"countnulls":8,"numdistinctvalues":63,"ndvbitvector":"{0,
> 1, 2, 3, 4, 5, 11, 12}{0, 1, 2, 3, 6}{0, 1, 2, 3, 4, 5, 7, 11}{0, 1, 2, 3,
> 4, 5, 7}{0, 1, 2, 3, 4, 5, 6}{0, 1, 2, 3, 4, 5, 8}{0, 1, 2, 3, 4}{0, 1, 2, 3,
> 4, 5, 7, 9}{0, 1, 2, 3, 4}{0}{0, 1, 2, 3, 4, 5, 7}{0, 1, 2, 3, 4, 5, 6, 7}{0,
> 1, 2, 3, 4, 8, 9, 14}{0, 1, 2, 3, 5}{0, 1, 2, 3, 4, 5, 6, 7}{0, 1, 2, 3, 4,
> 5, 6, 8}"}
> {noformat}
> {noformat}
> explain select count(s_store_sk) from store where s_number_employees > 200
> and s_number_employees < 295;
> {noformat}
> Above query would first apply 1002/3 = 334 for {{s_number_employees > 200}}
> and then 334 / 3 = 111 for {{s_number_employees < 295}}. Ideally it should
> return all 1002 rows for filter {{s_number_employees > 200}}.
> In TPC-DS Q79, this causes too less reduce tasks to be spun up causing
> runtime delays.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16252) Vectorization: Cannot vectorize: Aggregation Function UDF avg
[
https://issues.apache.org/jira/browse/HIVE-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934887#comment-15934887
]
Rajesh Balamohan commented on HIVE-16252:
-
Observed slow performance with this query as opposed to sum function.
> Vectorization: Cannot vectorize: Aggregation Function UDF avg
> --
>
> Key: HIVE-16252
> URL: https://issues.apache.org/jira/browse/HIVE-16252
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Rajesh Balamohan
>
> {noformat}
> select
> ss_store_sk, ss_item_sk, avg(ss_sales_price) as revenue
> from
> store_sales, date_dim
> where
> ss_sold_date_sk = d_date_sk
> and d_month_seq between 1212 and 1212 + 11
> group by ss_store_sk , ss_item_sk limit 100;
> 2017-03-20T00:59:49,526 INFO [680a4c08-1639-4bb9-8d6a-0bf0f30ef563 main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-03-20T00:59:49,526 DEBUG [680a4c08-1639-4bb9-8d6a-0bf0f30ef563 main]
> physical.Vectorizer: Using reduce tag 0
> 2017-03-20T00:59:49,527 DEBUG [680a4c08-1639-4bb9-8d6a-0bf0f30ef563 main]
> lazybinary.LazyBinarySerDe: LazyBinarySerDe initialized with:
> columnNames=[_col0] columnTypes=[struct]
> 2017-03-20T00:59:49,527 DEBUG [680a4c08-1639-4bb9-8d6a-0bf0f30ef563 main]
> vector.VectorizationContext: Input Expression = Column[KEY._col0], Vectorized
> Expression = col 0
> ...
> ...
> 2017-03-20T00:59:49,528 INFO [680a4c08-1639-4bb9-8d6a-0bf0f30ef563 main]
> physical.Vectorizer: Cannot vectorize: Aggregation Function UDF avg parameter
> expression for GROUPBY operator: Data type
> struct of Column[VALUE._col0] not
> supported
> {noformat}
> Env: Hive build from: commit 71f4930d95475e7e63b5acc55af3809aefcc71e0 (march
> 16)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16208) Vectorization: ProcessingModeHashAggregate::sumBatchSize is never reset
[
https://issues.apache.org/jira/browse/HIVE-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929844#comment-15929844
]
Rajesh Balamohan commented on HIVE-16208:
-
+1.
> Vectorization: ProcessingModeHashAggregate::sumBatchSize is never reset
> ---
>
> Key: HIVE-16208
> URL: https://issues.apache.org/jira/browse/HIVE-16208
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16208.1.patch
>
>
> When processing >2x the hash-table size in the vectorized group-by, the check
> for fall-back to streaming is wrong because {{sumBatchSize*minReduction}} is
> not reset when processing the next split.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure
[
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902119#comment-15902119
]
Rajesh Balamohan commented on HIVE-16150:
-
lgtm. +1. Thanks [~prasanth_j].
"LOG.info("Not using LLAP IO because it is not initialized");" may not be
wrapped in isInfoEnabled.
> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " +
> pathToPartitionInfo);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16142) ATSHook NPE via LLAP
[ https://issues.apache.org/jira/browse/HIVE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900603#comment-15900603 ] Rajesh Balamohan commented on HIVE-16142: - Is HIVE-16066 similar to this one? > ATSHook NPE via LLAP > > > Key: HIVE-16142 > URL: https://issues.apache.org/jira/browse/HIVE-16142 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16142.01.patch > > > Exceptions in the log of the form: > 2017-03-06T15:42:30,046 WARN [ATS Logger 0]: hooks.ATSHook > (ATSHook.java:run(318)) - Failed to submit to ATS for > hive_20170306154227_f41bc7cb-1a2f-40f1-a85b-b2bc260a451a > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:608) > ~[hive-exec-2.1.0.2.6.0.0-585.jar:2.1.0.2.6.0.0-585] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16078) improve abort checking in Tez/LLAP
[
https://issues.apache.org/jira/browse/HIVE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898976#comment-15898976
]
Rajesh Balamohan commented on HIVE-16078:
-
lgtm. +1.
> improve abort checking in Tez/LLAP
> --
>
> Key: HIVE-16078
> URL: https://issues.apache.org/jira/browse/HIVE-16078
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16078.01.patch, HIVE-16078.02.patch,
> HIVE-16078.03.patch, HIVE-16078.patch
>
>
> Sometimes, a fragment can run for a long time after a query fails. It looks
> from logs like the abort/interrupt were called correctly on the thread, yet
> the thread hangs around minutes after, doing the below. Other tasks for the
> same job appear to have exited correctly, after the same abort logic (at
> least, the same log lines, fwiw)
> {noformat}
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByValue(VectorCopyRow.java:317)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:263)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
>
[jira] [Updated] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16116:
Attachment: HIVE-16116.2.patch
Thanks [~pvary]. Yes, it affects {{setHiveVariables}} as well. Uploading the
revised patch.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch, HIVE-16116.2.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16116:
Attachment: HIVE-16116.1.patch
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option. Patch
fixes that case.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896792#comment-15896792
]
Rajesh Balamohan edited comment on HIVE-16116 at 3/6/17 6:26 AM:
-
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option.
Attaching the patch which fixes this case.
was (Author: rajesh.balamohan):
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option. Patch
fixes that case.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16116:
Status: Patch Available (was: Open)
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-16116:
---
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16078) improve abort checking in Tez/LLAP
[
https://issues.apache.org/jira/browse/HIVE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891566#comment-15891566
]
Rajesh Balamohan commented on HIVE-16078:
-
{noformat}
int newRows = Math.min(CHECK_INTERRUPTION_AFTER_ROWS_MAX,
Math.max(1, (int) (diff * checkInterruptionAfterRows)));
{noformat}
Would be good to have a min limit for this. Otherwise, based on the {{diff}},
{{newRows}} can be computed as very low value and that could trigger lots of
isInterrupted() checks.
Also, do we need similar checks in CommonMergeJoinOperator?
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L381
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L462
> improve abort checking in Tez/LLAP
> --
>
> Key: HIVE-16078
> URL: https://issues.apache.org/jira/browse/HIVE-16078
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16078.patch
>
>
> Sometimes, a fragment can run for a long time after a query fails. It looks
> from logs like the abort/interrupt were called correctly on the thread, yet
> the thread hangs around minutes after, doing the below. Other tasks for the
> same job appear to have exited correctly, after the same abort logic (at
> least, the same log lines, fwiw)
> {noformat}
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByValue(VectorCopyRow.java:317)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:263)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
> at
>
[jira] [Commented] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891525#comment-15891525 ] Rajesh Balamohan commented on HIVE-16082: - lgtm. +1 pending tests. > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16066) NPE in ExplainTask
[
https://issues.apache.org/jira/browse/HIVE-16066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889337#comment-15889337
]
Rajesh Balamohan commented on HIVE-16066:
-
This is not part of explain formatted or explain extended. Got this exception
as a part of ATSHook.
> NPE in ExplainTask
> --
>
> Key: HIVE-16066
> URL: https://issues.apache.org/jira/browse/HIVE-16066
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> {noformat}
> 2017-02-28T20:05:13,412 WARN [ATS Logger 0] hooks.ATSHook: Failed to submit
> plan to ATS for user_20170228200511_b05d6eaf-7599-4539-919c-5d3df8658c99
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:803)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:817)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputList(ExplainTask.java:658)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:984)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputMap(ExplainTask.java:592)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:970)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:691)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:1059)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputStagePlans(ExplainTask.java:1203)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExplainTask.getJSONPlan(ExplainTask.java:306)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16044: Attachment: HIVE-16044.2.patch Uploading same patch. > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-16044.1.patch, HIVE-16044.2.patch, > wihtoutPatch_Llap_shuffleHandler.png, withPatch_llap_shuffleHanlder.png > > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16044: Status: Patch Available (was: Open) > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-16044.1.patch, > wihtoutPatch_Llap_shuffleHandler.png, withPatch_llap_shuffleHanlder.png > > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16044: Priority: Minor (was: Major) > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-16044.1.patch, > wihtoutPatch_Llap_shuffleHandler.png, withPatch_llap_shuffleHanlder.png > > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-16044: Attachment: withPatch_llap_shuffleHanlder.png wihtoutPatch_Llap_shuffleHandler.png HIVE-16044.1.patch Attaching .1 patch and the tcpdump with/without patch for shuffle handler. This is the same as MAPREDUCE-6850. Created this jira for Hive, since LLAP has its own ShuffleHandler. > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: HIVE-16044.1.patch, > wihtoutPatch_Llap_shuffleHandler.png, withPatch_llap_shuffleHanlder.png > > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-16044: --- > LLAP: Shuffle Handler keep-alive connections are closed from the server side > > > Key: HIVE-16044 > URL: https://issues.apache.org/jira/browse/HIVE-16044 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > > LLAP's shufflehandler could be closing the keep-alive connections after > output is served. This could break the connection from server side. JDK http > logs may not be revealing this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15879) Fix HiveMetaStoreChecker.checkPartitionDirs method
[
https://issues.apache.org/jira/browse/HIVE-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882640#comment-15882640
]
Rajesh Balamohan commented on HIVE-15879:
-
>>>
it returns only approximate number of threads and it cannot be guaranteed that
it always returns the exact number of active threads. This still exposes the
method implementation to the msck hang bug in rare corner cases.
>>>
My comment on ThreadPoolExecutor.getActiveCount was for the msck-hang bug. That
bug should not surface with master branch.
Proposed change should not impact the single partition use case (e.g single
partition column having 10K partitions would not be impacted with this).
However, if there are multiple partition columns, iterative approach would be
better than the recursive approach. In recursive model, it gets blocked with in
higher level and proceeds in single threaded approach.
> Fix HiveMetaStoreChecker.checkPartitionDirs method
> --
>
> Key: HIVE-15879
> URL: https://issues.apache.org/jira/browse/HIVE-15879
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15879.01.patch, HIVE-15879.02.patch
>
>
> HIVE-15803 fixes the msck hang issue in
> HiveMetaStoreChecker.checkPartitionDirs method by adding a check to see if
> the Threadpool has any spare threads. If not it uses single threaded listing
> of the files.
> {noformat}
> if (pool != null) {
> synchronized (pool) {
> // In case of recursive calls, it is possible to deadlock with TP.
> Check TP usage here.
> if (pool.getActiveCount() < pool.getMaximumPoolSize()) {
> useThreadPool = true;
> }
> if (!useThreadPool) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Not using threadPool as active count:" +
> pool.getActiveCount()
> + ", max:" + pool.getMaximumPoolSize());
> }
> }
> }
> }
> {noformat}
> Based on the java doc of getActiveCount() below
> bq. Returns the approximate number of threads that are actively executing
> tasks.
> it returns only approximate number of threads and it cannot be guaranteed
> that it always returns the exact number of active threads. This still exposes
> the method implementation to the msck hang bug in rare corner cases.
> We could either:
> 1. Use a atomic counter to track exactly how many threads are actively running
> 2. Relook at the method itself to make it much simpler. Like eg, look into
> the possibility of changing the recursive implementation to an iterative
> implementation where worker threads pick tasks from a queue until the queue
> is empty.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-15879) Fix HiveMetaStoreChecker.checkPartitionDirs method
[
https://issues.apache.org/jira/browse/HIVE-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881827#comment-15881827
]
Rajesh Balamohan edited comment on HIVE-15879 at 2/24/17 2:53 AM:
--
Thanks for sharing the details [~vihangk1]. I have a different point of view
here.
I agree that ThreadPoolExecutor.getActiveCount() is approximate. It is
approximate because, by the time {{getActiveCount()}} iterates over the
running threads in the worker list, it is possible that some of the threads
which were executing are complete.
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#l1818.
So the reported numbers could be slightly higher than the actually running
threads. But it would never be less, as new Worker in ThreadPoolExecutor is
added with {{mainLock}}.
In the context of MSCK logic, this approximation should not be a problem.
This is due to the check of "(pool.getActiveCount() <
pool.getMaximumPoolSize())". In case threadpool executor reports approximate
value (i.e higher than the actual number of threads), thread pool would not be
used as per current logic. So in corner cases there can be instances where in
threadpool executor could have been used, but failed due to the approximate
(higher values) reported by ThreadPoolExecutor.
was (Author: rajesh.balamohan):
Thanks for sharing the details [~vihangk1]. I have a different point of view
here.
I agree that ThreadPoolExecutor.getActiveCount() is approximate. It is
approximate because, by the time {{getActiveCount()}} iterates over the
running threads in the worker list, it is possible that some of the threads
which were executing are complete.
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#l1818.
So the reported numbers could be slightly higher than the actually running
threads. But it would never be less, as new Worker in ThreadPoolExecutor is
added with {{mainLock}}.
In the context of MSCK logic, this approximation should not be a problem. This
is due to the check of "(pool.getActiveCount() < pool.getMaximumPoolSize())".
In case threadpool executor reports approximate value (i.e higher than the
actual number of threads), thread pool would not be used as per current logic.
So in corner cases there can be instances where in threadpool executor could
have been used, but failed due to the approximate (higher values) reported by
ThreadPoolExecutor.
> Fix HiveMetaStoreChecker.checkPartitionDirs method
> --
>
> Key: HIVE-15879
> URL: https://issues.apache.org/jira/browse/HIVE-15879
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15879.01.patch
>
>
> HIVE-15803 fixes the msck hang issue in
> HiveMetaStoreChecker.checkPartitionDirs method by adding a check to see if
> the Threadpool has any spare threads. If not it uses single threaded listing
> of the files.
> {noformat}
> if (pool != null) {
> synchronized (pool) {
> // In case of recursive calls, it is possible to deadlock with TP.
> Check TP usage here.
> if (pool.getActiveCount() < pool.getMaximumPoolSize()) {
> useThreadPool = true;
> }
> if (!useThreadPool) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Not using threadPool as active count:" +
> pool.getActiveCount()
> + ", max:" + pool.getMaximumPoolSize());
> }
> }
> }
> }
> {noformat}
> Based on the java doc of getActiveCount() below
> bq. Returns the approximate number of threads that are actively executing
> tasks.
> it returns only approximate number of threads and it cannot be guaranteed
> that it always returns the exact number of active threads. This still exposes
> the method implementation to the msck hang bug in rare corner cases.
> We could either:
> 1. Use a atomic counter to track exactly how many threads are actively running
> 2. Relook at the method itself to make it much simpler. Like eg, look into
> the possibility of changing the recursive implementation to an iterative
> implementation where worker threads pick tasks from a queue until the queue
> is empty.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15879) Fix HiveMetaStoreChecker.checkPartitionDirs method
[
https://issues.apache.org/jira/browse/HIVE-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881827#comment-15881827
]
Rajesh Balamohan commented on HIVE-15879:
-
Thanks for sharing the details [~vihangk1]. I have a different point of view
here.
I agree that ThreadPoolExecutor.getActiveCount() is approximate. It is
approximate because, by the time {{getActiveCount()}} iterates over the
running threads in the worker list, it is possible that some of the threads
which were executing are complete.
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#l1818.
So the reported numbers could be slightly higher than the actually running
threads. But it would never be less, as new Worker in ThreadPoolExecutor is
added with {{mainLock}}.
In the context of MSCK logic, this approximation should not be a problem. This
is due to the check of "(pool.getActiveCount() < pool.getMaximumPoolSize())".
In case threadpool executor reports approximate value (i.e higher than the
actual number of threads), thread pool would not be used as per current logic.
So in corner cases there can be instances where in threadpool executor could
have been used, but failed due to the approximate (higher values) reported by
ThreadPoolExecutor.
> Fix HiveMetaStoreChecker.checkPartitionDirs method
> --
>
> Key: HIVE-15879
> URL: https://issues.apache.org/jira/browse/HIVE-15879
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15879.01.patch
>
>
> HIVE-15803 fixes the msck hang issue in
> HiveMetaStoreChecker.checkPartitionDirs method by adding a check to see if
> the Threadpool has any spare threads. If not it uses single threaded listing
> of the files.
> {noformat}
> if (pool != null) {
> synchronized (pool) {
> // In case of recursive calls, it is possible to deadlock with TP.
> Check TP usage here.
> if (pool.getActiveCount() < pool.getMaximumPoolSize()) {
> useThreadPool = true;
> }
> if (!useThreadPool) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Not using threadPool as active count:" +
> pool.getActiveCount()
> + ", max:" + pool.getMaximumPoolSize());
> }
> }
> }
> }
> {noformat}
> Based on the java doc of getActiveCount() below
> bq. Returns the approximate number of threads that are actively executing
> tasks.
> it returns only approximate number of threads and it cannot be guaranteed
> that it always returns the exact number of active threads. This still exposes
> the method implementation to the msck hang bug in rare corner cases.
> We could either:
> 1. Use a atomic counter to track exactly how many threads are actively running
> 2. Relook at the method itself to make it much simpler. Like eg, look into
> the possibility of changing the recursive implementation to an iterative
> implementation where worker threads pick tasks from a queue until the queue
> is empty.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881701#comment-15881701 ] Rajesh Balamohan commented on HIVE-15964: - committed the addendum patch in master. > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch, > HIVE-15964.3.patch, HIVE-15964.4.patch, HIVE-15964.addendum.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Attachment: HIVE-15964.addendum.patch Yes [~prasanth_j]. Attaching the addendum patch for orc_ppd_schema_evol_3a.q.out. Will commit shortly. > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch, > HIVE-15964.3.patch, HIVE-15964.4.patch, HIVE-15964.addendum.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16020) LLAP : Reduce IPC connection misses
[
https://issues.apache.org/jira/browse/HIVE-16020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16020:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks [~sershe]. Committed to master.
> LLAP : Reduce IPC connection misses
> ---
>
> Key: HIVE-16020
> URL: https://issues.apache.org/jira/browse/HIVE-16020
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 2.2.0
>
> Attachments: HIVE-16020.1.patch
>
>
> {{RPC.getProxy}} created in {{TaskRunnerCallable}} does not pass
> SocketFactory. This would cause new SocketFactory to be created every time
> which would cause connection reuse issues. Also, UserGroupInformation can be
> reused at query level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~prasanth_j] and [~sershe]. Committed to master. > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch, > HIVE-15964.3.patch, HIVE-15964.4.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Attachment: HIVE-15964.4.patch > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch, > HIVE-15964.3.patch, HIVE-15964.4.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16014) HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread for pool size
[
https://issues.apache.org/jira/browse/HIVE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880136#comment-15880136
]
Rajesh Balamohan commented on HIVE-16014:
-
Thanks for creating this jira. Proposed change sounds reasonable.
> HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of
> hive.mv.files.thread for pool size
> --
>
> Key: HIVE-16014
> URL: https://issues.apache.org/jira/browse/HIVE-16014
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> HiveMetastoreChecker uses hive.mv.files.thread configuration value for
> determining the pool size as below :
> {noformat}
> private void checkPartitionDirs(Path basePath, Set allDirs, int
> maxDepth) throws IOException, HiveException {
> ConcurrentLinkedQueue basePaths = new ConcurrentLinkedQueue<>();
> basePaths.add(basePath);
> Set dirSet = Collections.newSetFromMap(new ConcurrentHashMap Boolean>());
> // Here we just reuse the THREAD_COUNT configuration for
> // HIVE_MOVE_FILES_THREAD_COUNT
> int poolSize = conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname,
> 15);
> // Check if too low config is provided for move files. 2x CPU is
> reasonable max count.
> poolSize = poolSize == 0 ? poolSize : Math.max(poolSize,
> Runtime.getRuntime().availableProcessors() * 2);
> {noformat}
> msck is commonly used to add the missing partitions for the table from the
> Filesystem. In such a case different pool sizes for HMSHandler and
> HiveMetastoreChecker can affect the performance. Eg. If
> {{hive.metastore.fshandler.threads}} is set to a lower value like 15 and
> {{hive.mv.files.thread}} is much higher like 100 or vice versa the smaller
> pool will become the bottleneck. If would be good to use
> {{hive.metastore.fshandler.threads}} to size the pool for
> HiveMetastoreChecker since the number missing partitions and number of
> partitions to be added will most likely be the same. In such a case the
> performance of the query will be optimum when both the pool sizes are same.
> Since it is possible to tune both the configs individually it will be very
> likely that they may be different. But since there is a strong co-relation
> between amount of work done by HiveMetastoreChecker and
> HiveMetastore.add_partitions call it might be a good idea to use
> {{hive.metastore.fshandler.threads}} for pool size instead of
> {{hive.mv.files.thread}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Attachment: HIVE-15964.3.patch addressing review comments from [~prasanth_j]. Added -ve test case as well. > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch, > HIVE-15964.3.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16020) LLAP : Reduce IPC connection misses
[
https://issues.apache.org/jira/browse/HIVE-16020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16020:
Attachment: HIVE-16020.1.patch
Changes:
1. Common SocketFactory passed to ContainerRunnerImpl, AMReporter,
TaskRunnerCallable.
2. UserGroupInformation passed at query level (this is part of [~sseth]'s
patch). Initial patch could be to provide at query level; can create subsequent
jira to cache at app level.
> LLAP : Reduce IPC connection misses
> ---
>
> Key: HIVE-16020
> URL: https://issues.apache.org/jira/browse/HIVE-16020
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
> Attachments: HIVE-16020.1.patch
>
>
> {{RPC.getProxy}} created in {{TaskRunnerCallable}} does not pass
> SocketFactory. This would cause new SocketFactory to be created every time
> which would cause connection reuse issues. Also, UserGroupInformation can be
> reused at query level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16020) LLAP : Reduce IPC connection misses
[
https://issues.apache.org/jira/browse/HIVE-16020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16020:
Status: Patch Available (was: Open)
> LLAP : Reduce IPC connection misses
> ---
>
> Key: HIVE-16020
> URL: https://issues.apache.org/jira/browse/HIVE-16020
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
> Attachments: HIVE-16020.1.patch
>
>
> {{RPC.getProxy}} created in {{TaskRunnerCallable}} does not pass
> SocketFactory. This would cause new SocketFactory to be created every time
> which would cause connection reuse issues. Also, UserGroupInformation can be
> reused at query level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15992) LLAP: NPE in LlapTaskCommunicator.getCompletedLogsUrl for unsuccessful attempt
[
https://issues.apache.org/jira/browse/HIVE-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15992:
Status: Patch Available (was: Open)
> LLAP: NPE in LlapTaskCommunicator.getCompletedLogsUrl for unsuccessful attempt
> --
>
> Key: HIVE-15992
> URL: https://issues.apache.org/jira/browse/HIVE-15992
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15992.1.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.getCompletedLogsUrl(LlapTaskCommunicator.java:563)
> at
> org.apache.tez.dag.app.TaskCommunicatorWrapper.getCompletedLogsUrl(TaskCommunicatorWrapper.java:92)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.getCompletedLogsUrl(TaskCommunicatorManager.java:674)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.getCompletedLogsUrl(TaskAttemptImpl.java:1147)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.logJobHistoryAttemptUnsuccesfulCompletion(TaskAttemptImpl.java:1089)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminateTransition.transition(TaskAttemptImpl.java:1332)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminatedBeforeRunningTransition.transition(TaskAttemptImpl.java:1430)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminatedBeforeRunningTransition.transition(TaskAttemptImpl.java:1416)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:799)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:122)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2296)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2281)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15992) LLAP: NPE in LlapTaskCommunicator.getCompletedLogsUrl for unsuccessful attempt
[
https://issues.apache.org/jira/browse/HIVE-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15992:
Attachment: HIVE-15992.1.patch
> LLAP: NPE in LlapTaskCommunicator.getCompletedLogsUrl for unsuccessful attempt
> --
>
> Key: HIVE-15992
> URL: https://issues.apache.org/jira/browse/HIVE-15992
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15992.1.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.getCompletedLogsUrl(LlapTaskCommunicator.java:563)
> at
> org.apache.tez.dag.app.TaskCommunicatorWrapper.getCompletedLogsUrl(TaskCommunicatorWrapper.java:92)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.getCompletedLogsUrl(TaskCommunicatorManager.java:674)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.getCompletedLogsUrl(TaskAttemptImpl.java:1147)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.logJobHistoryAttemptUnsuccesfulCompletion(TaskAttemptImpl.java:1089)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminateTransition.transition(TaskAttemptImpl.java:1332)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminatedBeforeRunningTransition.transition(TaskAttemptImpl.java:1430)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl$TerminatedBeforeRunningTransition.transition(TaskAttemptImpl.java:1416)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:799)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:122)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2296)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2281)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15941) Fix o.a.h.hive.ql.exec.tez.TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15941: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~sseth]. Committed to master. > Fix o.a.h.hive.ql.exec.tez.TezTask compilation issue with tez master > > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15941.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Attachment: HIVE-15964.2.patch Thanks [~sershe]. Attaching .2 version. Removed isAcidScan and added TODO for acid. Added llapreader.q.out for testclidriver. > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15964.1.patch, HIVE-15964.2.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Status: Patch Available (was: Open) > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15964.1.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Attachment: HIVE-15964.1.patch \cc [~prasanth_j] > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15964.1.patch > > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-15964: --- > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15964) LLAP: Llap IO codepath not getting invoked due to file column id mismatch
[ https://issues.apache.org/jira/browse/HIVE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15964: Component/s: llap > LLAP: Llap IO codepath not getting invoked due to file column id mismatch > - > > Key: HIVE-15964 > URL: https://issues.apache.org/jira/browse/HIVE-15964 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > > LLAP IO codepath is not getting invoked in certain cases when schema > evolution checks are done. Though "int --> long" (fileType to readerType) > conversions are allowed, the file type columns are not matched correctly when > such conversions need to happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15942) Q22 does not get vectorized due to grouping set evaluations
[
https://issues.apache.org/jira/browse/HIVE-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871029#comment-15871029
]
Rajesh Balamohan commented on HIVE-15942:
-
Mostly should get fixed with HIVE-14016. Need to check after HIVE-14016 gets
committed.
> Q22 does not get vectorized due to grouping set evaluations
> ---
>
> Key: HIVE-15942
> URL: https://issues.apache.org/jira/browse/HIVE-15942
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: query_plan_q22_HIVE-15942.txt
>
>
> Env: apache hive master with LLAP + tez master
> Query: q22 @ tpcds 10 TB scale
> Map-1 does not get vectorized. Hive logs point out that group set evaluation
> is preventing it from vectorization
> {noformat}
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [VALUE._col0, VALUE._col1, VALUE._col2]
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> GBY using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=VALUE._col0, 1=VALUE._col1, 2=VALUE._col2},
> scratchColumnTypeNames []
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process going to
> walk the operator stack to get vectorization context for RS
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: walkStackToFindVectorizationContext GBY has new
> vectorization context Context name GBY, level 0, sorted projectionColumnMap
> {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator RS
> using vectorization contextContext name GBY, level 0, sorted
> projectionColumnMap {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating MapWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Grouping sets not
> supported
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Pruning grouping set
> id not supported
> 2017-02-16T07:10:06,085 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Vectorizing ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: vectorizeReduceWork reducer Operator: SEL...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [KEY.reducesinkkey0, KEY.reducesinkkey1,
> KEY.reducesinkkey2, KEY.reducesinkkey3, KEY.reducesinkkey4]
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> SEL using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=KEY.reducesinkkey0, 1=KEY.reducesinkkey1,
> 2=KEY.reducesinkkey2, 3=KEY.reducesinkkey3, 4=KEY.reducesinkkey4},
> scratchColumnTypeNames []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15942) Q22 does not get vectorized due to grouping set evaluations
[
https://issues.apache.org/jira/browse/HIVE-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15942:
Attachment: query_plan_q22_HIVE-15942.txt
> Q22 does not get vectorized due to grouping set evaluations
> ---
>
> Key: HIVE-15942
> URL: https://issues.apache.org/jira/browse/HIVE-15942
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: query_plan_q22_HIVE-15942.txt
>
>
> Env: apache hive master with LLAP + tez master
> Query: q22 @ tpcds 10 TB scale
> Map-1 does not get vectorized. Hive logs point out that group set evaluation
> is preventing it from vectorization
> {noformat}
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [VALUE._col0, VALUE._col1, VALUE._col2]
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> GBY using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=VALUE._col0, 1=VALUE._col1, 2=VALUE._col2},
> scratchColumnTypeNames []
> 2017-02-16T07:10:06,074 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process going to
> walk the operator stack to get vectorization context for RS
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: walkStackToFindVectorizationContext GBY has new
> vectorization context Context name GBY, level 0, sorted projectionColumnMap
> {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator RS
> using vectorization contextContext name GBY, level 0, sorted
> projectionColumnMap {0=_col0, 1=_col1, 2=_col2}, scratchColumnTypeNames []
> 2017-02-16T07:10:06,075 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating MapWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Grouping sets not
> supported
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,084 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Cannot vectorize: GROUPBY operator: Pruning grouping set
> id not supported
> 2017-02-16T07:10:06,085 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Validating ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: Vectorizing ReduceWork...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: vectorizeReduceWork reducer Operator: SEL...
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process
> reduceColumnNames [KEY.reducesinkkey0, KEY.reducesinkkey1,
> KEY.reducesinkkey2, KEY.reducesinkkey3, KEY.reducesinkkey4]
> 2017-02-16T07:10:06,086 INFO [c9d014ef-5a60-4ef1-b7a8-5209da679ebf main]
> physical.Vectorizer: ReduceWorkVectorizationNodeProcessor process operator
> SEL using vectorization contextContext name __Reduce_Shuffle__, level 0,
> sorted projectionColumnMap {0=KEY.reducesinkkey0, 1=KEY.reducesinkkey1,
> 2=KEY.reducesinkkey2, 3=KEY.reducesinkkey3, 4=KEY.reducesinkkey4},
> scratchColumnTypeNames []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15941) Fix o.a.h.hive.ql.exec.tez.TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15941: Summary: Fix o.a.h.hive.ql.exec.tez.TezTask compilation issue with tez master (was: Fix TezTask compilation issue with tez master) > Fix o.a.h.hive.ql.exec.tez.TezTask compilation issue with tez master > > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15941.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15941) Fix TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15941: Status: Patch Available (was: Open) > Fix TezTask compilation issue with tez master > - > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15941.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15941) Fix TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869519#comment-15869519 ] Rajesh Balamohan commented on HIVE-15941: - Reported by [~anishek] > Fix TezTask compilation issue with tez master > - > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15941.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15941) Fix TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-15941: Attachment: HIVE-15941.1.patch > Fix TezTask compilation issue with tez master > - > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-15941.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15941) Fix TezTask compilation issue with tez master
[ https://issues.apache.org/jira/browse/HIVE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-15941: --- > Fix TezTask compilation issue with tez master > - > > Key: HIVE-15941 > URL: https://issues.apache.org/jira/browse/HIVE-15941 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
