[jira] [Created] (HIVE-27188) Explore usage of FilterApi.in(C column, Set values) in Parquet instead of nested OR
Rajesh Balamohan created HIVE-27188:
---
Summary: Explore usage of FilterApi.in(C column, Set values) in
Parquet instead of nested OR
Key: HIVE-27188
URL: https://issues.apache.org/jira/browse/HIVE-27188
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Following query can throw stackoverflow exception with "Xss256K".
Currently it generates nested OR filter
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/FilterPredicateLeafBuilder.java#L43-L52]
Instead, need to explore the possibility of using {color:#de350b}FilterApi.in(C
column, Set values) {color:#172b4d}in parquet{color}{color}
{noformat}
drop table if exists test;
create external table test (i int) stored as parquet;
insert into test values (1),(2),(3);
select count(*) from test where i in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72,
73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92,
93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141,
142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157,
158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173,
174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189,
190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221,
222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237,
238, 239, 240, 241, 242, 243);
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27184) Add class name profiling option in ProfileServlet
Rajesh Balamohan created HIVE-27184: --- Summary: Add class name profiling option in ProfileServlet Key: HIVE-27184 URL: https://issues.apache.org/jira/browse/HIVE-27184 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan With async-profiler "-e classame.method", it is possible to profile specific events. Currently profileServlet supports events like cpu, alloc, lock etc. It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27183) Iceberg: Table information is loaded multiple times
Rajesh Balamohan created HIVE-27183:
---
Summary: Iceberg: Table information is loaded multiple times
Key: HIVE-27183
URL: https://issues.apache.org/jira/browse/HIVE-27183
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
HMS::getTable invokes "HiveIcebergMetaHook::postGetTable" which internally
loads iceberg table again.
If this isn't needed or needed only for show-create-table, do not load the
table again.
{noformat}
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x00066f84eef0> (a
java.util.concurrent.CompletableFuture$Signaller)
at
java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at
java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.18/CompletableFuture.java:1796)
at
java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.18/ForkJoinPool.java:3128)
at
java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.18/CompletableFuture.java:1823)
at
java.util.concurrent.CompletableFuture.get(java.base@11.0.18/CompletableFuture.java:1998)
at org.apache.hadoop.util.functional.FutureIO.awaitFuture(FutureIO.java:77)
at
org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:196)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:263)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:177)
at
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$609/0x000840e18040.apply(Unknown
Source)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$610/0x000840e18440.run(Unknown
Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:153)
at
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)
at
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)
at
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:115)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:105)
at
org.apache.iceberg.mr.hive.IcebergTableUtil.lambda$getTable$1(IcebergTableUtil.java:99)
at
org.apache.iceberg.mr.hive.IcebergTableUtil$$Lambda$552/0x000840d59840.apply(Unknown
Source)
at
org.apache.iceberg.mr.hive.IcebergTableUtil.lambda$getTable$4(IcebergTableUtil.java:111)
at
org.apache.iceberg.mr.hive.IcebergTableUtil$$Lambda$557/0x000840d58c40.get(Unknown
Source)
at java.util.Optional.orElseGet(java.base@11.0.18/Optional.java:369)
at
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:108)
at
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:69)
at
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:73)
at
org.apache.iceberg.mr.hive.HiveIcebergMetaHook.postGetTable(HiveIcebergMetaHook.java:931)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.executePostGetTableHook(HiveMetaStoreClient.java:2638)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:2624)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:267)
at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:216)
at com.sun.proxy.$Proxy56.getTable(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
at
[jira] [Created] (HIVE-27159) Filters are not pushed down for decimal format in Parquet
Rajesh Balamohan created HIVE-27159:
---
Summary: Filters are not pushed down for decimal format in Parquet
Key: HIVE-27159
URL: https://issues.apache.org/jira/browse/HIVE-27159
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Decimal filters are not created and pushed down in parquet readers. This causes
latency delays and unwanted row processing in query execution.
It throws exception in runtime and processes more rows.
E.g Q13.
{noformat}
Parquet: (Map 1)
INFO : Task Execution Summary
INFO :
--
INFO : VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms)
INPUT_RECORDS OUTPUT_RECORDS
INFO :
--
INFO : Map 1 31254.00 0 0
549,181,950 133
INFO : Map 3 0.00 0 0
73,049 365
INFO : Map 4 2027.00 0 0
6,000,0001,689,919
INFO : Map 5 0.00 0 0
7,2001,440
INFO : Map 6517.00 0 0
1,920,800 493,920
INFO : Map 7 0.00 0 0
1,0021,002
INFO : Reducer 2 18716.00 0 0
1330
INFO :
--
ORC:
INFO : Task Execution Summary
INFO :
--
INFO : VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms)
INPUT_RECORDS OUTPUT_RECORDS
INFO :
--
INFO : Map 1 6556.00 0 0
267,146,063 152
INFO : Map 3 0.00 0 0
10,000 365
INFO : Map 4 2014.00 0 0
6,000,0001,689,919
INFO : Map 5 0.00 0 0
7,2001,440
INFO : Map 6504.00 0 0
1,920,800 493,920
INFO : Reducer 2 3159.00 0 0
1520
INFO :
--
{noformat}
{noformat}
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: (ss_hdemo_sk is not null and ss_addr_sk is not
null and ss_cdemo_sk is not null and ss_store_sk is not null and
((ss_sales_price >= 100) or (ss_sales_price <= 150) or (ss_sales_price >= 50)
or (ss_sales_price <= 100) or (ss_sales_price >= 150) or (ss_sales_price <=
200)) and ((ss_net_profit >= 100) or (ss_net_profit <= 200) or (ss_net_profit
>= 150) or (ss_net_profit <= 300) or (ss_net_profit >= 50) or (ss_net_profit <=
250))) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_112_container,
bigKeyColName:ss_hdemo_sk, smallTablePos:1, keyRatio:5.042575832290721E-6
Statistics: Num rows: 2750380056 Data size: 1321831086472
Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (ss_hdemo_sk is not null and ss_addr_sk is not
null and ss_cdemo_sk is not null and ss_store_sk is not null and
((ss_sales_price >= 100) or (ss_sales_price <= 150) or (ss_sales_price >= 50)
or (ss_sales_price <= 100) or (ss_sales_price >= 150) or (ss_sales_price <=
200)) and ((ss_net_profit >= 100) or (ss_net_profit <= 200) or (ss_net_profit
>= 150) or (ss_net_profit <= 300) or (ss_net_profit >= 50) or (ss_net_profit <=
250))) (type: boolean)
Statistics: Num rows: 2500252205 Data size: 1201619783884
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: ss_cdemo_sk (type: bigint), ss_hdemo_sk
(type: bigint), ss_addr_sk (type: bigint), ss_store_sk (type: bigint),
ss_quantity (type: int), ss_ext_sales_price (type: decimal(7,2)),
ss_ext_wholesale_cost (type: decimal(7,2)), ss_sold_date_sk (type: bigint),
ss_net_profit BETWEEN 100 AND 200 (type: boolean), ss_net_profit BETWEEN 150
AND 300 (type: boolean), ss_net_profit BETWEEN 50 AND 250 (type: boolean),
ss_sales_price BETWEEN 100 AND 150 (type: boolean), ss_sales_price BETWEEN 50
AND 100 (type: boolean), ss_sales_price BETWEEN 150 AND 200 (type: boolean)
[jira] [Created] (HIVE-27144) Alter table partitions need not DBNotificationListener for external tables
Rajesh Balamohan created HIVE-27144:
---
Summary: Alter table partitions need not DBNotificationListener
for external tables
Key: HIVE-27144
URL: https://issues.apache.org/jira/browse/HIVE-27144
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
DBNotificationListener for external tables may not be needed.
Even for "analyze table blah compute statistics for columns" for external
partitioned tables, it invokes DBNotificationListener for all partitions.
{noformat}
at org.datanucleus.store.query.Query.execute(Query.java:1726)
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:374)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
at
org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:11774)
at jdk.internal.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy33.addNotificationEvent(Unknown Source)
at
org.apache.hive.hcatalog.listener.DbNotificationListener.process(DbNotificationListener.java:1308)
at
org.apache.hive.hcatalog.listener.DbNotificationListener.onAlterPartition(DbNotificationListener.java:458)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$14.notify(MetaStoreListenerNotifier.java:161)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:328)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:390)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:863)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:6253)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:6201)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.18/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.18/NativeMethodAccessorImpl.java:62)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
at com.sun.proxy.$Proxy34.alter_partitions_req(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:21532)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:21511)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(java.base@11.0.18/Native
Method)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27119) Iceberg: Delete from table generates lot of files
Rajesh Balamohan created HIVE-27119:
---
Summary: Iceberg: Delete from table generates lot of files
Key: HIVE-27119
URL: https://issues.apache.org/jira/browse/HIVE-27119
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
With "delete" it generates lot of files due to the way data is sent to the
reducers. Files per partition is impacted by the number of reduce tasks.
One way could be to explicitly control the number of reducers; Creating this
ticket to have a long term fix.
{noformat}
explain delete from store_Sales where ss_customer_sk % 10 = 0;
INFO : Compiling
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b):
explain delete from store_Sales where ss_customer_sk % 10 = 0
INFO : No Stats for tpcds_1000_iceberg_mor_v4@store_sales, Columns:
ss_sold_time_sk, ss_cdemo_sk, ss_promo_sk, ss_ext_discount_amt,
ss_ext_sales_price, ss_net_profit, ss_addr_sk, ss_ticket_number,
ss_wholesale_cost, ss_item_sk, ss_ext_list_price, ss_sold_date_sk, ss_store_sk,
ss_coupon_amt, ss_quantity, ss_list_price, ss_sales_price, ss_customer_sk,
ss_ext_wholesale_cost, ss_net_paid, ss_ext_tax, ss_hdemo_sk, ss_net_paid_inc_tax
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:Explain,
type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b); Time
taken: 0.704 seconds
INFO : Executing
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b):
explain delete from store_Sales where ss_customer_sk % 10 = 0
INFO : Starting task [Stage-4:EXPLAIN] in serial mode
INFO : Completed executing
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b); Time
taken: 0.005 seconds
INFO : OK
Explain
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
Stage-0 depends on stages: Stage-2
Stage-3 depends on stages: Stage-0
STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b:377
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
DagName: hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b:377
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: ((ss_customer_sk % 10) = 0) (type: boolean)
Statistics: Num rows: 2755519629 Data size: 3643899155232
Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: ((ss_customer_sk % 10) = 0) (type: boolean)
Statistics: Num rows: 1377759814 Data size: 1821949576954
Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: PARTITION__SPEC__ID (type: int),
PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type:
bigint), ss_sold_time_sk (type: int), ss_item_sk (type: int), ss_customer_sk
(type: int), ss_cdemo_sk (type: int), ss_hdemo_sk (type: int), ss_addr_sk
(type: int), ss_store_sk (type: int), ss_promo_sk (type: int), ss_ticket_number
(type: bigint), ss_quantity (type: int), ss_wholesale_cost (type:
decimal(7,2)), ss_list_price (type: decimal(7,2)), ss_sales_price (type:
decimal(7,2)), ss_ext_discount_amt (type: decimal(7,2)), ss_ext_sales_price
(type: decimal(7,2)), ss_ext_wholesale_cost (type: decimal(7,2)),
ss_ext_list_price (type: decimal(7,2)), ss_ext_tax (type: decimal(7,2)),
ss_coupon_amt (type: decimal(7,2)), ss_net_paid (type: decimal(7,2)),
ss_net_paid_inc_tax (type: decimal(7,2)), ss_net_profit (type: decimal(7,2)),
ss_sold_date_sk (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14,
_col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24,
_col25, _col26
Statistics: Num rows: 1377759814 Data size: 1821949576954
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type:
bigint), _col2 (type: string), _col3 (type: bigint)
null sort order:
sort order:
Statistics: Num rows: 1377759814 Data size:
1821949576954 Basic stats: COMPLETE Column stats: NONE
value expressions: _col4 (type: int), _col5 (type:
int), _col6 (type: int), _col7 (type: int), _col8 (type: int), _col9 (type:
int), _col10 (type: int), _col11 (type: int), _col12 (type: bigint), _col13
(type: int), _col14 (type: decimal(7,2)), _col15 (type: decimal(7,2)), _col16
(type: decimal(7,2)), _col17
[jira] [Created] (HIVE-27099) Iceberg: select count(*) from table queries all data
Rajesh Balamohan created HIVE-27099:
---
Summary: Iceberg: select count(*) from table queries all data
Key: HIVE-27099
URL: https://issues.apache.org/jira/browse/HIVE-27099
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
select count is scanning all data. Though it has complete basic stats, it
launched tez job which wasn't needed. Second issue is, it ended up scanning
ENTIRE 148 GB dataset which is completely not required. It should have got the
data from parq files itself. Ideal situation is getting entire records from
manifest itself.
Data is stored in parquet format in external tables. This may be broken for
parquet, as for ORC it is able to read less data (footer info).
1. Consider fixing count( * ) for parq
2. Check if it is possible to read stats from iceberg manifests after #1.
{noformat}
explain select count(*) from store_sales;
Explain
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20230223031934_2abeb3b9-8c18-4ff7-a8f9-df7368010189:5
Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
DagName: hive_20230223031934_2abeb3b9-8c18-4ff7-a8f9-df7368010189:5
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
Statistics: Num rows: 2879966589 Data size: 195666988943
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
Statistics: Num rows: 2879966589 Data size: 195666988943
Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
minReductionHashAggr: 0.5
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
null sort order:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized
Reducer 2
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
53 rows selected (1.454 seconds)
0: jdbc:hive2://ve0:218> select count(*) from store_sales;
INFO : Query ID = hive_20230223031940_9ff5d61d-1fe2-4476-a561-7820e4a3a5f8
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20230223031940_9ff5d61d-1fe2-4476-a561-7820e4a3a5f8
INFO : Session is already open
INFO : Dag name: select count(*) from store_sales (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1676286357243_0061)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. container SUCCEEDED76776700
0 0
Reducer 2 .. container SUCCEEDED 1 100
0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 54.94 s
--
INFO : Status: DAG finished successfully in 54.85 seconds
INFO :
INFO : Query Execution Summary
INFO :
--
INFO : OPERATIONDURATION
INFO :
--
INFO : Compile Query
[jira] [Created] (HIVE-27084) Iceberg: Stats are not populated correctly during query compilation
Rajesh Balamohan created HIVE-27084:
---
Summary: Iceberg: Stats are not populated correctly during query
compilation
Key: HIVE-27084
URL: https://issues.apache.org/jira/browse/HIVE-27084
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
- Table stats are not properly used/computed during query compilation phase.
- Here is an example. Check out the query with the filter which give more data
than the regular query
This is just an example, real world queries can have bad query plans due to this
{{10470974584 with filter, vs 303658262936 without filter}}
{noformat}
explain select count(*) from store_sales where ss_sold_date_sk=2450822 and
ss_wholesale_cost > 0.0
Explain
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20230216065808_80d68e3f-3a6b-422b-9265-50bc707ae3c6:48
Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
DagName: hive_20230216065808_80d68e3f-3a6b-422b-9265-50bc707ae3c6:48
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: ((ss_sold_date_sk = 2450822) and
(ss_wholesale_cost > 0)) (type: boolean)
Statistics: Num rows: 2755519629 Data size: 303658262936
Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: ((ss_sold_date_sk = 2450822) and
(ss_wholesale_cost > 0)) (type: boolean)
Statistics: Num rows: 5 Data size: 550 Basic stats:
COMPLETE Column stats: NONE
Select Operator
Statistics: Num rows: 5 Data size: 550 Basic stats:
COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
minReductionHashAggr: 0.99
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 124 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
null sort order:
sort order:
Statistics: Num rows: 1 Data size: 124 Basic stats:
COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs (cache only)
Reducer 2
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 124 Basic stats: COMPLETE
Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 124 Basic stats: COMPLETE
Column stats: NONE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
58 rows selected (0.73 seconds)
explain select count(*) from store_sales where ss_sold_date_sk=2450822
INFO : Starting task [Stage-3:EXPLAIN] in serial mode
INFO : Completed executing
command(queryId=hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7); Time
taken: 0.061 seconds
INFO : OK
Explain
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7:49
Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
DagName: hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7:49
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: (ss_sold_date_sk = 2450822) (type: boolean)
Statistics: Num rows: 2755519629 Data size: 10470974584 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (ss_sold_date_sk = 2450822) (type: boolean)
Statistics: Num rows: 5 Data size: 18 Basic stats: COMPLETE
Column stats: NONE
Select Operator
Statistics: Num rows: 5 Data size: 18 Basic stats:
COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
minReductionHashAggr: 0.99
[jira] [Created] (HIVE-27050) Iceberg: MOR: Restrict reducer extrapolation to contain number of small files being created
Rajesh Balamohan created HIVE-27050: --- Summary: Iceberg: MOR: Restrict reducer extrapolation to contain number of small files being created Key: HIVE-27050 URL: https://issues.apache.org/jira/browse/HIVE-27050 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan Scenario: # Create a simple table in iceberg (MOR mode). e.g store_sales_delete_1 # Insert some data into it. # Run an update statement as follows ## "update store_sales_delete_1 set ss_sold_time_sk=699060 where ss_sold_time_sk=69906" Hive estimates the number of reducers as "1". But due to "hive.tez.max.partition.factor" which defaults to "2.0", it will double the number of reducers. To put in perspective, it will create very small positional delete files spreading across different reducers. This will cause problems during reading, as all files should be opened for reading. # When iceberg MOR tables are involved in update/delete/merges, disable "hive.tez.max.partition.factor"; or set it to "1.0" irrespective of the user setting; # Have explicit logs for easier debugging; User shouldn't be confused on why the setting is not taking into effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27049) Iceberg: Provide current snapshot version in show-create-table
Rajesh Balamohan created HIVE-27049: --- Summary: Iceberg: Provide current snapshot version in show-create-table Key: HIVE-27049 URL: https://issues.apache.org/jira/browse/HIVE-27049 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan It will be helpful to show "current snapshot" id in "show create table" statement. This will help in easier debugging. Otherwise, user has to explicitly query the metadata or read the JSON file to get this info. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27014) Iceberg: getSplits/planTasks should filter out relevant folders instead of scanning entire table
Rajesh Balamohan created HIVE-27014: --- Summary: Iceberg: getSplits/planTasks should filter out relevant folders instead of scanning entire table Key: HIVE-27014 URL: https://issues.apache.org/jira/browse/HIVE-27014 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan With dynamic partition pruning, only relevant folders in fact tables are scanned. In tez, DynamicPartitionPruner will set the relevant filters.In iceberg, these filters are applied after "Table:planTasks()" is invoked in iceberg. This forces entire table metadata to be scanned and then throw off the unwanted partitions. This makes split computation expensive (e.g for store_sales, it has to look at all 1800+ partitions and throw off unwanted partitions). For short running queries, it takes 3-5+ seconds for split computation. Creating this ticket as a placeholder to make use of the relevant filters from DPP. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27013) Provide an option to enable iceberg manifest caching via table properties
Rajesh Balamohan created HIVE-27013:
---
Summary: Provide an option to enable iceberg manifest caching via
table properties
Key: HIVE-27013
URL: https://issues.apache.org/jira/browse/HIVE-27013
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
{color:#22}I tried the following thinking that it would work with iceberg
manifest caching; but it didn't.{color}
{color:#22}{noformat}{color}
{color:#22}alter table store_sales set
tblproperties('io.manifest.cac{color}{color:#22}he-enabled'='true');
\{noformat}{color}
{color:#22}Creating this ticket as a placeholder to fix the same.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27010) Reduce compilation time
Rajesh Balamohan created HIVE-27010: --- Summary: Reduce compilation time Key: HIVE-27010 URL: https://issues.apache.org/jira/browse/HIVE-27010 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Context: Post HIVE-24645, compilation time for queries has increased. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27005) Iceberg: Col stats are not used in queries
Rajesh Balamohan created HIVE-27005:
---
Summary: Iceberg: Col stats are not used in queries
Key: HIVE-27005
URL: https://issues.apache.org/jira/browse/HIVE-27005
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
Attachments: col_stats.txt
1. Though, insert-queries compute colstats during runtime, they are not
persisted in HMS during final call.
2. Due to #1, col stats are not available during runtime for hive queries. This
includes col stats, NDV etc. So unless users explicitly run "analyse table"
statements, queries can be have suboptimal plans.
E.g
[col_stats.txt{^}!https://jira.cloudera.com/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.cloudera.com/secure/attachment/658390/658390_col_stats.txt](note
that there is no col stats being used)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27003) Iceberg: Vectorization missed out for update/delete due to virtual columns
Rajesh Balamohan created HIVE-27003: --- Summary: Iceberg: Vectorization missed out for update/delete due to virtual columns Key: HIVE-27003 URL: https://issues.apache.org/jira/browse/HIVE-27003 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan Attachments: delete_iceberg_vect.txt, update_iceberg_vect.txt Vectorization is missed out during table scan due to the addition of virtual columns during scans. I will attach the plan details here with. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26997) Iceberg: Vectorization gets disabled at runtime in merge-into statements
Rajesh Balamohan created HIVE-26997:
---
Summary: Iceberg: Vectorization gets disabled at runtime in
merge-into statements
Key: HIVE-26997
URL: https://issues.apache.org/jira/browse/HIVE-26997
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
Attachments: explain_merge_into.txt
*Query:*
Think of "ssv" table as a table containing trickle feed data in the following
query. "store_sales_delete_1" is the destination table.
{noformat}
MERGE INTO tpcds_1000_iceberg_mor_v4.store_sales_delete_1 t USING
tpcds_1000_update.ssv s ON (t.ss_item_sk = s.ss_item_sk
AND t.ss_customer_sk=s.ss_customer_sk
AND t.ss_sold_date_sk = "2451181"
AND ((Floor((s.ss_item_sk) / 1000) * 1000) BETWEEN 1000 AND 2000)
AND s.ss_ext_discount_amt < 0.0) WHEN matched
AND t.ss_ext_discount_amt IS NULL THEN
UPDATE
SET ss_ext_discount_amt = 0.0 WHEN NOT matched THEN
INSERT (ss_sold_time_sk,
ss_item_sk,
ss_customer_sk,
ss_cdemo_sk,
ss_hdemo_sk,
ss_addr_sk,
ss_store_sk,
ss_promo_sk,
ss_ticket_number,
ss_quantity,
ss_wholesale_cost,
ss_list_price,
ss_sales_price,
ss_ext_discount_amt,
ss_ext_sales_price,
ss_ext_wholesale_cost,
ss_ext_list_price,
ss_ext_tax,
ss_coupon_amt,
ss_net_paid,
ss_net_paid_inc_tax,
ss_net_profit,
ss_sold_date_sk)
VALUES (s.ss_sold_time_sk,
s.ss_item_sk,
s.ss_customer_sk,
s.ss_cdemo_sk,
s.ss_hdemo_sk,
s.ss_addr_sk,
s.ss_store_sk,
s.ss_promo_sk,
s.ss_ticket_number,
s.ss_quantity,
s.ss_wholesale_cost,
s.ss_list_price,
s.ss_sales_price,
s.ss_ext_discount_amt,
s.ss_ext_sales_price,
s.ss_ext_wholesale_cost,
s.ss_ext_list_price,
s.ss_ext_tax,
s.ss_coupon_amt,
s.ss_net_paid,
s.ss_net_paid_inc_tax,
s.ss_net_profit,
"2451181")
{noformat}
*Issue:*
# Map phase is not getting vectorized due to "PARTITION_{_}SPEC{_}_ID" column
{noformat}
Map notVectorizedReason: Select expression for SELECT operator: Virtual column
PARTITION__SPEC__ID is not supported {noformat}
2. "Reducer 2" stage isn't vectorized.
{noformat}
Reduce notVectorizedReason: exception: java.lang.RuntimeException: Full Outer
Small Table Key Mapping duplicate column 0 in ordered column map {0=(value
column: 30, type info: int), 1=(value column: 31, type info: int)} when adding
value column 53, type into int stack trace:
org.apache.hadoop.hive.ql.exec.vector.VectorColumnOrderedMap.add(VectorColumnOrderedMap.java:102),
org.apache.hadoop.hive.ql.exec.vector.VectorColumnSourceMapping.add(VectorColumnSourceMapping.java:41),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.canSpecializeMapJoin(Vectorizer.java:3865),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5246),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:988),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:874),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:841),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:251),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2298),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2246),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceWork(Vectorizer.java:2224),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertReduceWork(Vectorizer.java:2206),
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1038),
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111),
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180),
... {noformat}
I have attached the explain plan for this, which has details on this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26978) Stale "Runtime stats" causes poor query planning
Rajesh Balamohan created HIVE-26978: --- Summary: Stale "Runtime stats" causes poor query planning Key: HIVE-26978 URL: https://issues.apache.org/jira/browse/HIVE-26978 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan Attachments: Screenshot 2023-01-24 at 10.23.16 AM.png * Runtime stats can be stored in hiveserver or in metastore via "hive.query.reexecution.stats.persist.scope". * Though the table is dropped and recreated, it ends up showing old stats via "RUNTIME" stats. Here is an example (note that the table is empty, but gets datasize and numRows from RUNTIME stats) * This causes suboptimal plan for "MERGE INTO" queries by creating CUSTOM_EDGE instead of broadcast edge. !Screenshot 2023-01-24 at 10.23.16 AM.png|width=2053,height=753! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26975) MERGE: Wrong reducer estimate causing smaller files to be created
Rajesh Balamohan created HIVE-26975:
---
Summary: MERGE: Wrong reducer estimate causing smaller files to be
created
Key: HIVE-26975
URL: https://issues.apache.org/jira/browse/HIVE-26975
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
* "Merge into" estimates wrong number of reducers causing more number of small
files to be created.* e.g 400+ files in 3+ MB file each.*
* This can be reproduced by writing data into "store_sales" table in iceberg
format via another source table (using merge-into).
** e.g Running this few times will create wrong number of reduce tasks
causing lot of small files to be created in iceberg table.
{noformat}
MERGE INTO store_sales_t t
using ssv s
ON ( t.ss_item_sk = s.ss_item_sk
AND t.ss_customer_sk = s.ss_customer_sk
AND t.ss_sold_date_sk = "2451181"
AND ( ( Floor(( s.ss_item_sk ) / 1000) * 1000 ) BETWEEN 1000 AND 2000 )
AND s.ss_ext_discount_amt < 0.0 )
WHEN matched AND t.ss_ext_discount_amt IS NULL THEN
UPDATE SET ss_ext_discount_amt = 0.0
WHEN NOT matched THEN
INSERT ( ss_sold_time_sk,
ss_item_sk,
ss_customer_sk,
ss_cdemo_sk,
ss_hdemo_sk,
ss_addr_sk,
ss_store_sk,
ss_promo_sk,
ss_ticket_number,
ss_quantity,
ss_wholesale_cost,
ss_list_price,
ss_sales_price,
ss_ext_discount_amt,
ss_ext_sales_price,
ss_ext_wholesale_cost,
ss_ext_list_price,
ss_ext_tax,
ss_coupon_amt,
ss_net_paid,
ss_net_paid_inc_tax,
ss_net_profit,
ss_sold_date_sk )
VALUES ( s.ss_sold_time_sk,
s.ss_item_sk,
s.ss_customer_sk,
s.ss_cdemo_sk,
s.ss_hdemo_sk,
s.ss_addr_sk,
s.ss_store_sk,
s.ss_promo_sk,
s.ss_ticket_number,
s.ss_quantity,
s.ss_wholesale_cost,
s.ss_list_price,
s.ss_sales_price,
s.ss_ext_discount_amt,
s.ss_ext_sales_price,
s.ss_ext_wholesale_cost,
s.ss_ext_list_price,
s.ss_ext_tax,
s.ss_coupon_amt,
s.ss_net_paid,
s.ss_net_paid_inc_tax,
s.ss_net_profit,
"2451181")
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26974) CTL from iceberg table should copy partition fields correctly
Rajesh Balamohan created HIVE-26974: --- Summary: CTL from iceberg table should copy partition fields correctly Key: HIVE-26974 URL: https://issues.apache.org/jira/browse/HIVE-26974 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan # Create iceberg table. Ensure it to have a partition field. # run "create external table like x" # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26951) Setting details in PositionDeleteInfo takes up lot of CPU cycles
Rajesh Balamohan created HIVE-26951: --- Summary: Setting details in PositionDeleteInfo takes up lot of CPU cycles Key: HIVE-26951 URL: https://issues.apache.org/jira/browse/HIVE-26951 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan Attachments: Screenshot 2023-01-17 at 11.29.29 AM.png, Screenshot 2023-01-17 at 11.29.36 AM.png !Screenshot 2023-01-17 at 11.29.29 AM.png|width=898,height=532! !Screenshot 2023-01-17 at 11.29.36 AM.png|width=1000,height=591! This was observed with merge-into statements. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26950) (CTLT) Create external table like V2 table is not preserving table properties
Rajesh Balamohan created HIVE-26950: --- Summary: (CTLT) Create external table like V2 table is not preserving table properties Key: HIVE-26950 URL: https://issues.apache.org/jira/browse/HIVE-26950 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan # Create an external iceberg V2 table. e.g t1 # "create external table t2 like t1" <--- This ends up creating V1 table and "format-version=2" is not retained and "'format'='iceberg/parquet'" is also not retained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26944) FileSinkOperator shouldn't check for compactiontable for every row being processed
Rajesh Balamohan created HIVE-26944: --- Summary: FileSinkOperator shouldn't check for compactiontable for every row being processed Key: HIVE-26944 URL: https://issues.apache.org/jira/browse/HIVE-26944 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2023-01-16 at 10.32.24 AM.png -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26928) LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata cache is disabled
Rajesh Balamohan created HIVE-26928:
---
Summary: LlapIoImpl::getParquetFooterBuffersFromCache throws
exception when metadata cache is disabled
Key: HIVE-26928
URL: https://issues.apache.org/jira/browse/HIVE-26928
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
When metadata / LLAP cache is disabled, "iceberg + parquet" throws the
following error.
It should check for "metadatacache" correctly or fix it in LlapIoImpl.
{noformat}
Caused by: java.lang.NullPointerException: Metadata cache must not be null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.getParquetFooterBuffersFromCache(LlapIoImpl.java:467)
at
org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.parquetRecordReader(HiveVectorizedReader.java:227)
at
org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:162)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:65)
at
org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:77)
at
org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:196)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.openVectorized(IcebergInputFormat.java:331)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.open(IcebergInputFormat.java:377)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextTask(IcebergInputFormat.java:270)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.initialize(IcebergInputFormat.java:266)
at
org.apache.iceberg.mr.mapred.AbstractMapredIcebergRecordReader.(AbstractMapredIcebergRecordReader.java:40)
at
org.apache.iceberg.mr.hive.vector.HiveIcebergVectorizedRecordReader.(HiveIcebergVectorizedRecordReader.java:41)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26927) Iceberg: Add support for set_current_snapshotid
Rajesh Balamohan created HIVE-26927:
---
Summary: Iceberg: Add support for set_current_snapshotid
Key: HIVE-26927
URL: https://issues.apache.org/jira/browse/HIVE-26927
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Rajesh Balamohan
Currently, hive supports "rollback" feature. Once rolledback, it is not
possible to move from older snapshot to newer snapshot.
It ends up throwing
{color:#0747a6}"org.apache.iceberg.exceptions.ValidationException: Cannot roll
back to snapshot, not an ancestor of the current state:" {color}error.
It will be good to support "set_current_snapshot" function to move to different
snapshot ids.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26917) Upgrade parquet to 1.12.3
Rajesh Balamohan created HIVE-26917: --- Summary: Upgrade parquet to 1.12.3 Key: HIVE-26917 URL: https://issues.apache.org/jira/browse/HIVE-26917 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26913) HiveVectorizedReader::parquetRecordReader should reuse footer information
Rajesh Balamohan created HIVE-26913: --- Summary: HiveVectorizedReader::parquetRecordReader should reuse footer information Key: HIVE-26913 URL: https://issues.apache.org/jira/browse/HIVE-26913 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan HiveVectorizedReader::parquetRecordReader should reuse details of parquet footer, instead of reading it again. It reads parquet footer here: [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java#L230-L232] Again it reads the footer here for constructing vectorized recordreader [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java#L249] [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java#L50] Check the codepath of VectorizedParquetRecordReader::setupMetadataAndParquetSplit [https://github.com/apache/hive/blob/6b0139188aba6a95808c8d1bec63a651ec9e4bdc/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L180] It should be possible to share "ParquetMetadata" in VectorizedParuqetRecordReader. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26874) Iceberg: Positional delete files are not cached
Rajesh Balamohan created HIVE-26874: --- Summary: Iceberg: Positional delete files are not cached Key: HIVE-26874 URL: https://issues.apache.org/jira/browse/HIVE-26874 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan With iceberg v2 (MOR mode), "positional delete" files are not cached causing runtime delays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26714) Iceberg delete files are read twice during query processing causing delays
Rajesh Balamohan created HIVE-26714: --- Summary: Iceberg delete files are read twice during query processing causing delays Key: HIVE-26714 URL: https://issues.apache.org/jira/browse/HIVE-26714 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Rajesh Balamohan Attachments: Screenshot 2022-11-08 at 9.37.17 PM.png Delete positions are read twice in query processing causing delays in runtime. !Screenshot 2022-11-08 at 9.37.17 PM.png|width=707,height=629! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX
Rajesh Balamohan created HIVE-26699: --- Summary: Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX Key: HIVE-26699 URL: https://issues.apache.org/jira/browse/HIVE-26699 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Hive reads JSON metadata information (TableMetadataParser::read()) multiple times; E.g during query compilation, AM split computation, stats computation, during commits etc. With large JSON files (due to multiple inserts), it takes a lot longer time with S3 FS with "fs.s3a.experimental.input.fadvise" set to "random". (e.g in the order of 10x).To be on safer side, it will be good to set this to "normal" mode in configs, when reading iceberg tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26686) Iceberg: Having lot of snapshots impacts runtime due to multiple loads of the table
Rajesh Balamohan created HIVE-26686:
---
Summary: Iceberg: Having lot of snapshots impacts runtime due to
multiple loads of the table
Key: HIVE-26686
URL: https://issues.apache.org/jira/browse/HIVE-26686
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When large number of snpashots are present in manifest file, it adversely
impacts the runtime of the queries. (e.g 15 mts trickle feed).
Having more snapshots will slow down runtime in 2 additional places.
1. At the time of populating statistics, it tries to load the table details
again. i.e refresh table invocation
2. At the time of hive metastore hook (HiveIcebergMetaHook::doPreAlterTable),
during pre alter table.
Need to check if entire table information along with snapshot details are
needed for this.
{noformat}
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:437)
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:261)
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
at
org.apache.hive.iceberg.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3251)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:264)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:177)
at
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$685/0x000840e1b440.apply(Unknown
Source)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$686/0x000840e1a840.run(Unknown
Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:153)
at
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)
at
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)
at
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:116)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:106)
at
org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getBasicStatistics(HiveIcebergStorageHandler.java:309)
at
org.apache.hadoop.hive.ql.stats.BasicStatsTask$BasicStatsProcessor.(BasicStatsTask.java:138)
at
org.apache.hadoop.hive.ql.stats.BasicStatsTask.aggregateStats(BasicStatsTask.java:301)
at
org.apache.hadoop.hive.ql.stats.BasicStatsTask.process(BasicStatsTask.java:108)
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:360)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:333)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:250)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:111)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:806)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:540)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:534)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:232)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:89)
at
[jira] [Created] (HIVE-26540) Iceberg: Select queries after update/delete become expensive in reading contents
Rajesh Balamohan created HIVE-26540:
---
Summary: Iceberg: Select queries after update/delete become
expensive in reading contents
Key: HIVE-26540
URL: https://issues.apache.org/jira/browse/HIVE-26540
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
- Create basic date_dim table in tpcds. Store it in iceberg v2 format
- Update few 1000 records couple of times
- Run a simple select query {{select count ( * ) from date_dim_ice where d_qoy
= 11 and d_dom=2 and d_fy_week_seq=3;}}
This takes 8-18 seconds where ACID takes 1.5 seconds.
Basic issue is that, it reads files multiple times (i.e both data and delete
files).
Lines of interest:
IcebergInputFormat.java
{noformat}
InternalRecordWrapper wrapper = new
InternalRecordWrapper(readSchema.asStruct());
Evaluator filter = new Evaluator(readSchema.asStruct(), residual,
caseSensitive);
return CloseableIterable.filter(iter, record ->
filter.eval(wrapper.wrap((StructLike) record)));
{noformat}
{noformat}
case GENERIC:
DeleteFilter deletes = new GenericDeleteFilter(table.io(),
currentTask, table.schema(), readSchema);
Schema requiredSchema = deletes.requiredSchema();
return deletes.filter(openGeneric(currentTask, requiredSchema));
{noformat}
These get evaluated for each row in the data file, causing delay.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26532) Remove logger from critical path in VectorMapJoinInnerLongOperator::processBatch
Rajesh Balamohan created HIVE-26532: --- Summary: Remove logger from critical path in VectorMapJoinInnerLongOperator::processBatch Key: HIVE-26532 URL: https://issues.apache.org/jira/browse/HIVE-26532 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2022-09-12 at 10.03.43 AM.png !Screenshot 2022-09-12 at 10.03.43 AM.png|width=895,height=872! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
Rajesh Balamohan created HIVE-26529: --- Summary: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat Key: HIVE-26529 URL: https://issues.apache.org/jira/browse/HIVE-26529 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been disabled in HiveIcebergInputFormat. This causes regressions in queries. [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] It will be good to restore DECIMAL_64 support in iceberg input format. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26520) Improve dynamic partition pruning operator when subqueries are involved
Rajesh Balamohan created HIVE-26520:
---
Summary: Improve dynamic partition pruning operator when
subqueries are involved
Key: HIVE-26520
URL: https://issues.apache.org/jira/browse/HIVE-26520
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
Attachments: q58_test.pdf
Dynamic partition pruning operator sends entire date_dim table and due to this,
entire catalog_sales data is scanned causing huge IO and decoding cost.
If dynamic partition pruning operator was created after the "date_dim" subquery
has been evaluated, it would have saved huge IO cost. E.g It would have just
taken 6-7 partition scans instead of 1800+ partitions.
Consider the following simplified query as example
{noformat}
select count(*) from (select i_item_id item_id
,sum(cs_ext_sales_price) cs_item_rev
from catalog_sales
,item
,date_dim
where cs_item_sk = i_item_sk
and d_date in (select d_date
from date_dim
where d_week_seq = (select d_week_seq
from date_dim
where d_date = '1998-02-21'))
and cs_sold_date_sk = d_date_sk
group by i_item_id) a;
CBO PLAN:
HiveAggregate(group=[{}], agg#0=[count()])
HiveProject(i_item_id=[$0])
HiveAggregate(group=[{4}])
HiveSemiJoin(condition=[=($6, $7)], joinType=[semi])
HiveJoin(condition=[=($2, $5)], joinType=[inner])
HiveJoin(condition=[=($0, $3)], joinType=[inner])
HiveProject(cs_item_sk=[$14], cs_ext_sales_price=[$22],
cs_sold_date_sk=[$33])
HiveFilter(condition=[AND(IS NOT NULL($33), IS NOT NULL($14))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external,
catalog_sales]], table:alias=[catalog_sales])
HiveProject(i_item_sk=[$0], i_item_id=[$1])
HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external,
item]], table:alias=[item])
HiveProject(d_date_sk=[$0], d_date=[$2])
HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($0))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external,
date_dim]], table:alias=[date_dim])
HiveProject(d_date=[$0])
HiveJoin(condition=[=($1, $3)], joinType=[inner])
HiveJoin(condition=[true], joinType=[inner])
HiveProject(d_date=[$2], d_week_seq=[$4])
HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($4))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, date_dim]],
table:alias=[date_dim])
HiveProject(cnt=[$0])
HiveFilter(condition=[<=(sq_count_check($0), 1)])
HiveProject(cnt=[$0])
HiveAggregate(group=[{}], cnt=[COUNT()])
HiveFilter(condition=[=($2, 1998-02-21)])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, date_dim]],
table:alias=[date_dim])
HiveProject(d_week_seq=[$4])
HiveFilter(condition=[AND(=($2, 1998-02-21), IS NOT NULL($4))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external,
date_dim]], table:alias=[date_dim])
{noformat}
I will attach the formatted plan for reference as well. If planner generated
the dynamic partition pruning event after "date_dim" got evaluated in "Map 7",
it would be been very efficient.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26507) Iceberg: In place metadata generation may not work for certain datatypes
Rajesh Balamohan created HIVE-26507:
---
Summary: Iceberg: In place metadata generation may not work for
certain datatypes
Key: HIVE-26507
URL: https://issues.apache.org/jira/browse/HIVE-26507
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
"alter table" statements can be used for generating iceberg metadata
information (i.e for converting external tables -> iceberg tables).
As part of this process, it also converts certain datatypes to iceberg
compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion"
enables this conversion.
This could cause certain issues at runtime. Here is an example
{noformat}
Before conversion:
==
-- external table
select count(*) from customer_demographics where cd_gender = 'F' and
cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
27440
after conversion:
=
-- iceberg table
select count(*) from customer_demographics where cd_gender = 'F' and
cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
0
select count(*) from customer_demographics where cd_gender = 'F' and
cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';
27440
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26496) FetchOperator scans delete_delta folders multiple times causing slowness
Rajesh Balamohan created HIVE-26496:
---
Summary: FetchOperator scans delete_delta folders multiple times
causing slowness
Key: HIVE-26496
URL: https://issues.apache.org/jira/browse/HIVE-26496
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Rajesh Balamohan
FetchOperator scans way too many number of files/directories than needed.
For e.g here is a layout of a table which had set of updates and deletes. There
are set of "delta" and "delete_delta" folders which are created.
{noformat}
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/base_001
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_002_002_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_003_003_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_004_004_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_005_005_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_006_006_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_007_007_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_008_008_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_009_009_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_010_010_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_011_011_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_012_012_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_013_013_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_014_014_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_015_015_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_016_016_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_017_017_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_018_018_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_019_019_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_020_020_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_021_021_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_022_022_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_002_002_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_003_003_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_004_004_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_005_005_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_006_006_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_007_007_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_008_008_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_009_009_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_010_010_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_011_011_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_012_012_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_013_013_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_014_014_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_015_015_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_016_016_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_017_017_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_018_018_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_019_019_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_020_020_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_021_021_
{noformat}
When user runs *{color:#0747a6}{{select * from date_dim}}{color}* from beeline,
FetchOperator tries to compute splits in "date_dim". This "base" and "delta"
folders and computes
[jira] [Created] (HIVE-26491) Iceberg: Drop table should purge the data for V2 tables
Rajesh Balamohan created HIVE-26491: --- Summary: Iceberg: Drop table should purge the data for V2 tables Key: HIVE-26491 URL: https://issues.apache.org/jira/browse/HIVE-26491 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan # create external table stored by iceberg in orc format. Convert this to iceberg v2 table format via alter table statements. This should ideally have set "'external.table.purge'='true'" property by default which is missing for V2 tables. # insert data into it # Drop the table. This drops the metadata information, but retains actual data. Set "'external.table.purge'='true'" as default for iceberg (if it hasn't been set yet). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26490) Iceberg: Residual expression is constructed for the task from multiple places causing CPU burn
Rajesh Balamohan created HIVE-26490: --- Summary: Iceberg: Residual expression is constructed for the task from multiple places causing CPU burn Key: HIVE-26490 URL: https://issues.apache.org/jira/browse/HIVE-26490 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan Attachments: Screenshot 2022-08-22 at 12.58.47 PM.jpg "HiveIcebergInputFormat.residualForTask(task, job)" is invoked from multiple places causing CPU burn. !Screenshot 2022-08-22 at 12.58.47 PM.jpg|width=918,height=932! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26194) Unable to interrupt query in the middle of long compilation
Rajesh Balamohan created HIVE-26194:
---
Summary: Unable to interrupt query in the middle of long
compilation
Key: HIVE-26194
URL: https://issues.apache.org/jira/browse/HIVE-26194
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Rajesh Balamohan
*Issue:*
* Certain queries can take lot longer time to compile, depending on the number
of interactions with HMS.
* When user tries to cancel such queries in the middle of compilation, it
doesn't work. It interrupts the process only when the entire compilation phase
is complete.
* Example is given below (Q66 at 10 TB TPCDS)
{noformat}
. . . . . . . . . . . . . . . . . . . . . . .>,d_year
. . . . . . . . . . . . . . . . . . . . . . .> )
. . . . . . . . . . . . . . . . . . . . . . .> ) x
. . . . . . . . . . . . . . . . . . . . . . .> group by
. . . . . . . . . . . . . . . . . . . . . . .> w_warehouse_name
. . . . . . . . . . . . . . . . . . . . . . .>,w_warehouse_sq_ft
. . . . . . . . . . . . . . . . . . . . . . .>,w_city
. . . . . . . . . . . . . . . . . . . . . . .>,w_county
. . . . . . . . . . . . . . . . . . . . . . .>,w_state
. . . . . . . . . . . . . . . . . . . . . . .>,w_country
. . . . . . . . . . . . . . . . . . . . . . .>,ship_carriers
. . . . . . . . . . . . . . . . . . . . . . .>,year
. . . . . . . . . . . . . . . . . . . . . . .> order by w_warehouse_name
. . . . . . . . . . . . . . . . . . . . . . .> limit 100;
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
...
...
...
,w_city
,w_county
,w_state
,w_country
,ship_carriers
,year
order by w_warehouse_name
limit 100
INFO : Semantic Analysis Completed (retrial = false)
ERROR : FAILED: command has been interrupted: after analyzing query.
INFO : Compiling
command(queryId=hive_20220502040541_14c76b6f-f6d2-4ab3-ad82-522f17ede63a) has
been interrupted after 32.872 seconds <<< Notice that
it interrupted only after entire compilation is done at 32 seconds.
Error: Query was cancelled. Illegal Operation state transition from CANCELED to
ERROR (state=01000,code=0)
{noformat}
This becomes an issue in busy cluster.
Interrupt handling should be fixed in compilation phase.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HIVE-26185) Need support for metadataonly operations with iceberg (e.g select distinct on partition column)
Rajesh Balamohan created HIVE-26185:
---
Summary: Need support for metadataonly operations with iceberg
(e.g select distinct on partition column)
Key: HIVE-26185
URL: https://issues.apache.org/jira/browse/HIVE-26185
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Rajesh Balamohan
{noformat}
select distinct ss_sold_date_sk from store_sales
{noformat}
This query scans 1800+ rows in hive acid. But takes ages to process with
NullScanOptimiser during compilation phase
(https://issues.apache.org/jira/browse/HIVE-24262)
{noformat}
Hive ACID
INFO : Executing
command(queryId=hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14):
select distinct ss_sold_date_sk from store_sales
INFO : Compute 'ndembla-test2' is active.
INFO : Query ID = hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14
INFO : Tez session hasn't been created yet. Opening session
INFO : Dag name: select distinct ss_sold_date_s...store_sales (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1651102345385_)
INFO : Status: DAG finished successfully in 1.81 seconds
INFO : DAG ID: dag_1651102345385__5
INFO :
INFO : Query Execution Summary
INFO :
--
INFO : OPERATIONDURATION
INFO :
--
INFO : Compile Query 55.47s
INFO : Prepare Plan2.32s
INFO : Get Query Coordinator (AM) 0.13s
INFO : Submit Plan 0.03s
INFO : Start DAG 0.09s
INFO : Run DAG 1.80s
INFO :
--
INFO :
INFO : Task Execution Summary
INFO :
--
INFO : VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms)
INPUT_RECORDS OUTPUT_RECORDS
INFO :
--
INFO : Map 1 1009.00 0 0
1,8241,824
INFO : Reducer 2 0.00 0 0
1,8240
INFO :
--
INFO :
{noformat}
However, same query scans *2.8 Billion records.* in iceberg format. This can be
fixed.
{noformat}
INFO : Executing
command(queryId=hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72):
select distinct ss_sold_date_sk from store_sales
INFO : Compute 'ndembla-test2' is active.
INFO : Query ID = hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72
INFO : Tez session hasn't been created yet. Opening session
INFO : Dag name: select distinct ss_sold_date_s...store_sales (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1651102345385_)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. llap SUCCEEDED 7141 714100
0 0
Reducer 2 .. llap SUCCEEDED 2 200
0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 18.48 s
--
INFO : Status: DAG finished successfully in 17.97 seconds
INFO : DAG ID: dag_1651102345385__4
INFO :
INFO : Query Execution Summary
INFO :
--
INFO : OPERATIONDURATION
INFO :
--
INFO : Compile Query 1.81s
INFO : Prepare Plan0.04s
INFO : Get Query Coordinator
[jira] [Created] (HIVE-26181) Add details on the number of partitions/entries in dynamic partition pruning
Rajesh Balamohan created HIVE-26181: --- Summary: Add details on the number of partitions/entries in dynamic partition pruning Key: HIVE-26181 URL: https://issues.apache.org/jira/browse/HIVE-26181 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan Related ticket: HIVE-26008 It will be good to print details on the number of partition pruning entries for debugging and for understanding the eff* of the query. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HIVE-26128) Enabling dynamic runtime filtering in iceberg tables throws exception at runtime
Rajesh Balamohan created HIVE-26128:
---
Summary: Enabling dynamic runtime filtering in iceberg tables
throws exception at runtime
Key: HIVE-26128
URL: https://issues.apache.org/jira/browse/HIVE-26128
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
E.g TPCDS Q2 at 10 TB scale throws the following error when run with
"hive.disable.unsafe.external.table.operations=false". Iceberg tables were
created as external tables and setting
"hive.disable.unsafe.external.table.operations=false" will enable it to have
dynamic runtime filtering; but throws the following error at runtime
{noformat}
]Vertex failed, vertexName=Map 6, vertexId=vertex_1649658279052__1_03,
diagnostics=[Vertex vertex_1649658279052__1_03 [Map 6] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: date_dim initializer failed,
vertex=vertex_1649658279052__1_03 [Map 6],
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translateLeaf(HiveIcebergFilterFactory.java:114)
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:86)
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:80)
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.generateFilterExpression(HiveIcebergFilterFactory.java:59)
at
org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getSplits(HiveIcebergInputFormat.java:92)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:592)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:900)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:274)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:192)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:173)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
]Vertex killed, vertexName=Map 13, vertexId=vertex_1649658279052__1_07,
diagnostics=[Vertex received Kill in INITED state., Vertex
vertex_1649658279052__1_07 [Map 13] killed/failed due
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 10,
vertexId=vertex_1649658279052__1_06, diagnostics=[Vertex received Kill in
INITED state., Vertex vertex_1649658279052__1_06 [Map 10] killed/failed due
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 5,
vertexId=vertex_1649658279052__1_04, diagnostics=[Vertex received Kill in
INITED state., Vertex vertex_1649658279052__1_04 [Map 5] killed/failed due
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 4,
vertexId=vertex_1649658279052__1_11, diagnostics=[Vertex received Kill in
NEW state., Vertex vertex_1649658279052__1_11 [Reducer 4] killed/failed due
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3,
vertexId=vertex_1649658279052__1_10, diagnostics=[Vertex received Kill in
INITED state., Vertex vertex_1649658279052__1_10 [Reducer 3] killed/failed
due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 12,
vertexId=vertex_1649658279052__1_09, diagnostics=[Vertex received Kill in
INITED state., Vertex vertex_1649658279052__1_09 [Reducer 12] killed/failed
due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 1,
vertexId=vertex_1649658279052__1_08, diagnostics=[Vertex received Kill in
INITED state., Vertex vertex_1649658279052__1_08 [Map 1] killed/failed due
[jira] [Created] (HIVE-26115) Parquet footer is read 3 times when reading iceberg data
Rajesh Balamohan created HIVE-26115: --- Summary: Parquet footer is read 3 times when reading iceberg data Key: HIVE-26115 URL: https://issues.apache.org/jira/browse/HIVE-26115 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan Attachments: Screenshot 2022-04-05 at 10.08.27 AM.png, Screenshot 2022-04-05 at 10.08.35 AM.png, Screenshot 2022-04-05 at 10.08.50 AM.png, Screenshot 2022-04-05 at 10.09.03 AM.png !Screenshot 2022-04-05 at 10.08.27 AM.png|width=627,height=331! Here is the breakup of 3 footer reads per file. !Screenshot 2022-04-05 at 10.08.35 AM.png|width=1109,height=500! !Screenshot 2022-04-05 at 10.08.50 AM.png|width=1067,height=447! !Screenshot 2022-04-05 at 10.09.03 AM.png|width=827,height=303! HIVE-25827 already talks about the initial 2 footer reads per file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26110) bulk insert into partitioned table creates lots of files in iceberg
Rajesh Balamohan created HIVE-26110: --- Summary: bulk insert into partitioned table creates lots of files in iceberg Key: HIVE-26110 URL: https://issues.apache.org/jira/browse/HIVE-26110 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan For e.g, create web_returns table in tpcds in iceberg format and try to copy over data from regular table. More like "insert into web_returns_iceberg as select * from web_returns". This inserts the data correctly, however there are lot of files present in each partition. IMO, dynamic sort optimisation isn't working fine and this causes records not to be grouped in the final phase. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26091) Support DecimalFilterPredicateLeafBuilder for parquet
Rajesh Balamohan created HIVE-26091:
---
Summary: Support DecimalFilterPredicateLeafBuilder for parquet
Key: HIVE-26091
URL: https://issues.apache.org/jira/browse/HIVE-26091
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java#L41
It will nice to have DecimalFilterPredicateLeafBuilder. This will help in
supporting SARG pushdowns.
{noformat}
2022-03-30 08:59:50,040 [ERROR] [TezChild]
|read.ParquetFilterPredicateConverter|: fail to build predicate filter leaf
with errorsorg.apache.hadoop.hive.ql.metadata.HiveException: Conversion to
Parquet FilterPredicate not supported for DECIMAL
org.apache.hadoop.hive.ql.metadata.HiveException: Conversion to Parquet
FilterPredicate not supported for DECIMAL
at
org.apache.hadoop.hive.ql.io.parquet.LeafFilterFactory.getLeafFilterBuilderByType(LeafFilterFactory.java:223)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.buildFilterPredicateFromPredicateLeaf(ParquetFilterPredicateConverter.java:130)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:111)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:97)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:71)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:88)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.toFilterPredicate(ParquetFilterPredicateConverter.java:57)
at
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.setFilter(ParquetRecordReaderBase.java:184)
at
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:124)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:158)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:429)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:282)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:265)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
[jira] [Created] (HIVE-26072) Enable vectorization for stats gathering (tablescan op)
Rajesh Balamohan created HIVE-26072: --- Summary: Enable vectorization for stats gathering (tablescan op) Key: HIVE-26072 URL: https://issues.apache.org/jira/browse/HIVE-26072 Project: Hive Issue Type: Bug Components: Hive Reporter: Rajesh Balamohan https://issues.apache.org/jira/browse/HIVE-24510 enabled vectorization for compute_bit_vector. But tablescan operator for stats gathering is disabled by default. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java#L2577] Need to enable vectorization for this. This can significantly reduce runtimes for analyze statements for large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26035) Move to directsql for ObjectStore::addPartitions
Rajesh Balamohan created HIVE-26035:
---
Summary: Move to directsql for ObjectStore::addPartitions
Key: HIVE-26035
URL: https://issues.apache.org/jira/browse/HIVE-26035
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
Currently {{addPartitions}} uses datanuclues and is super slow for large number
of partitions. It will be good to move to direct sql. Lots of repeated SQLs can
be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26013) Parquet predicate filters are not properly propogated to task configs at runtime
Rajesh Balamohan created HIVE-26013:
---
Summary: Parquet predicate filters are not properly propogated to
task configs at runtime
Key: HIVE-26013
URL: https://issues.apache.org/jira/browse/HIVE-26013
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
Hive ParquetRecordReader sets the predicate filter in the config for parquet
libs to read.
Ref:
[https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L188]
{code:java}
ParquetInputFormat.setFilterPredicate(conf, p);
{code}
This internally sets {color:#FF}"parquet.private.read.filter.predicate"
{color}variable in config.
Ref:
[https://github.com/apache/parquet-mr/blob/master/parquet-hadoop%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fparquet%2Fhadoop%2FParquetInputFormat.java#L231]
Config set in compilation phase isn't visible at runtime for the tasks. This
causes filters to be lost and tasks run with excessive IO.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26008) Dynamic partition pruning not sending right partitions with subqueries
Rajesh Balamohan created HIVE-26008:
---
Summary: Dynamic partition pruning not sending right partitions
with subqueries
Key: HIVE-26008
URL: https://issues.apache.org/jira/browse/HIVE-26008
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
DPP isn't working fine when there are subqueries involved. Here is an example
query (q83).
Note that "date_dim" has another query involved. Due to this, DPP operator ends
up sending entire "date_dim" to the fact tables.
Because of this, data scanned for fact tables are way higher and query runtime
is increased.
For context, on a very small cluster, this query ran for 265 seconds and with
the rewritten query it finished in 11 seconds!. Fact table scan was 10MB vs 10
GB.
{noformat}
HiveJoin(condition=[=($2, $5)], joinType=[inner])
HiveJoin(condition=[=($0, $3)], joinType=[inner])
HiveProject(cr_item_sk=[$1], cr_return_quantity=[$16],
cr_returned_date_sk=[$26])
HiveFilter(condition=[AND(IS NOT NULL($26), IS NOT
NULL($1))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1,
catalog_returns]], table:alias=[catalog_returns])
HiveProject(i_item_sk=[$0], i_item_id=[$1])
HiveFilter(condition=[AND(IS NOT NULL($1), IS NOT
NULL($0))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1,
item]], table:alias=[item])
HiveProject(d_date_sk=[$0], d_date=[$2])
HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($0))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1,
date_dim]], table:alias=[date_dim])
HiveProject(d_date=[$0])
HiveSemiJoin(condition=[=($1, $2)], joinType=[semi])
HiveProject(d_date=[$2], d_week_seq=[$4])
HiveFilter(condition=[AND(IS NOT NULL($4), IS NOT
NULL($2))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1,
date_dim]], table:alias=[date_dim])
HiveProject(d_week_seq=[$4])
HiveFilter(condition=[AND(IN($2, 1998-01-02:DATE,
1998-10-15:DATE, 1998-11-10:DATE), IS NOT NULL($4))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1,
date_dim]], table:alias=[date_dim])
{noformat}
*Original Query & Plan: *
{noformat}
explain cbo with sr_items as
(select i_item_id item_id,
sum(sr_return_quantity) sr_item_qty
from store_returns,
item,
date_dim
where sr_item_sk = i_item_sk
and d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and sr_returned_date_sk = d_date_sk
group by i_item_id),
cr_items as
(select i_item_id item_id,
sum(cr_return_quantity) cr_item_qty
from catalog_returns,
item,
date_dim
where cr_item_sk = i_item_sk
and d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and cr_returned_date_sk = d_date_sk
group by i_item_id),
wr_items as
(select i_item_id item_id,
sum(wr_return_quantity) wr_item_qty
from web_returns,
item,
date_dim
where wr_item_sk = i_item_sk
and d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and wr_returned_date_sk = d_date_sk
group by i_item_id)
select sr_items.item_id
,sr_item_qty
,sr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 sr_dev
,cr_item_qty
,cr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 cr_dev
,wr_item_qty
,wr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 wr_dev
,(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 average
from sr_items
,cr_items
,wr_items
where sr_items.item_id=cr_items.item_id
and sr_items.item_id=wr_items.item_id
order by sr_items.item_id
,sr_item_qty
limit 100
INFO : Starting task [Stage-3:EXPLAIN] in serial mode
INFO : Completed executing
command(queryId=hive_20220307055109_88ad0cbd-bd40-45bc-92ae-ab15fa6b1da4); Time
taken: 0.973 seconds
INFO : OK
Explain
CBO PLAN:
HiveSortLimit(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC], fetch=[100])
HiveProject(item_id=[$0], sr_item_qty=[$4], sr_dev=[*(/(/($5, CAST(+(+($4,
$1), $7)):DOUBLE), 3), 100)], cr_item_qty=[$1], cr_dev=[*(/(/($2, CAST(+(+($4,
$1), $7)):DOUBLE), 3), 100)], wr_item_qty=[$7], wr_dev=[*(/(/($8, CAST(+(+($4,
$1), $7)):DOUBLE), 3), 100)], average=[/(CAST(+(+($4, $1), $7)):DECIMAL(19, 0),
3:DECIMAL(1, 0))])
HiveJoin(condition=[=($0, $6)], joinType=[inner])
HiveJoin(condition=[=($3, $0)], joinType=[inner])
HiveProject($f0=[$0], $f1=[$1], EXPR$0=[CAST($1):DOUBLE])
HiveAggregate(group=[{4}], agg#0=[sum($1)])
[jira] [Created] (HIVE-25981) Avoid checking for archived parts in analyze table
Rajesh Balamohan created HIVE-25981:
---
Summary: Avoid checking for archived parts in analyze table
Key: HIVE-25981
URL: https://issues.apache.org/jira/browse/HIVE-25981
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
Analyze table on large partitioned table is expensive due to unwanted checks on
archived data.
{noformat}
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3908)
- locked <0x0003d4c4c070> (a
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler)
at com.sun.proxy.$Proxy56.listPartitionsWithAuthInfo(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:3845)
at
org.apache.hadoop.hive.ql.exec.ArchiveUtils.conflictingArchiveNameOrNull(ArchiveUtils.java:299)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.validate(SemanticAnalyzer.java:13579)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:241)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:561)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:555)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:285)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-25958) Optimise BasicStatsNoJobTask
Rajesh Balamohan created HIVE-25958:
---
Summary: Optimise BasicStatsNoJobTask
Key: HIVE-25958
URL: https://issues.apache.org/jira/browse/HIVE-25958
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
When there are large number of files are present, it takes lot of time for
analyzing table (for stats) takes lot longer time especially on cloud
platforms. Each file is read in sequential fashion for computing stats, which
can be optimized.
{code:java}
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:293)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:506)
- locked <0x000642995b10> (a org.apache.hadoop.fs.s3a.S3AInputStream)
at
org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:775)
- locked <0x000642995b10> (a org.apache.hadoop.fs.s3a.S3AInputStream)
at
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:116)
at
org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:574)
at
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:282)
at
org.apache.orc.impl.RecordReaderImpl.readAllDataStreams(RecordReaderImpl.java:1172)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1128)
at
org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1281)
at
org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1316)
at org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:302)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:68)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:83)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:367)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.(OrcInputFormat.java:276)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2027)
at
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask$FooterStatCollector.run(BasicStatsNoJobTask.java:235)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"HiveServer2-Background-Pool: Thread-5161" #5161 prio=5 os_prio=0
tid=0x7f271217d800 nid=0x21b7 waiting on condition [0x7f26fce88000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0006bee1b3a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
at
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.shutdownAndAwaitTermination(BasicStatsNoJobTask.java:426)
at
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.aggregateStats(BasicStatsNoJobTask.java:338)
at
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.process(BasicStatsNoJobTask.java:121)
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:250) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-25927) Fix DataWritableReadSupport
Rajesh Balamohan created HIVE-25927: --- Summary: Fix DataWritableReadSupport Key: HIVE-25927 URL: https://issues.apache.org/jira/browse/HIVE-25927 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2022-02-04 at 4.57.22 AM.png !Screenshot 2022-02-04 at 4.57.22 AM.png|width=530,height=406! Takes n^2 ops to match columns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-25913) Dynamic Partition Pruning Operator: Not working in iceberg tables
Rajesh Balamohan created HIVE-25913:
---
Summary: Dynamic Partition Pruning Operator: Not working in
iceberg tables
Key: HIVE-25913
URL: https://issues.apache.org/jira/browse/HIVE-25913
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Notice "Dynamic Partitioning Event Operator" missing in Map 3 in iceberg tables.
This causes heavy IO in iceberg tables leading to perf degradation.
{noformat}
ACID table
==
explain select count(*) from store_sales, date_dim where d_month_seq between
1212 and 1212+11 and ss_store_sk is not null and ss_sold_date_sk=d_date_sk;
Explain
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20220131032425_be2fab7f-7943-4aa1-bbdd-289139ea0f90:17
Edges:
Map 1 <- Map 3 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
DagName: hive_20220131032425_be2fab7f-7943-4aa1-bbdd-289139ea0f90:17
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: ss_store_sk is not null (type: boolean)
Statistics: Num rows: 27503885621 Data size: 434880571744
Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ss_store_sk is not null (type: boolean)
Statistics: Num rows: 26856185846 Data size: 424639398832
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: ss_sold_date_sk (type: bigint)
outputColumnNames: _col0
Statistics: Num rows: 26856185846 Data size: 214849486768
Basic stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
0 _col0 (type: bigint)
1 _col0 (type: bigint)
input vertices:
1 Map 3
Statistics: Num rows: 5279977323 Data size: 42239818584
Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
minReductionHashAggr: 0.99
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
null sort order:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: may be used (ACID table)
Map 3
Map Operator Tree:
TableScan
alias: date_dim
filterExpr: (d_month_seq BETWEEN 1212 AND 1223 and d_date_sk
is not null) (type: boolean)
Statistics: Num rows: 73049 Data size: 876588 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (d_month_seq BETWEEN 1212 AND 1223 and d_date_sk
is not null) (type: boolean)
Statistics: Num rows: 359 Data size: 4308 Basic stats:
COMPLETE Column stats: COMPLETE
Select Operator
expressions: d_date_sk (type: bigint)
outputColumnNames: _col0
Statistics: Num rows: 359 Data size: 2872 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: bigint)
null sort order: a
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 359 Data size: 2872 Basic stats:
COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: bigint)
outputColumnNames: _col0
Statistics: Num rows: 359 Data size: 2872 Basic stats:
COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: bigint)
minReductionHashAggr: 0.5013927
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 179 Data size: 1432 Basic
stats: COMPLETE Column stats: COMPLETE
Dynamic Partitioning Event Operator
[jira] [Created] (HIVE-25845) Support ColumnIndexes for Parq files
Rajesh Balamohan created HIVE-25845: --- Summary: Support ColumnIndexes for Parq files Key: HIVE-25845 URL: https://issues.apache.org/jira/browse/HIVE-25845 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan https://issues.apache.org/jira/browse/PARQUET-1201 [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L271-L273] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file
Rajesh Balamohan created HIVE-25827: --- Summary: Parquet file footer is read multiple times, when multiple splits are created in same file Key: HIVE-25827 URL: https://issues.apache.org/jira/browse/HIVE-25827 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: image-2021-12-21-03-19-38-577.png With large files, it is possible that multiple splits are created in the same file. With current codebase, "ParquetRecordReaderBase" ends up reading file footer for each split. It can be optimized not to read footer information multiple times for the same file. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L160] [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L91] !image-2021-12-21-03-19-38-577.png|width=1363,height=1256! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks
Rajesh Balamohan created HIVE-25483: --- Summary: TxnHandler::acquireLock should close the DB conn to avoid connection leaks Key: HIVE-25483 URL: https://issues.apache.org/jira/browse/HIVE-25483 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan TxnHandler::acquireLock should close DB connection on exiting the function. [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688] [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726] [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25482) Add option to enable connectionLeak detection for Hikari datasource
Rajesh Balamohan created HIVE-25482:
---
Summary: Add option to enable connectionLeak detection for Hikari
datasource
Key: HIVE-25482
URL: https://issues.apache.org/jira/browse/HIVE-25482
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
There are corner cases where we observed connection leaks to DB.
It will be good to add an option to provide connection leak timeout parameter
in HikariCPDataSourceProvider.
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java#L69]
e.g following should help Hikari to warn about connection leak, when a
connection is not returned to the pool for 1 hour.
{noformat}
config.setLeakDetectionThreshold(3600*1000); {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25414) Optimise Hive::addWriteNotificationLog: Reduce FS call per notification
Rajesh Balamohan created HIVE-25414: --- Summary: Optimise Hive::addWriteNotificationLog: Reduce FS call per notification Key: HIVE-25414 URL: https://issues.apache.org/jira/browse/HIVE-25414 Project: Hive Issue Type: Sub-task Reporter: Rajesh Balamohan AddWriteNotification is slow due to FS interactions (i.e to get the set of insert file information). This can be avoided as FileStatus can be passed instead of Path from parent methods. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3572] [https://github.com/apache/hive/blob/96b39cd5190f0cfadb677e3f3b7ead1d663921b2/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3620] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25413) Optimise ObjectStore::alterPartitions to reduce DB calls for getPartitions
Rajesh Balamohan created HIVE-25413: --- Summary: Optimise ObjectStore::alterPartitions to reduce DB calls for getPartitions Key: HIVE-25413 URL: https://issues.apache.org/jira/browse/HIVE-25413 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L5121] For dynamic load partitions (e.g when 1000s of parts being updated), it takes lot of time to fetch one partition at a time. It would be good to have "List getMPartitions" to reduce number of interactions with DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25391) Turn off priv synchronizer by default
Rajesh Balamohan created HIVE-25391: --- Summary: Turn off priv synchronizer by default Key: HIVE-25391 URL: https://issues.apache.org/jira/browse/HIVE-25391 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan In large deployments, good set of scale issues were identified with priv synchronizer. It will be good to turn it off by default, and let the users enable it on need basis. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25366) Reduce number of Table calls in updatePartitonColStatsInternal
Rajesh Balamohan created HIVE-25366: --- Summary: Reduce number of Table calls in updatePartitonColStatsInternal Key: HIVE-25366 URL: https://issues.apache.org/jira/browse/HIVE-25366 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan For every partition, table details are reloaded again which is completely wasteful. It will be good to pass the Table details fetched once for all partitions. [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java#L1091] [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L9342] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25324) Add option to disable PartitionManagementTask
Rajesh Balamohan created HIVE-25324:
---
Summary: Add option to disable PartitionManagementTask
Key: HIVE-25324
URL: https://issues.apache.org/jira/browse/HIVE-25324
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Rajesh Balamohan
When large number of tables (e.g 2000) and databases are present,
PartitionManagementTask scans all tables and partitions causing pressure on HMS.
Currently there is no way to disable PartitionManagementTask as well. Round
about option is to provide pattern via
"metastore.partition.management.database.pattern /
metastore.partition.management.table.pattern".
It will be good to provide an option to disable it completely.{color:#807d6e}
{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions
Rajesh Balamohan created HIVE-25178: --- Summary: Reduce number of getPartition calls during loadDynamicPartitions Key: HIVE-25178 URL: https://issues.apache.org/jira/browse/HIVE-25178 Project: Hive Issue Type: Improvement Components: Hive Reporter: Rajesh Balamohan When dynamic partitions are loaded, Hive::loadDynamicPartition loads all partitions from HMS causing heavy load on it. This becomes worse when large number of partitions are present in tables. Only relevant partitions being loaded in dynamic partitions can be queried from HMS for partition existence. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25077) Direct SQL to fetch column privileges in refreshPrivileges may be broken in postgres
Rajesh Balamohan created HIVE-25077:
---
Summary: Direct SQL to fetch column privileges in
refreshPrivileges may be broken in postgres
Key: HIVE-25077
URL: https://issues.apache.org/jira/browse/HIVE-25077
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
HIVE-22512 tried to fix direct-sql for col privileges.
However, "GRANT_OPTION" field in "TBL_COL_PRIVS" is marked as smallint in
postgres. In code, it is retrieved as boolean.
Ref:
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1533]
{code:java}
boolean grantOption =
MetastoreDirectSqlUtils.extractSqlBoolean(privLine[grantOptionIndex]);
{code}
[https://github.com/apache/hive/blob/048336bd0c21163920557a60c88135b1d5b42d3d/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L530]
MetastoreDirectSqlUtils::extractSqlBoolean should handle integers to support
directSQL in postgres.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25075) Hive::loadPartitionInternal establishes HMS connection for every partition for external tables
Rajesh Balamohan created HIVE-25075:
---
Summary: Hive::loadPartitionInternal establishes HMS connection
for every partition for external tables
Key: HIVE-25075
URL: https://issues.apache.org/jira/browse/HIVE-25075
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2522
{code}
boolean needRecycle = !tbl.isTemporary()
&&
ReplChangeManager.shouldEnableCm(Hive.get().getDatabase(tbl.getDbName()),
tbl.getTTable());
{code}
Hive.get() breaks the current connection with HMS. Due to this, for external
table partition loads, it establishes HMS connection for partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25073) Optimise HiveAlterHandler::alterPartitions
Rajesh Balamohan created HIVE-25073: --- Summary: Optimise HiveAlterHandler::alterPartitions Key: HIVE-25073 URL: https://issues.apache.org/jira/browse/HIVE-25073 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Rajesh Balamohan Table details are populated again and again for each partition, which can be avoided. https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L5892 https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L808 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25072) Optimise ObjectStore::alterPartitions
Rajesh Balamohan created HIVE-25072:
---
Summary: Optimise ObjectStore::alterPartitions
Key: HIVE-25072
URL: https://issues.apache.org/jira/browse/HIVE-25072
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Rajesh Balamohan
Avoid fetching table details for every partition in the table.
Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L5104
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L4986
Following stacktrace may be relevant for apache master as well.
{noformat}
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2113)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2152)
at
org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4951)
at
org.apache.hadoop.hive.metastore.ObjectStore.alterPartitions(ObjectStore.java:5057)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.alterPartitions(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:798)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:5695)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:5647)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.alter_partitions_req(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18557)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18541)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25032) Optimise PartitionManagementTask
Rajesh Balamohan created HIVE-25032: --- Summary: Optimise PartitionManagementTask Key: HIVE-25032 URL: https://issues.apache.org/jira/browse/HIVE-25032 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan When large number of tables and dbs are present, it takes lot of time to iterate over all tables in PartitionManagementTask. If table parameters are populated in TableMeta, it will cut down lots of interactions with HMS in PartitionManagementTask. Code Ref: [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L126] [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L132] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25014) Optimise ObjectStore::updateTableColumnStatistics
Rajesh Balamohan created HIVE-25014: --- Summary: Optimise ObjectStore::updateTableColumnStatistics Key: HIVE-25014 URL: https://issues.apache.org/jira/browse/HIVE-25014 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan "ObjectStore::updateTableColumnStatistics" fetches table level details multiple times. Need to consider reducing the getTable lookups to avoid pressure on HMS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24960) HMS: Fix initial delay of housekeeping threads in HMS
Rajesh Balamohan created HIVE-24960: --- Summary: HMS: Fix initial delay of housekeeping threads in HMS Key: HIVE-24960 URL: https://issues.apache.org/jira/browse/HIVE-24960 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Rajesh Balamohan HMS starts house keeping threads in background. However, its initial delay is set to the frequency mentioned in MetastoreTaskThread. [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L761] For e.g, "hive.hook.proto.events.clean.freq" defaults to "1D" and the cleaner starts its initial iteration after a day. For frequently stopped/started clusters in cloud, this will not be ideal. It will be good to introduce small random delay for the initial scheduling of these threads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24943) Initiator: Short circuit compaction checks on tables/partitions when DB isn't eligible for compaction
Rajesh Balamohan created HIVE-24943:
---
Summary: Initiator: Short circuit compaction checks on
tables/partitions when DB isn't eligible for compaction
Key: HIVE-24943
URL: https://issues.apache.org/jira/browse/HIVE-24943
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
When the entire database isn't qualified for compaction due to replication,
initiator could potentially skip the entire table/partition checks to save HMS
resources. Currently it runs in tight loop for each table/partition for the
database which isn't eligible for compaction.
E.g Note that for "delete_orc_10" database repl parameters are enabled. So
until "ReplUtils.isFirstIncPending(database.getParameters());", it ends up
considering that entire database as "not chosen for compaction".
Ref:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L469]
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L469]
{noformat}
++
| createtab_stmt |
++
| CREATE TABLE `delete_orc_10.test_con_s3_part_1`( |
| `d_date_id` string, |
| `d_date` date, |
| `d_dom` int, |
| `d_year` int)|
| PARTITIONED BY ( |
| `d_date_sk` bigint) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION |
| 's3a://blah/delete_orc_10.db/test_con_s3_part_1' |
| TBLPROPERTIES (|
| 'bucketing_version'='2', |
| 'transactional'='true', |
| 'transactional_properties'='default',|
| 'transient_lastDdlTime'='1610926861')|
++
++
| createdb_stmt|
++
| CREATE DATABASE `delete_orc_10`|
| LOCATION |
| 's3a://blah/delete_orc_10.db' |
| WITH DBPROPERTIES (|
|
'hive.repl.ckpt.key'='s3a://blha/loc/dHBjZHNfYmluX3BhcnRpdGlvbmVkX29yY18xMDAwMA==/7f1c5529-e17a-4750-bf16-a9861c9589dc/hive',
|
| 'hive.repl.first.inc.pending'='true',|
| 'repl.last.id'='95653') |
++
{noformat}
{noformat}
<14>1 2021-03-25T23:42:50.095Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.CompactorThread" level="INFO" thread="Thread-12"] Compaction
is disabled for database delete_orc_10
<14>1 2021-03-25T23:42:50.095Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.Initiator" level="INFO" thread="Thread-12"] Checking to see if
we should compact delete_orc_10.test_con_s3_part_1.d_date_sk=2447933
<14>1 2021-03-25T23:42:50.118Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.CompactorThread" level="INFO" thread="Thread-12"] Compaction
is disabled for database delete_orc_10
<14>1 2021-03-25T23:42:50.118Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.Initiator" level="INFO" thread="Thread-12"] Checking to see if
we should compact delete_orc_10.test_con_s3_part_debug.d_date_sk=2471951
<14>1 2021-03-25T23:42:50.142Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.CompactorThread" level="INFO" thread="Thread-12"] Compaction
is disabled for database delete_orc_10
<14>1 2021-03-25T23:42:50.142Z
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060
class="compactor.Initiator" level="INFO" thread="Thread-12"] Checking to see if
we should compact
[jira] [Created] (HIVE-24937) Fix sync bottleneck in SyslogParser
Rajesh Balamohan created HIVE-24937: --- Summary: Fix sync bottleneck in SyslogParser Key: HIVE-24937 URL: https://issues.apache.org/jira/browse/HIVE-24937 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: sample-thread-dump.txt https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java#L255 Sample threadump attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24892) Replace getContentSummary::getLength with listStatus(recursive) for blobstores
Rajesh Balamohan created HIVE-24892: --- Summary: Replace getContentSummary::getLength with listStatus(recursive) for blobstores Key: HIVE-24892 URL: https://issues.apache.org/jira/browse/HIVE-24892 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStats.java#L219 For blobstores, getContentSummary() is super slow. It would be good to replace this with "fs.listFiles(path, true)". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24862) Fix race condition causing NPE during dynamic partition loading
Rajesh Balamohan created HIVE-24862:
---
Summary: Fix race condition causing NPE during dynamic partition
loading
Key: HIVE-24862
URL: https://issues.apache.org/jira/browse/HIVE-24862
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Following properties default to 15 threads.
{noformat}
hive.load.dynamic.partitions.thread
hive.mv.files.thread
{noformat}
During loadDynamicPartitions, it ends ups initializing {{newFiles}} without
synchronization (HIVE-20661, HIVE-24738).
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2871]
This causes race condition when dynamic partition thread internally makes use
of {{hive.mv.files.threads}} in copyFiles/replaceFiles.
This causes "NPE" during retrieval in {{addInsertFileInformation()}}.
e.g stacktrace
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3396)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1740)
at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1740)
at
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:3540)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2414)
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24838) Reduce FS creation in Warehouse::getDnsPath for object stores
Rajesh Balamohan created HIVE-24838: --- Summary: Reduce FS creation in Warehouse::getDnsPath for object stores Key: HIVE-24838 URL: https://issues.apache.org/jira/browse/HIVE-24838 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2021-03-02 at 11.09.01 AM.png [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java#L143] Warehouse::getDnsPath gets invoked from multiple places (e.g getDatabase() etc). In certain cases like dynamic partition loads, lot of calls FS instantiation calls can be avoided for object stores. It would be good to check for BlobStorages and if so, it should be possible to avoid FS creation. [https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/BlobStorageUtils.java#L33] !Screenshot 2021-03-02 at 11.09.01 AM.png|width=372,height=296! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24806) Compactor: Initiator should lazy evaluate findUserToRunAs()
Rajesh Balamohan created HIVE-24806:
---
Summary: Compactor: Initiator should lazy evaluate
findUserToRunAs()
Key: HIVE-24806
URL: https://issues.apache.org/jira/browse/HIVE-24806
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L232
{noformat}
cache.putIfAbsent(fullTableName, findUserToRunAs(sd.getLocation(), t));
{noformat}
This ends up evaluating findUserToRunAs() everytime, and looks up from
FileSystem on every call (thousands of times in large database).
This can be lazy initialized instead (e.g computeIfAbsent);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
Rajesh Balamohan created HIVE-24805:
---
Summary: Compactor: Initiator shouldn't fetch table details again
and again for partitioned tables
Key: HIVE-24805
URL: https://issues.apache.org/jira/browse/HIVE-24805
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Initiator shouldn't be fetch table details for all its partitions. When there
are large number of databases/tables, it takes lot of time for Initiator to
complete its initial iteration and load on DB also goes higher.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
For all the following partitions, table details would be the same. However, it
ends up fetching table details from HMS again and again.
{noformat}
2021-02-22 08:13:16,106 INFO
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see
if we should compact
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
2021-02-22 08:13:16,124 INFO
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see
if we should compact
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
2021-02-22 08:13:16,140 INFO
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see
if we should compact
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
2021-02-22 08:13:16,149 INFO
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see
if we should compact
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
2021-02-22 08:13:16,158 INFO
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see
if we should compact
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24790) Batch column stats updates to HMS
Rajesh Balamohan created HIVE-24790:
---
Summary: Batch column stats updates to HMS
Key: HIVE-24790
URL: https://issues.apache.org/jira/browse/HIVE-24790
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When large number of partitions are inserted/updated, it would be good to batch
column statistics updates to HMS.
Currently, HS2 ends up throwing read timeout exception when updating HMS.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L180
{noformat}
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_252]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:171)
~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_252]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
~[?:1.8.0_252]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
~[?:1.8.0_252]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
~[?:1.8.0_252]
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-3.1]
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
~[hive-exec-3.1]
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
~[hive-exec-3.1]
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
~[hive-exec-3.1]
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:62)
~[hive-exec-3.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
~[hive-exec-3.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
~[hive-exec-3.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
~[hive-exec-3.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:4561)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:4548)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:2496)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.setPartitionColumnStatistics(SessionHiveMetaStoreClient.java:515)
~[hive-exec-3.1]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source)
~[?:?]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3431)
~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source)
~[?:?]
at
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:5213)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:192)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:87)
~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
~[hive-exec-3.1]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
~[hive-exec-3.1]
[jira] [Created] (HIVE-24776) Reduce HMS DB calls during stats updates
Rajesh Balamohan created HIVE-24776:
---
Summary: Reduce HMS DB calls during stats updates
Key: HIVE-24776
URL: https://issues.apache.org/jira/browse/HIVE-24776
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
When adding large number of partitions (100s/1000s) in a table, it ends up
making lots of getTable calls which are not needed.
Lines mentioned below may vary slightly in apache-master.
{noformat}
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620)
at
org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637)
at
org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150)
at
org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578)
at
org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588)
at
org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown
Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.set_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19060)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24774) Reduce FS listing during dynamic partition loading
Rajesh Balamohan created HIVE-24774:
---
Summary: Reduce FS listing during dynamic partition loading
Key: HIVE-24774
URL: https://issues.apache.org/jira/browse/HIVE-24774
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When loading large number of partitions in cloud storage, notification log
takes lot longer time to list newly added files.
It would be good to explore if FileStatus can be reused from
Hive::listFilesCreatedByQuery or from copyFiles
{noformat}
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4171)
at
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
at
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3519)
at
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3504)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:2984)
at
org.apache.hadoop.hive.ql.exec.MoveTask.handleDynParts(MoveTask.java:562)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:440)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:730)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:490)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:484)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24764) insert overwrite on a partition resets row count stats in other partitions
Rajesh Balamohan created HIVE-24764:
---
Summary: insert overwrite on a partition resets row count stats in
other partitions
Key: HIVE-24764
URL: https://issues.apache.org/jira/browse/HIVE-24764
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
After insert overwrite on a partition, stats on other partitions are messed up.
Subsequent queries end up with plans with PARTIAL stats. In certain cases, this
leads to suboptimal query plans.
{noformat}
drop table if exists test_stats;
drop table if exists test_stats_2;
create table test_stats(i int, j bigint);
create table test_stats_2(i int) partitioned by (j bigint);
insert into test_stats values (1, 1), (2, 2), (3, 3), (4, 4), (5, NULL);
-- select * from test_stats;
1 1
2 2
3 3
4 4
5
insert overwrite table test_stats_2 partition(j) select i, j from test_stats
where j is not null;
-- After executing this statement, stat gets messed up.
insert overwrite table test_stats_2 partition(j) select i, j from test_stats
where j is null;
-- select * from test_stats_2;
1 1
2 2
3 3
4 4
5
-- This would return "PARTIAL" stats instead of "COMPLETE"
explain select i, count(*) as c from test_stats_2 group by i order by c desc
limit 10;
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
DagId: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
DagName: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: test_stats_2
Statistics: Num rows: 125 Data size: 500 Basic stats: PARTIAL
Column stats: COMPLETE
Select Operator
expressions: i (type: int)
outputColumnNames: i
Statistics: Num rows: 125 Data size: 500 Basic stats:
PARTIAL Column stats: COMPLETE
Group By Operator
aggregations: count()
keys: i (type: int)
minReductionHashAggr: 0.99
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 125 Data size: 1500 Basic stats:
PARTIAL Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: int)
null sort order: a
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 125 Data size: 1500 Basic stats:
PARTIAL Column stats: COMPLETE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: may be used (ACID table)
Reducer 2
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL
Column stats: COMPLETE
Top N Key Operator
sort order: -
keys: _col1 (type: bigint)
null sort order: a
Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL
Column stats: COMPLETE
top n: 10
Reduce Output Operator
key expressions: _col1 (type: bigint)
null sort order: a
sort order: -
Statistics: Num rows: 62 Data size: 744 Basic stats:
PARTIAL Column stats: COMPLETE
TopN Hash Memory Usage: 0.04
value expressions: _col0 (type: int)
Reducer 3
Execution mode: vectorized, llap
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey0 (type:
bigint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL
Column stats: COMPLETE
Limit
Number of rows: 10
Statistics: Num rows: 10 Data size: 120 Basic stats: PARTIAL
Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 10 Data size: 120 Basic stats:
PARTIAL Column stats: COMPLETE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output
[jira] [Created] (HIVE-24710) PTFRowContainer could be reading more number of blocks than needed
Rajesh Balamohan created HIVE-24710:
---
Summary: PTFRowContainer could be reading more number of blocks
than needed
Key: HIVE-24710
URL: https://issues.apache.org/jira/browse/HIVE-24710
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
PTFRowContainer could be reading the same block repeatedly for the first block.
Default block size is around 25000. For the first 25000 rowIdx, it would read
the block repeatedly due to ("rowIdx < currentReadBlockStartRow ") condition.
{noformat}
public Row getAt(int rowIdx) throws HiveException {
int blockSize = getBlockSize();
if ( rowIdx < currentReadBlockStartRow || rowIdx >=
currentReadBlockStartRow + blockSize ) {
readBlock(getBlockNum(rowIdx));
}
return getReadBlockRow(rowIdx - currentReadBlockStartRow);
}
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24695) Clean up session resources, if TezSession is unable to start
Rajesh Balamohan created HIVE-24695: --- Summary: Clean up session resources, if TezSession is unable to start Key: HIVE-24695 URL: https://issues.apache.org/jira/browse/HIVE-24695 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan There are cases when TezSessionState would not be able to start. (e.g resource constraints on YARN queues). However by this time, session directories & certain resources are localized. (e.g, hive-exec jars are stored in hdfs:///tmp/hive/hive/_tez_session_dir/*/hive-exec*.jar). When tezClient is not started, it does not clear up the resources. This leaks ~100MB data in HDFS per failure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24663) Batch process in ColStatsProcessor
Rajesh Balamohan created HIVE-24663:
---
Summary: Batch process in ColStatsProcessor
Key: HIVE-24663
URL: https://issues.apache.org/jira/browse/HIVE-24663
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
When large number of partitions (>20K) are processed, ColStatsProcessor runs
into DB issues.
{{ db.setPartitionColumnStatistics(request);}} gets stuck for hours together
and in some cases postgres stops processing.
It would be good to introduce small batches for stats gathering in
ColStatsProcessor instead of bulk update.
Ref:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L181
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L199
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24662) Append txn_id to table path location
Rajesh Balamohan created HIVE-24662: --- Summary: Append txn_id to table path location Key: HIVE-24662 URL: https://issues.apache.org/jira/browse/HIVE-24662 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan It would be good to append txn_id in the path during table creation. This would help later in optimizing for drop/create table scenario (as drop can purge the data later and create would point to new location automatically). It would also help in having undrop table feature later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24649) Optimise Hive::addWriteNotificationLog for large data inserts
Rajesh Balamohan created HIVE-24649:
---
Summary: Optimise Hive::addWriteNotificationLog for large data
inserts
Key: HIVE-24649
URL: https://issues.apache.org/jira/browse/HIVE-24649
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When loading dynamic partition with large dataset, it spends lot of time in
"Hive::loadDynamicPartitions --> addWriteNotificationLog".
Though it is for same for same table, it ends up loading table and partition
details for every partition and writes to notification log.
Also, "Partition" details may be already present in {{PartitionDetails}} object
in {{Hive::loadDynamicPartitions}}. This is unnecessarily recomputed again in
{{HiveMetaStore::add_write_notification_log}}
Lines of interest:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3028
https://github.com/apache/hive/blob/89073a94354f0cc14ec4ae0a43e05aae29276b4d/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8500
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24596) Explain ddl for debugging
Rajesh Balamohan created HIVE-24596: --- Summary: Explain ddl for debugging Key: HIVE-24596 URL: https://issues.apache.org/jira/browse/HIVE-24596 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan For debugging query issues, basic details like table schema, statistics, partition details, query plans are needed. It would be good to have "explain ddl" support, which can generate these details. This can help in recreating the schema and planner issues without sample data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24546) Avoid unwanted cloud storage call during dynamic partition load
Rajesh Balamohan created HIVE-24546:
---
Summary: Avoid unwanted cloud storage call during dynamic
partition load
Key: HIVE-24546
URL: https://issues.apache.org/jira/browse/HIVE-24546
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
{code:java}
private void createDpDirCheckSrc(final Path dpStagingPath, final Path
dpFinalPath) throws IOException {
if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
fs.mkdirs(dpStagingPath);
// move task will create dp final path
if (reporter != null) {
reporter.incrCounter(counterGroup,
Operator.HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, 1);
}
}
}
{code}
{noformat}
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:370)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1960)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3164)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2899)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1723)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4157)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createDpDir(FileSinkOperator.java:948)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.updateDPCounters(FileSinkOperator.java:916)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:849)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:814)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:1200)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:1324)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1036)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
at
org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24537) Optimise locking in LlapTaskSchedulerService
Rajesh Balamohan created HIVE-24537: --- Summary: Optimise locking in LlapTaskSchedulerService Key: HIVE-24537 URL: https://issues.apache.org/jira/browse/HIVE-24537 Project: Hive Issue Type: Improvement Components: llap Reporter: Rajesh Balamohan Attachments: Screenshot 2020-12-15 at 11.41.49 AM.png 1. Read lock should suffice for "notifyStarted()". 2. Locking in "allocateTask()" can be optimised. 3. Optimize preemptTasks() & preemptTasksFromMap(). This would help in reducing the codepath with writeLock. Currently, it iterates through all tasks. !Screenshot 2020-12-15 at 11.41.49 AM.png|width=847,height=446! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24520) Fix stackoverflow error in HiveMetaStore::get_partitions_by_names
Rajesh Balamohan created HIVE-24520:
---
Summary: Fix stackoverflow error in
HiveMetaStore::get_partitions_by_names
Key: HIVE-24520
URL: https://issues.apache.org/jira/browse/HIVE-24520
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Need to fix the recursive call of the same method.
(May have been introduced as a part of
https://issues.apache.org/jira/browse/HIVE-22017)
{code:java}
@Override
@Deprecated
public List get_partitions_by_names(final String dbName, final
String tblName,
final List partNames)
throws TException {
return get_partitions_by_names(dbName, tblName, partNames);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
Rajesh Balamohan created HIVE-24519:
---
Summary: Optimize MV: Materialized views should not rebuild when
tables are not modified
Key: HIVE-24519
URL: https://issues.apache.org/jira/browse/HIVE-24519
Project: Hive
Issue Type: Improvement
Components: Materialized views
Reporter: Rajesh Balamohan
e.g
{noformat}
create materialized view c_c_address as
select c_customer_sk from customer c, customer_address ca where
c_current_addr_sk = ca.ca_address_id;
ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
rebuild, when source tables are not modified
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24515) Analyze table job can be skipped when stats populated are already accurate
Rajesh Balamohan created HIVE-24515:
---
Summary: Analyze table job can be skipped when stats populated are
already accurate
Key: HIVE-24515
URL: https://issues.apache.org/jira/browse/HIVE-24515
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
For non-partitioned tables, stats detail should be present in table level,
e.g
{noformat}
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"d_current_day":"true"...
}}
{noformat}
For partitioned tables, stats detail should be present in partition level,
{noformat}
store_sales(ss_sold_date_sk=2451819)
{totalSize=0, numRows=0, rawDataSize=0,
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"ss_addr_sk":"true"}}
{noformat}
When stats populated are already accurate, {{analyze table tn compute
statistics for columns}} should skip launching the job.
For ACID tables, stats are auto computed and it can skip computing stats again
when stats are accurate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24472) Optimize LlapTaskSchedulerService::preemptTasksFromMap
Rajesh Balamohan created HIVE-24472: --- Summary: Optimize LlapTaskSchedulerService::preemptTasksFromMap Key: HIVE-24472 URL: https://issues.apache.org/jira/browse/HIVE-24472 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2020-12-03 at 12.13.03 PM.png !Screenshot 2020-12-03 at 12.13.03 PM.png|width=1063,height=571! speculativeTasks could possibly include node information to reduce CPU burn in LlapTaskSchedulerService::preemptTasksFromMap -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24466) insert queries should not launch job when condition in the query would output 0 rows
Rajesh Balamohan created HIVE-24466:
---
Summary: insert queries should not launch job when condition in
the query would output 0 rows
Key: HIVE-24466
URL: https://issues.apache.org/jira/browse/HIVE-24466
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
{noformat}
-- This query would not generate any output and does not launch a job
select * from tpcds_bin_partitioned_orc_3.store_sales where 1 = 2;
-- This query generates a job (M -> R -> R) and runs for 30+ seconds in 2 node
cluster to generate 0 rows.
insert into table delete_orc_10.test_sales_1 select * from
tpcds_bin_partitioned_orc_3.store_sales where 1 = 2;
insert overwrite table delete_orc_10.test_sales_1 select * from
tpcds_bin_partitioned_orc_3.store_sales where ss_sold_date_sk >=2450816+300
and ss_sold_date_sk <= (2450816+100);
INFO : Status: Running (Executing on YARN cluster with App id
application_1606875286859_0001)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. llap SUCCEEDED 1 100
0 0
Reducer 2 .. llap SUCCEEDED 2 200
0 5
Reducer 3 .. llap SUCCEEDED 2 200
0 9
--
VERTICES: 03/03 [==>>] 100% ELAPSED TIME: 28.61 s
--
INFO : Status: DAG finished successfully in 18.72 seconds
INFO :
INFO : Query Execution Summary
INFO :
--
INFO : OPERATIONDURATION
INFO :
--
INFO : Compile Query 14.06s
INFO : Prepare Plan0.17s
INFO : Get Query Coordinator (AM) 0.14s
INFO : Submit Plan 0.03s
INFO : Start DAG 0.05s
INFO : Run DAG18.72s
INFO :
--
{noformat}
It would be good to stop launching the job, when the condition is not valid in
the query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24443) Optimise VectorSerializeRow for primitives
Rajesh Balamohan created HIVE-24443: --- Summary: Optimise VectorSerializeRow for primitives Key: HIVE-24443 URL: https://issues.apache.org/jira/browse/HIVE-24443 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2020-11-30 at 9.39.31 AM.png !Screenshot 2020-11-30 at 9.39.31 AM.png|width=826,height=477! One option could be to have specific serializer embedded in "Field" object within VectorSerializeRow. This would avoid unwanted switching every time. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSerializeRow.java#L63] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24416) Optimise HiveCharWritable::getStrippedValue
Rajesh Balamohan created HIVE-24416: --- Summary: Optimise HiveCharWritable::getStrippedValue Key: HIVE-24416 URL: https://issues.apache.org/jira/browse/HIVE-24416 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2020-11-24 at 10.36.51 AM.png !Screenshot 2020-11-24 at 10.36.51 AM.png|width=1126,height=404! Multiple text conversions can be avoided in HiveCharWritable::getStrippedValue for removing trailing spaces. https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/io/HiveCharWritable.java#L87 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24409) Use LazyBinarySerDe2 in PlanUtils::getReduceValueTableDesc
Rajesh Balamohan created HIVE-24409: --- Summary: Use LazyBinarySerDe2 in PlanUtils::getReduceValueTableDesc Key: HIVE-24409 URL: https://issues.apache.org/jira/browse/HIVE-24409 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Attachments: Screenshot 2020-11-23 at 10.52.49 AM.png !Screenshot 2020-11-23 at 10.52.49 AM.png|width=858,height=493! Lines of interest: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java#L535] (non-vectorized path due to stats) [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java#L581] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24368) Optimise AcidUtils::getAcidFilesForStats for ACID tables
Rajesh Balamohan created HIVE-24368:
---
Summary: Optimise AcidUtils::getAcidFilesForStats for ACID tables
Key: HIVE-24368
URL: https://issues.apache.org/jira/browse/HIVE-24368
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
After insert, hive gathers statistics for ACID table and that becomes expensive
over time, due to number of delta folders and scanning .
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L2648]
{noformat}
public static List getAcidFilesForStats(
Table table, Path dir, Configuration jc, FileSystem fs) throws
IOException {
...
Directory acidInfo = AcidUtils.getAcidState(fs, dir, jc, idList, null,
false, hdfsDirSnapshots);
...
..+ other calls
...
}
{noformat}
Runtime keeps increasing as more deltas are generated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24367) Explore whether HiveAlterHandler::alterTable can be optimised for non-partitioned tablesInbox
Rajesh Balamohan created HIVE-24367:
---
Summary: Explore whether HiveAlterHandler::alterTable can be
optimised for non-partitioned tablesInbox
Key: HIVE-24367
URL: https://issues.apache.org/jira/browse/HIVE-24367
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
{color:#22}Writing lots of delta in non-partitioned table creates runtime
issues, when lot of delta folders are present.{color}
{color:#22} {color}
{color:#22}Following code in HiveAlterHandler is invoked for every insert
operation. It computes {{{color}
{color:#22}updateTableStatsSlow}} for every insert causing runtime
delays.{color}
{color:#22} {color}
{noformat}
if (MetaStoreUtils.requireCalStats(null, null, newt, environmentContext) &&
!isPartitionedTable) {
Database db = msdb.getDatabase(catName, newDbName);
assert(isReplicated == HiveMetaStore.HMSHandler.isDbReplicationTarget(db));
// Update table stats. For partitioned table, we update stats in
alterPartition()
MetaStoreUtils.updateTableStatsSlow(db, newt, wh, false, true,
environmentContext);
}
{noformat}
{color:#22}It would be good to explore whether only the newly added delta
can be listed for computing stats. This would avoid huge listing call during
stats collection.{color}
{color:#22}e.g queries to repro{color}
{noformat}
CREATE TABLE IF NOT EXISTS test (name String, value int);
INSERT INTO test VALUES('K1',1);
INSERT INTO test VALUES('K2',2);
..
..
..
INSERT INTO test VALUES('K2',2)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24313) Optimise stats collection for file sizes on cloud storage
Rajesh Balamohan created HIVE-24313:
---
Summary: Optimise stats collection for file sizes on cloud storage
Key: HIVE-24313
URL: https://issues.apache.org/jira/browse/HIVE-24313
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When stats information is not present (e.g external table), RelOptHiveTable
computes basic stats at runtime.
Following is the codepath.
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java#L598]
{code:java}
Statistics stats = StatsUtils.collectStatistics(hiveConf, partitionList,
hiveTblMetadata, hiveNonPartitionCols,
nonPartColNamesThatRqrStats, colStatsCached,
nonPartColNamesThatRqrStats, true);
{code}
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L322]
{code:java}
for (Partition p : partList.getNotDeniedPartns()) {
BasicStats basicStats = basicStatsFactory.build(Partish.buildFor(table,
p));
partStats.add(basicStats);
}
{code}
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStats.java#L205]
{code:java}
try {
ds = getFileSizeForPath(path);
} catch (IOException e) {
ds = 0L;
}
{code}
For a table & query with large number of partitions, this takes long time to
compute statistics and increases compilation time. It would be good to fix it
with "ForkJoinPool" (
partList.getNotDeniedPartns().parallelStream().forEach((p) )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
