[
https://issues.apache.org/jira/browse/HIVE-21935?focusedWorklogId=594630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-594630
]
ASF GitHub Bot logged work on HIVE-21935:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 11/May/21 16:51
Start Date: 11/May/21 16:51
Worklog Time Spent: 10m
Work Description: abstractdog commented on pull request #2242:
URL: https://github.com/apache/hive/pull/2242#issuecomment-838819612
the change makes sense to me, could you confirm that this use-case is
covered by a q.test?
I was looking for qtests, I found similar ones but only with struct type
(StructColumnVector, which doesn’t inherit MultiValuedColumnVector)
e.g.:
schema_evol_text_vecrow_part_all_complex.q
I think q.test for this use should have the following properties:
1. use text format file for reading
2. force vector/row serde deserialize:
```
SET hive.vectorized.use.vectorized.input.format=false;
SET hive.vectorized.use.vector.serde.deserialize=true;
SET hive.vectorized.use.row.serde.deserialize=true;
```
3. use map/list type
4. work on small batches or a large amount of rows <1024 (to force batch
reuse = reuse of deserializerBatch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 594630)
Time Spent: 0.5h (was: 20m)
> Hive Vectorization : degraded performance with vectorize UDF
> --------------------------------------------------------------
>
> Key: HIVE-21935
> URL: https://issues.apache.org/jira/browse/HIVE-21935
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
> Affects Versions: 3.1.1
> Environment: Hive-3, JDK-8
> Reporter: Rajkumar Singh
> Assignee: Mustafa İman
> Priority: Major
> Labels: performance, pull-request-available
> Attachments: CustomSplit-1.0-SNAPSHOT.jar
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> with vectorization turned on and hive.vectorized.adaptor.usage.mode=all we
> were seeing severe performance degradation. looking at the task jstacks it
> seems that it is running the code which vectorizes UDF and stuck in some loop.
> {code:java}
> jstack -l 14954 | grep 0x3af0 -A20
> "TezChild" #15 daemon prio=5 os_prio=0 tid=0x00007f157538d800 nid=0x3af0
> runnable [0x00007f1547581000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorAssignRow.assignRowColumn(VectorAssignRow.java:573)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorAssignRow.assignRowColumn(VectorAssignRow.java:350)
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:150)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:271)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.ListIndexColScalar.evaluate(ListIndexColScalar.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:889)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> [yarn@hdp32b ~]$ jstack -l 14954 | grep 0x3af0 -A20
> "TezChild" #15 daemon prio=5 os_prio=0 tid=0x00007f157538d800 nid=0x3af0
> runnable [0x00007f1547581000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.ensureSize(BytesColumnVector.java:554)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorAssignRow.assignRowColumn(VectorAssignRow.java:570)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorAssignRow.assignRowColumn(VectorAssignRow.java:350)
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:150)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:271)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.ListIndexColScalar.evaluate(ListIndexColScalar.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:889)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> {code}
> after setting the hive.vectorized.adaptor.usage.mode=none query did complete
> much faster.
> Steps To Reproduce:
> 1. Create Table:
> {code}
> +----------------------------------------------------+
> | createtab_stmt |
> +----------------------------------------------------+
> | CREATE EXTERNAL TABLE `splittestloc`( |
> | `id` int, |
> | `value` string) |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
> | WITH SERDEPROPERTIES ( |
> | 'field.delim'=',', |
> | 'serialization.format'=',') |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.mapred.TextInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
> | LOCATION |
> | 'hdfs://hdp31a.hdp.local:8020/tmp/splittableloc' |
> | TBLPROPERTIES ( |
> | 'bucketing_version'='2', |
> | 'transient_lastDdlTime'='1561482451') |
> +----------------------------------------------------+
> {code}
> 2. Sample data: table has some 40M rows and sample data is generated using
> following script.
> {code}
> for i in {1..40000000} ; do echo $i,"start#mid#"$i >> data.log ; done
> {code}
> 3. I believe this should be reproducible with hive generic split but I am
> attaching the custom UDF to split the string.
> 4. create a function
> {code}
> add jar /tmp/CustomSplit-1.0-SNAPSHOT.jar;
> create temporary function mysplit as 'com.rajkrrsingh.split.test.CustomSplit'
> {code}
> 5. run the following query which will reproduce the issue if vectorization
> turned on.
> {code}
> create temporary table tmp2 as select id,mysplit(value,"#")[2] from
> splittestloc
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
