[
https://issues.apache.org/jira/browse/DRILL-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507218#comment-16507218
]
ASF GitHub Bot commented on DRILL-6340:
---------------------------------------
bitblender commented on issue #1302: DRILL-6340: Output Batch Control in
Project using the RecordBatchSizer
URL: https://github.com/apache/drill/pull/1302#issuecomment-396010364
I don't think my tests are contributing a whole lot to the slowdown. With
this change, Project will split large batches in tests(like 45 M batches in
TestLargeFileCompilation) to ~16M batches. This somehow seems to be slowing
down operators like Sort and TopN a fair bit compared to the no-split runs.
This needs further investigation to see if these slow-down are un-avoidable .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Output Batch Control in Project using the RecordBatchSizer
> ----------------------------------------------------------
>
> Key: DRILL-6340
> URL: https://issues.apache.org/jira/browse/DRILL-6340
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Reporter: Karthikeyan Manivannan
> Assignee: Karthikeyan Manivannan
> Priority: Major
> Fix For: 1.14.0
>
>
> This bug is for tracking the changes required to implement Output Batch
> Sizing in Project using the RecordBatchSizer. The challenge in doing this
> mainly lies in dealing with expressions that produce variable-length columns.
> The following doc talks about some of the design approaches for dealing with
> such variable-length columns.
> [https://docs.google.com/document/d/1h0WsQsen6xqqAyyYSrtiAniQpVZGmQNQqC1I2DJaxAA/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
