[
https://issues.apache.org/jira/browse/DRILL-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943659#comment-15943659
]
Jinfeng Ni commented on DRILL-5384:
-----------------------------------
Tried your example data:
{code}
select t.`order`.id from dfs.`/tmp/2.json` t order by t.customer.id;
00-00 Screen
00-01 Project(EXPR$0=[$0])
00-02 SelectionVectorRemover
00-03 Sort(sort0=[$1], dir0=[ASC])
00-04 Project(EXPR$0=[ITEM($0, 'id')], EXPR$1=[ITEM($1, 'id')])
00-05 Scan(groupscan=[EasyGroupScan
[selectionRoot=file:/tmp/2.json, numFiles=1, columns=[`order`.`id`,
`customer`.`id`], files=[file:/tmp/2.json]]])
{code}
Here is the generated code for the project operator. We can see the evaluation
part is actually empty.
{code}
Compiling (source size=569 B):
1:
2: package org.apache.drill.exec.test.generated;
3:
4: import org.apache.drill.exec.exception.SchemaChangeException;
5: import org.apache.drill.exec.ops.FragmentContext;
6: import org.apache.drill.exec.record.RecordBatch;
7:
8: public class ProjectorGen0 {
9:
10:
11: public void doEval(int inIndex, int outIndex)
12: throws SchemaChangeException
13: {
14: }
15:
16: public void doSetup(FragmentContext context, RecordBatch incoming,
RecordBatch outgoing)
17: throws SchemaChangeException
18: {
19: }
20:
21: public void __DRILL_INIT__()
22: throws SchemaChangeException
23: {
24: }
25:
26: }
{code}
> Sort cannot directly access map members, causes a data copy
> -----------------------------------------------------------
>
> Key: DRILL-5384
> URL: https://issues.apache.org/jira/browse/DRILL-5384
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Priority: Minor
>
> Suppose we have a JSON structure for "orders" like this:
> {code}
> { customer: { id: 10, name: "fred" },
> order: { id: 20, product: "Frammis 1000" } }
> {code}
> Suppose I want to sort by customer.id. Today, Drill will project customer.id
> up to the top level as a temporary, hidden field. Drill will copy the data
> from the customer.id vector to this new temporary field. Drill then sorts on
> the temporary column, and uses another project to remove the columns.
> Clearly, this work, but it has a cost:
> * Extra two project operators.
> * Extra memory copy.
> * Sort must buffer both the original and copied data. This can double memory
> use in the worst case.
> All of this is done simply to avoid having to reference "customer.id" in the
> sort.
> But, as explained in DRILL-5376, maps are just nested tuples; there is no
> need to copy the data, the data is already right there in a value vector. The
> problem is that Drill's map implementation makes it hard for the generated
> code to get at the "customer.id" vector.
> This ticket asks to allow the sort to work directly with nested scalars to
> avoid the overhead explained above. To do this:
> 1. Fix nested scalar access to allow the generated code to easily access a
> nested scalar.
> 2. Allow a sort key of the form "customer.id".
> 3. Modify the planner to generate such sort keys instead of the dual projects.
> The result will be a leaner, faster sort operation when sorting on scalars
> within a map.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
