Re: [PR] DRILL-8375: Support for non-projected complex vectors (drill)

[via GitHub]

cgivre commented on PR #2867:
URL: https://github.com/apache/drill/pull/2867#issuecomment-1879917039

   Once we merge this, we should also rebase 
https://github.com/apache/drill/pull/2515 on the current master and merge that 
as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

UNION and LIST non-projection support in EVF

[Paul Rogers]

Hi All,

Happy New Year!

I dusted off the work to add non-projection support in EVF for the UNION
and LIST types. I believe that only REPEATED LIST is missing.

"Non-projection support" just means that you can read a JSON file that
requires a UNION or LIST vector, and tell EVF to NOT actually project that
vector. That is, if the union column is u and the list is l, and you have
other columns a, b, c, you can write "SELECT a, b, c FROM 'myFile.json'".
JSON wil read all columns, but EVF won't actually create vectors for the
columns u and l that you didn't want.

If you write a format plugin, your code can write to the "dummy" UNION
column writer, blissfully ignorant of the fact that there is no actual
vector underneath: the data just goes into the bit bucket.

The work mostly required fiddling with existing mechanisms. A new test file
demonstrates that the changes work. All the existing tests suggest that the
work didn't break anything.

Reviews greatly appreciated. https://github.com/apache/drill/pull/2867

It would be great to get this into the next release, but doing so isn't
critical, unless Luoc still needs this work.

Thanks,

- Paul