[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Reattaching to resubmit to jenkins. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Ok, this is rebase to master, which took a fair amount of work because of HIVE-13523. In particular, I've removed the DataReaderFactory and MetadataReaderFactory, because they weren't providing any abstraction. They were baked into ReaderImpl with final fields. I also merged DataReader and MetadataReader so that RecordReaderImpl just has a single connection to HDFS open rather than two. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Ok, LLAP because it is going under the hood to access TreeReaders directly, needs to call ensureSize on the ColumnVector. You can see the delta for this patch on the pull request. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Matthew figured out that I needed to clear the selectedInUse flag in the VectorRowBatch, so I've reverted the change to the tests. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Ok, I updated the patch as Matt suggested and now all of previously failing vectorization qfile tests pass. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch This patch addresses the qfile test failures and Matt's review comments: * replaced long batchSize with int. * deleted confusing TODO comment * renamed and commented the TreeReader.nextBatch method. * fixed the DecimalTreeReader The github pull request has the changes as a separate commit, so you can isolate the new changes. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch This patch adds support for reading complex types to the vectorized ORC reader. Important changes: * RecordReader.nextBatch now requires a non-null batch. This simplifies life in the recursive case when the TreeReader doesn't have enough information to create the appropriate value. * RecordReader.nextBatch now returns a boolean so that it is easy to embed as the loop condition. * ORC's SchemaEvolution can handle adding or removing columns from the end of internal structs. * Created tree readers for union, list, and map types. * TreeReader.nextVector now takes an isNull mask since it can cascade from the outer structs. * A new TreeReader fills in null values for null columns caused by schema evolution. * The TimestampTreeReader does not use the row by row reader internally. * The TreeReaderFactory now gets the file & reader types. * TestVectorOrcFile now uses the vectorized reader exclusively. * The reader passes down the TypeDescription for the file to the RecordReader. * TypeDescription can now be placed into hash tables. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Status: Patch Available (was: Open) > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12159: Fix Version/s: (was: 2.0.0) Removing fixed version (2.0) from Unresolved JIRA in preparation for the release. Please use target version field instead (if not already set) if you think this should be shipped as part of 2.0 > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Oleg Zhurakousky > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
