[jira] [Updated] (HIVE-7428) OrcSplit fails to account for columnar projections in its size estimates
[ https://issues.apache.org/jira/browse/HIVE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7428: -- Affects Version/s: 2.0.0 > OrcSplit fails to account for columnar projections in its size estimates > > > Key: HIVE-7428 > URL: https://issues.apache.org/jira/browse/HIVE-7428 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-7428.1.patch, HIVE-7428.2.patch > > > Currently, ORC generates splits based on stripe offset + stripe length. > This means that the splits for all columnar projections are exactly the same > size, despite reading the footer which gives the estimated sizes for each > column. > This is a hold-out from FileSplit which uses getLen() as the I/O cost of > reading a file in a map-task. > RCFile didn't have a footer with column statistics information, but for ORC > this would be extremely useful to reduce task overheads when processing > extremely wide tables with highly selective column projections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7428) OrcSplit fails to account for columnar projections in its size estimates
[ https://issues.apache.org/jira/browse/HIVE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7428: -- Affects Version/s: 1.3.0 1.2.0 > OrcSplit fails to account for columnar projections in its size estimates > > > Key: HIVE-7428 > URL: https://issues.apache.org/jira/browse/HIVE-7428 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-7428.1.patch, HIVE-7428.2.patch > > > Currently, ORC generates splits based on stripe offset + stripe length. > This means that the splits for all columnar projections are exactly the same > size, despite reading the footer which gives the estimated sizes for each > column. > This is a hold-out from FileSplit which uses getLen() as the I/O cost of > reading a file in a map-task. > RCFile didn't have a footer with column statistics information, but for ORC > this would be extremely useful to reduce task overheads when processing > extremely wide tables with highly selective column projections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7428) OrcSplit fails to account for columnar projections in its size estimates
[ https://issues.apache.org/jira/browse/HIVE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7428: -- Attachment: HIVE-7428.2.patch > OrcSplit fails to account for columnar projections in its size estimates > > > Key: HIVE-7428 > URL: https://issues.apache.org/jira/browse/HIVE-7428 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-7428.1.patch, HIVE-7428.2.patch > > > Currently, ORC generates splits based on stripe offset + stripe length. > This means that the splits for all columnar projections are exactly the same > size, despite reading the footer which gives the estimated sizes for each > column. > This is a hold-out from FileSplit which uses getLen() as the I/O cost of > reading a file in a map-task. > RCFile didn't have a footer with column statistics information, but for ORC > this would be extremely useful to reduce task overheads when processing > extremely wide tables with highly selective column projections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7428) OrcSplit fails to account for columnar projections in its size estimates
[ https://issues.apache.org/jira/browse/HIVE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-7428: Attachment: HIVE-7428.1.patch Same trunk patch from HIVE-10397. I might need to rebase it to master. > OrcSplit fails to account for columnar projections in its size estimates > > > Key: HIVE-7428 > URL: https://issues.apache.org/jira/browse/HIVE-7428 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-7428.1.patch > > > Currently, ORC generates splits based on stripe offset + stripe length. > This means that the splits for all columnar projections are exactly the same > size, despite reading the footer which gives the estimated sizes for each > column. > This is a hold-out from FileSplit which uses getLen() as the I/O cost of > reading a file in a map-task. > RCFile didn't have a footer with column statistics information, but for ORC > this would be extremely useful to reduce task overheads when processing > extremely wide tables with highly selective column projections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7428) OrcSplit fails to account for columnar projections in its size estimates
[ https://issues.apache.org/jira/browse/HIVE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7428: -- Assignee: Prasanth Jayachandran (was: Gopal V) > OrcSplit fails to account for columnar projections in its size estimates > > > Key: HIVE-7428 > URL: https://issues.apache.org/jira/browse/HIVE-7428 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Prasanth Jayachandran > > Currently, ORC generates splits based on stripe offset + stripe length. > This means that the splits for all columnar projections are exactly the same > size, despite reading the footer which gives the estimated sizes for each > column. > This is a hold-out from FileSplit which uses getLen() as the I/O cost of > reading a file in a map-task. > RCFile didn't have a footer with column statistics information, but for ORC > this would be extremely useful to reduce task overheads when processing > extremely wide tables with highly selective column projections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
