[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[
https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737000#comment-14737000
]
Hive QA commented on HIVE-11705:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12754759/HIVE-11705.02.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9422 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testTimeOutReaper
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5210/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5210/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12754759 - PreCommit-HIVE-TRUNK-Build
> refactor SARG stripe filtering for ORC into a method
>
>
> Key: HIVE-11705
> URL: https://issues.apache.org/jira/browse/HIVE-11705
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11705.01.patch, HIVE-11705.02.patch,
> HIVE-11705.patch
>
>
> For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny
> item to create it on OrcInputFormat.
> For metastore path, these methods will be called from expression proxy
> similar to current objectstore expr filtering; it will change to have
> serialized sarg and column list to come from request instead of conf;
> includedCols/etc. will also come from request instead of assorted java
> objects.
> The types and stripe stats will need to be extracted from HBase. This is a
> little bit of a problem, since ideally we want to be inside HBase
> filter/coprocessor/ I'd need to take a look to see if this is possible...
> since that filter would need to either deserialize orc, or we would need to
> store types and stats information in some other, non-ORC manner on write. The
> latter is probably a better idea, although it's dangerous because there's no
> sync between this code and ORC itself.
> Meanwhile minimize dependencies for stripe picking to essentials (and conf
> which is easy to remove).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[ https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737462#comment-14737462 ] Prasanth Jayachandran commented on HIVE-11705: -- [~sershe] I am assuming this is a patch for trunk, if so can you please remove the unused methods and put them under a separate jira in hbase-metastore branch. Apart from the unused future methods, the patch looks good to me +1. > refactor SARG stripe filtering for ORC into a method > > > Key: HIVE-11705 > URL: https://issues.apache.org/jira/browse/HIVE-11705 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11705.01.patch, HIVE-11705.02.patch, > HIVE-11705.patch > > > For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny > item to create it on OrcInputFormat. > For metastore path, these methods will be called from expression proxy > similar to current objectstore expr filtering; it will change to have > serialized sarg and column list to come from request instead of conf; > includedCols/etc. will also come from request instead of assorted java > objects. > The types and stripe stats will need to be extracted from HBase. This is a > little bit of a problem, since ideally we want to be inside HBase > filter/coprocessor/ I'd need to take a look to see if this is possible... > since that filter would need to either deserialize orc, or we would need to > store types and stats information in some other, non-ORC manner on write. The > latter is probably a better idea, although it's dangerous because there's no > sync between this code and ORC itself. > Meanwhile minimize dependencies for stripe picking to essentials (and conf > which is easy to remove). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[ https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731707#comment-14731707 ] Prasanth Jayachandran commented on HIVE-11705: -- Left some comments in RB. > refactor SARG stripe filtering for ORC into a method > > > Key: HIVE-11705 > URL: https://issues.apache.org/jira/browse/HIVE-11705 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11705.01.patch, HIVE-11705.patch > > > For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny > item to create it on OrcInputFormat. > For metastore path, these methods will be called from expression proxy > similar to current objectstore expr filtering; it will change to have > serialized sarg and column list to come from request instead of conf; > includedCols/etc. will also come from request instead of assorted java > objects. > The types and stripe stats will need to be extracted from HBase. This is a > little bit of a problem, since ideally we want to be inside HBase > filter/coprocessor/ I'd need to take a look to see if this is possible... > since that filter would need to either deserialize orc, or we would need to > store types and stats information in some other, non-ORC manner on write. The > latter is probably a better idea, although it's dangerous because there's no > sync between this code and ORC itself. > Meanwhile minimize dependencies for stripe picking to essentials (and conf > which is easy to remove). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[
https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727564#comment-14727564
]
Hive QA commented on HIVE-11705:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12753641/HIVE-11705.01.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9391 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5148/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5148/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5148/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12753641 - PreCommit-HIVE-TRUNK-Build
> refactor SARG stripe filtering for ORC into a method
>
>
> Key: HIVE-11705
> URL: https://issues.apache.org/jira/browse/HIVE-11705
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11705.01.patch, HIVE-11705.patch
>
>
> For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny
> item to create it on OrcInputFormat.
> For metastore path, these methods will be called from expression proxy
> similar to current objectstore expr filtering; it will change to have
> serialized sarg and column list to come from request instead of conf;
> includedCols/etc. will also come from request instead of assorted java
> objects.
> The types and stripe stats will need to be extracted from HBase. This is a
> little bit of a problem, since ideally we want to be inside HBase
> filter/coprocessor/ I'd need to take a look to see if this is possible...
> since that filter would need to either deserialize orc, or we would need to
> store types and stats information in some other, non-ORC manner on write. The
> latter is probably a better idea, although it's dangerous because there's no
> sync between this code and ORC itself.
> Meanwhile minimize dependencies for stripe picking to essentials (and conf
> which is easy to remove).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[ https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727743#comment-14727743 ] Sergey Shelukhin commented on HIVE-11705: - test failures are due to HIVE-11689 > refactor SARG stripe filtering for ORC into a method > > > Key: HIVE-11705 > URL: https://issues.apache.org/jira/browse/HIVE-11705 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11705.01.patch, HIVE-11705.patch > > > For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny > item to create it on OrcInputFormat. > For metastore path, these methods will be called from expression proxy > similar to current objectstore expr filtering; it will change to have > serialized sarg and column list to come from request instead of conf; > includedCols/etc. will also come from request instead of assorted java > objects. > The types and stripe stats will need to be extracted from HBase. This is a > little bit of a problem, since ideally we want to be inside HBase > filter/coprocessor/ I'd need to take a look to see if this is possible... > since that filter would need to either deserialize orc, or we would need to > store types and stats information in some other, non-ORC manner on write. The > latter is probably a better idea, although it's dangerous because there's no > sync between this code and ORC itself. > Meanwhile minimize dependencies for stripe picking to essentials (and conf > which is easy to remove). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[
https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725857#comment-14725857
]
Hive QA commented on HIVE-11705:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12753440/HIVE-11705.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9384 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5137/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5137/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5137/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12753440 - PreCommit-HIVE-TRUNK-Build
> refactor SARG stripe filtering for ORC into a method
>
>
> Key: HIVE-11705
> URL: https://issues.apache.org/jira/browse/HIVE-11705
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11705.patch
>
>
> For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny
> item to create it on OrcInputFormat.
> For metastore path, these methods will be called from expression proxy
> similar to current objectstore expr filtering; it will change to have
> serialized sarg and column list to come from request instead of conf;
> includedCols/etc. will also come from request instead of assorted java
> objects.
> The types and stripe stats will need to be extracted from HBase. This is a
> little bit of a problem, since ideally we want to be inside HBase
> filter/coprocessor/ I'd need to take a look to see if this is possible...
> since that filter would need to either deserialize orc, or we would need to
> store types and stats information in some other, non-ORC manner on write. The
> latter is probably a better idea, although it's dangerous because there's no
> sync between this code and ORC itself.
> Meanwhile minimize dependencies for stripe picking to essentials (and conf
> which is easy to remove).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[ https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14726632#comment-14726632 ] Swarnim Kulkarni commented on HIVE-11705: - Left minor comments on RB. Otherwise +1(NB) > refactor SARG stripe filtering for ORC into a method > > > Key: HIVE-11705 > URL: https://issues.apache.org/jira/browse/HIVE-11705 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11705.01.patch, HIVE-11705.patch > > > For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny > item to create it on OrcInputFormat. > For metastore path, these methods will be called from expression proxy > similar to current objectstore expr filtering; it will change to have > serialized sarg and column list to come from request instead of conf; > includedCols/etc. will also come from request instead of assorted java > objects. > The types and stripe stats will need to be extracted from HBase. This is a > little bit of a problem, since ideally we want to be inside HBase > filter/coprocessor/ I'd need to take a look to see if this is possible... > since that filter would need to either deserialize orc, or we would need to > store types and stats information in some other, non-ORC manner on write. The > latter is probably a better idea, although it's dangerous because there's no > sync between this code and ORC itself. > Meanwhile minimize dependencies for stripe picking to essentials (and conf > which is easy to remove). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11705) refactor SARG stripe filtering for ORC into a method
[ https://issues.apache.org/jira/browse/HIVE-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724569#comment-14724569 ] Sergey Shelukhin commented on HIVE-11705: - [~prasanth_j] another small patch :) https://reviews.apache.org/r/37985/ > refactor SARG stripe filtering for ORC into a method > > > Key: HIVE-11705 > URL: https://issues.apache.org/jira/browse/HIVE-11705 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11705.patch > > > For footer cache PPD to metastore, we'd need a method to do the PPD. Tiny > item to create it on OrcInputFormat. > For metastore path, these methods will be called from expression proxy > similar to current objectstore expr filtering; it will change to have > serialized sarg and column list to come from request instead of conf; > includedCols/etc. will also come from request instead of assorted java > objects. > The types and stripe stats will need to be extracted from HBase. This is a > little bit of a problem, since ideally we want to be inside HBase > filter/coprocessor/ I'd need to take a look to see if this is possible... > since that filter would need to either deserialize orc, or we would need to > store types and stats information in some other, non-ORC manner on write. The > latter is probably a better idea, although it's dangerous because there's no > sync between this code and ORC itself. > Meanwhile minimize dependencies for stripe picking to essentials (and conf > which is easy to remove). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
