[jira] [Commented] (HIVE-13831) Error pushing predicates to HBase storage handler
[
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307783#comment-15307783
]
Jesus Camacho Rodriguez commented on HIVE-13831:
{noformat}
Test Name DurationAge
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout 3 min 9
sec 1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 1
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
1 min 6 sec 2
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
5.4 sec 24
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
3.8 sec 28
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 52 sec44
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
1 min 23 sec44
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.7 sec 44
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.36 sec44
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 2 min 7 sec 44
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
66 ms 64
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate11 sec
96
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
12 sec 96
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 34 sec
96
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 47 sec96
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
7.6 sec 96
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
1 min 7 sec 96
{noformat}
> Error pushing predicates to HBase storage handler
> -
>
> Key: HIVE-13831
> URL: https://issues.apache.org/jira/browse/HIVE-13831
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13831.01.patch, HIVE-13831.02.patch,
> HIVE-13831.patch
>
>
> Discovered while working on HIVE-13693.
> There is an error on the predicates that we can push to HBaseStorageHandler.
> In particular, range predicates of the shape {{(bounded, open)}} and {{(open,
> bounded)}} over long or int columns get pushed and return wrong results.
> The problem has to do with the storage order for keys in HBase. Keys are
> sorted lexicographically. Since the byte representation of negative values
> comes after the positive values, open range predicates need special handling
> that we do not have right now.
> Thus, for instance, when we push the predicate {{key > 2}}, we return all
> records with column _key_ greater than 2, plus the records with negative
> values for the column _key_. This problem does not get exposed if a filter is
> kept in the Hive operator tree, but we should not assume the latest.
> This fix avoids pushing this kind of predicates to the storage handler,
> returning them in the _residual_ part of the predicate that cannot be pushed.
> In the future, special handling might be added to support this kind of
> predicates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13831) Error pushing predicates to HBase storage handler
[
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304370#comment-15304370
]
Ashutosh Chauhan commented on HIVE-13831:
-
+1 lets take up pushing down predicates for binary format in a follow-up.
> Error pushing predicates to HBase storage handler
> -
>
> Key: HIVE-13831
> URL: https://issues.apache.org/jira/browse/HIVE-13831
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13831.01.patch, HIVE-13831.02.patch,
> HIVE-13831.patch
>
>
> Discovered while working on HIVE-13693.
> There is an error on the predicates that we can push to HBaseStorageHandler.
> In particular, range predicates of the shape {{(bounded, open)}} and {{(open,
> bounded)}} over long or int columns get pushed and return wrong results.
> The problem has to do with the storage order for keys in HBase. Keys are
> sorted lexicographically. Since the byte representation of negative values
> comes after the positive values, open range predicates need special handling
> that we do not have right now.
> Thus, for instance, when we push the predicate {{key > 2}}, we return all
> records with column _key_ greater than 2, plus the records with negative
> values for the column _key_. This problem does not get exposed if a filter is
> kept in the Hive operator tree, but we should not assume the latest.
> This fix avoids pushing this kind of predicates to the storage handler,
> returning them in the _residual_ part of the predicate that cannot be pushed.
> In the future, special handling might be added to support this kind of
> predicates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13831) Error pushing predicates to HBase storage handler
[
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301941#comment-15301941
]
Jesus Camacho Rodriguez commented on HIVE-13831:
Fails are not related to this patch.
> Error pushing predicates to HBase storage handler
> -
>
> Key: HIVE-13831
> URL: https://issues.apache.org/jira/browse/HIVE-13831
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13831.patch
>
>
> Discovered while working on HIVE-13693.
> There is an error on the predicates that we can push to HBaseStorageHandler.
> In particular, range predicates of the shape {{(bounded, open)}} and {{(open,
> bounded)}} over long or int columns get pushed and return wrong results.
> The problem has to do with the storage order for keys in HBase. Keys are
> sorted lexicographically. Since the byte representation of negative values
> comes after the positive values, open range predicates need special handling
> that we do not have right now.
> Thus, for instance, when we push the predicate {{key > 2}}, we return all
> records with column _key_ greater than 2, plus the records with negative
> values for the column _key_. This problem does not get exposed if a filter is
> kept in the Hive operator tree, but we should not assume the latest.
> This fix avoids pushing this kind of predicates to the storage handler,
> returning them in the _residual_ part of the predicate that cannot be pushed.
> In the future, special handling might be added to support this kind of
> predicates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13831) Error pushing predicates to HBase storage handler
[
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301549#comment-15301549
]
Hive QA commented on HIVE-13831:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805876/HIVE-13831.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 10025 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-join2.q-input17.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_identity_project_remove_skip
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_seqfile
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_uniquejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
{noformat}
Test results:
[jira] [Commented] (HIVE-13831) Error pushing predicates to HBase storage handler
[
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300526#comment-15300526
]
Jesus Camacho Rodriguez commented on HIVE-13831:
[~ashutoshc], forgot to ask before, could you review the patch? Thanks
> Error pushing predicates to HBase storage handler
> -
>
> Key: HIVE-13831
> URL: https://issues.apache.org/jira/browse/HIVE-13831
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13831.patch
>
>
> Discovered while working on HIVE-13693.
> There is an error on the predicates that we can push to HBaseStorageHandler.
> In particular, range predicates of the shape {{(bounded, open)}} and {{(open,
> bounded)}} over long or int columns get pushed and return wrong results.
> The problem has to do with the storage order for keys in HBase. Keys are
> sorted lexicographically. Since the byte representation of negative values
> comes after the positive values, open range predicates need special handling
> that we do not have right now.
> Thus, for instance, when we push the predicate {{key > 2}}, we return all
> records with column _key_ greater than 2, plus the records with negative
> values for the column _key_. This problem does not get exposed if a filter is
> kept in the Hive operator tree, but we should not assume the latest.
> This fix avoids pushing this kind of predicates to the storage handler,
> returning them in the _residual_ part of the predicate that cannot be pushed.
> In the future, special handling might be added to support this kind of
> predicates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
