[jira] [Commented] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[
https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257584#comment-15257584
]
Hive QA commented on HIVE-13341:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800428/HIVE-13341.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/82/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/82/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-82/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-82/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at eead54c HIVE-13241 : LLAP: Incremental Caching marks some small
chunks as "incomplete CB" (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
Removing llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at eead54c HIVE-13241 : LLAP: Incremental Caching marks some small
chunks as "incomplete CB" (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800428 - PreCommit-HIVE-MASTER-Build
> Stats state is not captured correctly: differentiate load table and create
> table
>
>
> Key: HIVE-13341
> URL: https://issues.apache.org/jira/browse/HIVE-13341
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Statistics
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch,
> HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch,
> HIVE-13341.06.patch, HIVE-13341.07.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[
https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257583#comment-15257583
]
Hive QA commented on HIVE-13536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800571/HIVE-13536.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 9963 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForceLocalityTest1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityUnknownHost
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testNoForceLocalityCounterTest1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testNodeDisabled
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testNodeReEnabled
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testPreemption
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testSimpleLocalAllocation
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testSimpleNoLocalityAllocation
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
[jira] [Assigned] (HIVE-13593) HiveServer2: Better performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holman Lan reassigned HIVE-13593: - Assignee: Holman Lan > HiveServer2: Better performance instrumentation for HIVE-12049 (serializing > thrift ResultSets in tasks) > --- > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Holman Lan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[
https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257487#comment-15257487
]
Hive QA commented on HIVE-13541:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800391/HIVE-13541.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 69 failed/errored test(s), 9939 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-vectorization_10.q-orc_merge5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_leftsemijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dynpart_hashjoin_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_between_columns
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_expressions
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_arithmetic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_pushdown
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
[jira] [Commented] (HIVE-13241) LLAP: Incremental Caching marks some small chunks as "incomplete CB"
[
https://issues.apache.org/jira/browse/HIVE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257479#comment-15257479
]
Lefty Leverenz commented on HIVE-13241:
---
Doc note: This adds *hive.llap.orc.gap.cache* to HiveConf.java, so it needs to
be documented in the LLAP section of Configuration Properties for release 2.1.0
with a crossreference at the beginning of the ORC section.
* [Configuration Properties -- LLAP |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP]
* [Configuration Properties -- ORC File Format |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat]
> LLAP: Incremental Caching marks some small chunks as "incomplete CB"
>
>
> Key: HIVE-13241
> URL: https://issues.apache.org/jira/browse/HIVE-13241
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13241.01.patch, HIVE-13241.patch
>
>
> Run #3 of a query with 1 node still has cache misses.
> {code}
> LLAP IO Summary
> --
> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS ALLOCATION
> USED TOTAL_IO
> --
> Map 111 1116 01.65GB93.61MB 0B
>0B32.72s
> --
> {code}
> {code}
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x1c44401d(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x1c44401d(2)
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x4e51b032(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x4e51b032(2)
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addOneCompressionBuffer(1161)) - Found CB at 1373931,
> chunk length 86587, total 86590, compressed
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addIncompleteCompressionBuffer(1241)) - Replacing
> data range [1373931, 1408408), size: 34474(!) type: direct (and 0 previous
> chunks) with incomplete CB start: 1373931 end: 1408408 in the buffers
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:createRgColumnStreamData(441)) - Getting data for
> column 7 RG 14 stream DATA at 1460521, 319811 index position 0: compressed
> [1626961, 1780332)
> {code}
> {code}
> 2016-03-08T21:05:38,925 INFO
> [IO-Elevator-Thread-7[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.OrcEncodedDataReader (OrcEncodedDataReader.java:readFileData(878)) -
> Disk ranges after disk read (file 5372745, base offset 3): [{start: 18986
> end: 20660 cache buffer: 0x660faf7c(1)}, {start: 20660 end: 35775 cache
> buffer: 0x1dcb1d97(1)}, {start: 318852 end: 422353 cache buffer:
> 0x6c7f9a05(1)}, {start: 1148616 end: 1262468 cache buffer: 0x196e1d41(1)},
> {start: 1262468 end: 1376342 cache buffer: 0x201255f(1)}, {data range
> [1376342, 1410766), size: 34424 type: direct}, {start: 1631359 end: 1714694
> cache buffer: 0x47e3a72d(1)}, {start: 1714694 end: 1785770 cache buffer:
> 0x57dca266(1)}, {start: 4975035 end: 5095215 cache buffer: 0x3e3139c9(1)},
> {start: 5095215 end: 5197863 cache buffer: 0x3511c88d(1)}, {start: 7448387
> end: 7572268 cache buffer: 0x6f11dbcd(1)}, {start: 7572268 end: 7696182 cache
> buffer: 0x5d6c9bdb(1)}, {data range [7696182, 7710537), size: 14355 type:
> direct}, {start: 8235756 end: 8345367 cache buffer: 0x6a241ece(1)}, {start:
> 8345367 end: 8455009 cache buffer: 0x51caf6a7(1)}, {data range [8455009,
> 8497906), size: 42897 type: direct}, {start:
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: HIVE-13598.1.patch > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: (was: HIVE-13598.1.patch) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13241) LLAP: Incremental Caching marks some small chunks as "incomplete CB"
[
https://issues.apache.org/jira/browse/HIVE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-13241:
--
Labels: TODOC2.1 (was: )
> LLAP: Incremental Caching marks some small chunks as "incomplete CB"
>
>
> Key: HIVE-13241
> URL: https://issues.apache.org/jira/browse/HIVE-13241
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13241.01.patch, HIVE-13241.patch
>
>
> Run #3 of a query with 1 node still has cache misses.
> {code}
> LLAP IO Summary
> --
> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS ALLOCATION
> USED TOTAL_IO
> --
> Map 111 1116 01.65GB93.61MB 0B
>0B32.72s
> --
> {code}
> {code}
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x1c44401d(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x1c44401d(2)
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x4e51b032(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x4e51b032(2)
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addOneCompressionBuffer(1161)) - Found CB at 1373931,
> chunk length 86587, total 86590, compressed
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addIncompleteCompressionBuffer(1241)) - Replacing
> data range [1373931, 1408408), size: 34474(!) type: direct (and 0 previous
> chunks) with incomplete CB start: 1373931 end: 1408408 in the buffers
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:createRgColumnStreamData(441)) - Getting data for
> column 7 RG 14 stream DATA at 1460521, 319811 index position 0: compressed
> [1626961, 1780332)
> {code}
> {code}
> 2016-03-08T21:05:38,925 INFO
> [IO-Elevator-Thread-7[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.OrcEncodedDataReader (OrcEncodedDataReader.java:readFileData(878)) -
> Disk ranges after disk read (file 5372745, base offset 3): [{start: 18986
> end: 20660 cache buffer: 0x660faf7c(1)}, {start: 20660 end: 35775 cache
> buffer: 0x1dcb1d97(1)}, {start: 318852 end: 422353 cache buffer:
> 0x6c7f9a05(1)}, {start: 1148616 end: 1262468 cache buffer: 0x196e1d41(1)},
> {start: 1262468 end: 1376342 cache buffer: 0x201255f(1)}, {data range
> [1376342, 1410766), size: 34424 type: direct}, {start: 1631359 end: 1714694
> cache buffer: 0x47e3a72d(1)}, {start: 1714694 end: 1785770 cache buffer:
> 0x57dca266(1)}, {start: 4975035 end: 5095215 cache buffer: 0x3e3139c9(1)},
> {start: 5095215 end: 5197863 cache buffer: 0x3511c88d(1)}, {start: 7448387
> end: 7572268 cache buffer: 0x6f11dbcd(1)}, {start: 7572268 end: 7696182 cache
> buffer: 0x5d6c9bdb(1)}, {data range [7696182, 7710537), size: 14355 type:
> direct}, {start: 8235756 end: 8345367 cache buffer: 0x6a241ece(1)}, {start:
> 8345367 end: 8455009 cache buffer: 0x51caf6a7(1)}, {data range [8455009,
> 8497906), size: 42897 type: direct}, {start: 9035815 end: 9159708 cache
> buffer: 0x306480e0(1)}, {start: 9159708 end: 9283629 cache buffer:
> 0x9ef7774(1)}, {data range [9283629, 9297965), size: 14336 type: direct},
> {start: 9989884 end: 10113731 cache buffer: 0x43f7cae9(1)}, {start: 10113731
> end: 10237589 cache buffer: 0x458e63fe(1)}, {data range [10237589, 10252034),
> size: 14445 type: direct}, {start: 11897896 end: 12021787 cache buffer:
> 0x51f9982f(1)}, {start: 12021787 end: 12145656 cache buffer: 0x23df01b3(1)},
> {data range [12145656, 12160046), size: 14390
[jira] [Commented] (HIVE-13241) LLAP: Incremental Caching marks some small chunks as "incomplete CB"
[
https://issues.apache.org/jira/browse/HIVE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257473#comment-15257473
]
Lefty Leverenz commented on HIVE-13241:
---
[~sershe], you committed without addressing my review comments
(https://reviews.apache.org/r/45062/) and I didn't have time to remind you
because there were only a few hours after the +1.
My suggestions weren't a big deal, but they deserved better treatment. Now
that I've whined in public, I forgive you. ;)
> LLAP: Incremental Caching marks some small chunks as "incomplete CB"
>
>
> Key: HIVE-13241
> URL: https://issues.apache.org/jira/browse/HIVE-13241
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13241.01.patch, HIVE-13241.patch
>
>
> Run #3 of a query with 1 node still has cache misses.
> {code}
> LLAP IO Summary
> --
> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS ALLOCATION
> USED TOTAL_IO
> --
> Map 111 1116 01.65GB93.61MB 0B
>0B32.72s
> --
> {code}
> {code}
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x1c44401d(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x1c44401d(2)
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x4e51b032(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x4e51b032(2)
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addOneCompressionBuffer(1161)) - Found CB at 1373931,
> chunk length 86587, total 86590, compressed
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addIncompleteCompressionBuffer(1241)) - Replacing
> data range [1373931, 1408408), size: 34474(!) type: direct (and 0 previous
> chunks) with incomplete CB start: 1373931 end: 1408408 in the buffers
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:createRgColumnStreamData(441)) - Getting data for
> column 7 RG 14 stream DATA at 1460521, 319811 index position 0: compressed
> [1626961, 1780332)
> {code}
> {code}
> 2016-03-08T21:05:38,925 INFO
> [IO-Elevator-Thread-7[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.OrcEncodedDataReader (OrcEncodedDataReader.java:readFileData(878)) -
> Disk ranges after disk read (file 5372745, base offset 3): [{start: 18986
> end: 20660 cache buffer: 0x660faf7c(1)}, {start: 20660 end: 35775 cache
> buffer: 0x1dcb1d97(1)}, {start: 318852 end: 422353 cache buffer:
> 0x6c7f9a05(1)}, {start: 1148616 end: 1262468 cache buffer: 0x196e1d41(1)},
> {start: 1262468 end: 1376342 cache buffer: 0x201255f(1)}, {data range
> [1376342, 1410766), size: 34424 type: direct}, {start: 1631359 end: 1714694
> cache buffer: 0x47e3a72d(1)}, {start: 1714694 end: 1785770 cache buffer:
> 0x57dca266(1)}, {start: 4975035 end: 5095215 cache buffer: 0x3e3139c9(1)},
> {start: 5095215 end: 5197863 cache buffer: 0x3511c88d(1)}, {start: 7448387
> end: 7572268 cache buffer: 0x6f11dbcd(1)}, {start: 7572268 end: 7696182 cache
> buffer: 0x5d6c9bdb(1)}, {data range [7696182, 7710537), size: 14355 type:
> direct}, {start: 8235756 end: 8345367 cache buffer: 0x6a241ece(1)}, {start:
> 8345367 end: 8455009 cache buffer: 0x51caf6a7(1)}, {data range [8455009,
> 8497906), size: 42897 type: direct}, {start: 9035815 end: 9159708 cache
> buffer: 0x306480e0(1)}, {start: 9159708 end: 9283629 cache buffer:
> 0x9ef7774(1)}, {data range [9283629, 9297965), size: 14336 type: direct},
> {start: 9989884 end: 10113731 cache buffer: 0x43f7cae9(1)}, {start:
[jira] [Commented] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257461#comment-15257461 ] Sergey Shelukhin commented on HIVE-12878: - Went halfway thru the recent diffs and then my head started to hurt. I will finish tomorrow... > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257453#comment-15257453
]
Sergey Shelukhin commented on HIVE-13447:
-
Test failures are unrelated.
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[ https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257442#comment-15257442 ] Lefty Leverenz commented on HIVE-13607: --- Typo in the first patch: drvier. > Change website references to HQL/HiveQL to SQL > -- > > Key: HIVE-13607 > URL: https://issues.apache.org/jira/browse/HIVE-13607 > Project: Hive > Issue Type: Improvement > Components: Website >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13607.patch > > > When it started Hive's SQL dialect was far enough from standard SQL that the > developers called it HQL or HiveQL. > Over the years Hive's SQL dialect has matured. It still has some oddities > but it is explicitly pushing towards SQL 2011 conformance. Calling the > language anything but SQL now is confusing for users. > In addition to changing the website I propose to make changes in the wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13609) Fix UDTFs to allow local fetch task to fetch rows forwarded by GenericUDTF.close()
[ https://issues.apache.org/jira/browse/HIVE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13609: -- Status: Patch Available (was: Open) > Fix UDTFs to allow local fetch task to fetch rows forwarded by > GenericUDTF.close() > -- > > Key: HIVE-13609 > URL: https://issues.apache.org/jira/browse/HIVE-13609 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13609.1.patch > > > From [~ashutoshc]'s comments in HIVE-13586, attempt to fix whatever is > causing the local fetch task to not get the rows forwarded by UDTF close(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13609) Fix UDTFs to allow local fetch task to fetch rows forwarded by GenericUDTF.close()
[ https://issues.apache.org/jira/browse/HIVE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13609: -- Attachment: HIVE-13609.1.patch Not totally sure if this is the right way to do it, but we basically need to close the fetch task's operator before the fetch task is totally done retrieving rows. [~ashutoshc] can you take a look? > Fix UDTFs to allow local fetch task to fetch rows forwarded by > GenericUDTF.close() > -- > > Key: HIVE-13609 > URL: https://issues.apache.org/jira/browse/HIVE-13609 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13609.1.patch > > > From [~ashutoshc]'s comments in HIVE-13586, attempt to fix whatever is > causing the local fetch task to not get the rows forwarded by UDTF close(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257430#comment-15257430
]
Lefty Leverenz commented on HIVE-12637:
---
Doc note: This adds *hive.txn.retryable.sqlex.regex* to HiveConf.java, so it
needs to be documented in the Transactions and Compactor section of
Configuration Properties for releases 1.3.0 and 2.1.0.
It also needs to be added to the Configuration section of the Hive Transactions
doc.
* [Configuration Properties -- Transactions and Compactor |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-TransactionsandCompactor]
* [Hive Transactions -- Configuration -- New Configuration Parameters for
Transactions |
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-NewConfigurationParametersforTransactions]
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Labels: TODOC1.3, TODOC2.1
> Attachments: HIVE-12637.1.patch, HIVE-12637.2.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-12637:
--
Labels: TODOC1.3 TODOC2.1 (was: )
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Labels: TODOC1.3, TODOC2.1
> Attachments: HIVE-12637.1.patch, HIVE-12637.2.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257425#comment-15257425
]
Lefty Leverenz commented on HIVE-12637:
---
Nudge: [~wzheng], please update Fix Version/s.
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Labels: TODOC1.3, TODOC2.1
> Attachments: HIVE-12637.1.patch, HIVE-12637.2.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257411#comment-15257411 ] Sergey Shelukhin commented on HIVE-13463: - +1 > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463-4.patch, HIVE-13463.4.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257406#comment-15257406 ] Sergey Shelukhin commented on HIVE-13097: - Should this be committed now? > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Patch Available (was: Open) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: HIVE-13598.1.patch cc-ing [~ashutoshc] for review. > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257398#comment-15257398 ] Nemon Lou commented on HIVE-13602: -- It's 24581 on my computer. I must have checked the wrong stages from mapreduce job UI. After set hive.optimize.constant.propagation=false; the result is right: INFO : Table tpch_flat_orc_2.q16_cbo_debug2 stats: [numFiles=1, numRows=24581, totalSize=803640, rawDataSize=786232] > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.3.0, 2.0.0, 1.2.2 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Description: Running tpch with factor 2, q16 returns 1,160 rows when CBO is on, while returns 24,581 rows when CBO is off. See attachment for detail . was: Running tpch with factor 2, q16 returns 1,160 rows when CBO is on, while returns 59,616 rows when CBO is off. See attachment for detail . > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.3.0, 2.0.0, 1.2.2 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 24,581 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13241) LLAP: Incremental Caching marks some small chunks as "incomplete CB"
[
https://issues.apache.org/jira/browse/HIVE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13241:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Committed to master after some rebasing.
> LLAP: Incremental Caching marks some small chunks as "incomplete CB"
>
>
> Key: HIVE-13241
> URL: https://issues.apache.org/jira/browse/HIVE-13241
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13241.01.patch, HIVE-13241.patch
>
>
> Run #3 of a query with 1 node still has cache misses.
> {code}
> LLAP IO Summary
> --
> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS ALLOCATION
> USED TOTAL_IO
> --
> Map 111 1116 01.65GB93.61MB 0B
>0B32.72s
> --
> {code}
> {code}
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x1c44401d(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x1c44401d(2)
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x4e51b032(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x4e51b032(2)
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addOneCompressionBuffer(1161)) - Found CB at 1373931,
> chunk length 86587, total 86590, compressed
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addIncompleteCompressionBuffer(1241)) - Replacing
> data range [1373931, 1408408), size: 34474(!) type: direct (and 0 previous
> chunks) with incomplete CB start: 1373931 end: 1408408 in the buffers
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:createRgColumnStreamData(441)) - Getting data for
> column 7 RG 14 stream DATA at 1460521, 319811 index position 0: compressed
> [1626961, 1780332)
> {code}
> {code}
> 2016-03-08T21:05:38,925 INFO
> [IO-Elevator-Thread-7[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.OrcEncodedDataReader (OrcEncodedDataReader.java:readFileData(878)) -
> Disk ranges after disk read (file 5372745, base offset 3): [{start: 18986
> end: 20660 cache buffer: 0x660faf7c(1)}, {start: 20660 end: 35775 cache
> buffer: 0x1dcb1d97(1)}, {start: 318852 end: 422353 cache buffer:
> 0x6c7f9a05(1)}, {start: 1148616 end: 1262468 cache buffer: 0x196e1d41(1)},
> {start: 1262468 end: 1376342 cache buffer: 0x201255f(1)}, {data range
> [1376342, 1410766), size: 34424 type: direct}, {start: 1631359 end: 1714694
> cache buffer: 0x47e3a72d(1)}, {start: 1714694 end: 1785770 cache buffer:
> 0x57dca266(1)}, {start: 4975035 end: 5095215 cache buffer: 0x3e3139c9(1)},
> {start: 5095215 end: 5197863 cache buffer: 0x3511c88d(1)}, {start: 7448387
> end: 7572268 cache buffer: 0x6f11dbcd(1)}, {start: 7572268 end: 7696182 cache
> buffer: 0x5d6c9bdb(1)}, {data range [7696182, 7710537), size: 14355 type:
> direct}, {start: 8235756 end: 8345367 cache buffer: 0x6a241ece(1)}, {start:
> 8345367 end: 8455009 cache buffer: 0x51caf6a7(1)}, {data range [8455009,
> 8497906), size: 42897 type: direct}, {start: 9035815 end: 9159708 cache
> buffer: 0x306480e0(1)}, {start: 9159708 end: 9283629 cache buffer:
> 0x9ef7774(1)}, {data range [9283629, 9297965), size: 14336 type: direct},
> {start: 9989884 end: 10113731 cache buffer: 0x43f7cae9(1)}, {start: 10113731
> end: 10237589 cache buffer: 0x458e63fe(1)}, {data range [10237589, 10252034),
> size: 14445 type: direct}, {start: 11897896 end: 12021787 cache buffer:
> 0x51f9982f(1)}, {start: 12021787 end:
[jira] [Commented] (HIVE-12439) CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements
[ https://issues.apache.org/jira/browse/HIVE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257385#comment-15257385 ] Lefty Leverenz commented on HIVE-12439: --- Doc done: [~sladymon] documented the new configuration parameters in the MetaStore section (right after *hive.metastore.try.direct.sql.ddl*) with help from [~sushanth]. Thanks, Shannon and Sushanth! I optimistically removed the TODOC1.3 and TODOC2.1 labels although review comments are still welcome. * [hive.direct.sql.max.query.length | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.direct.sql.max.query.length] * [hive.direct.sql.max.elements.in.clause | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.direct.sql.max.elements.in.clause] * [hive.direct.sql.max.elements.values.clause | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.direct.sql.max.elements.values.clause] > CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements > -- > > Key: HIVE-12439 > URL: https://issues.apache.org/jira/browse/HIVE-12439 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-12439.1.patch, HIVE-12439.2.patch, > HIVE-12439.3.patch > > > # add a safeguard to make sure IN clause is not too large; break up by txn id > to delete from TXN_COMPONENTS where tc_txnid in ... > # TxnHandler. openTxns() - use 1 insert with many rows in values() clause, > rather than 1 DB roundtrip per row -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12439) CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements
[ https://issues.apache.org/jira/browse/HIVE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12439: -- Labels: (was: TODOC1.3 TODOC2.1) > CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements > -- > > Key: HIVE-12439 > URL: https://issues.apache.org/jira/browse/HIVE-12439 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-12439.1.patch, HIVE-12439.2.patch, > HIVE-12439.3.patch > > > # add a safeguard to make sure IN clause is not too large; break up by txn id > to delete from TXN_COMPONENTS where tc_txnid in ... > # TxnHandler. openTxns() - use 1 insert with many rows in values() clause, > rather than 1 DB roundtrip per row -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13445) LLAP: token should encode application and cluster ids
[ https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13445: Attachment: HIVE-13445.04.patch Renamed the secret to ID, since we don't know if it will be secret (probably not). Added the tests and addressed some other RB feedback. Also I noticed clusterId wasn't verified, so I added that. > LLAP: token should encode application and cluster ids > - > > Key: HIVE-13445 > URL: https://issues.apache.org/jira/browse/HIVE-13445 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch, > HIVE-13445.03.patch, HIVE-13445.04.patch, HIVE-13445.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12837: - Attachment: HIVE-12837.5.patch I don't think test failures are related. But to make sure, I cloned patch 4 to patch 5, for another round of precommit test. > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch, HIVE-12837.5.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13159: -- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13159: -- Attachment: HIVE-13159.patch This patch provides a NoPoolConnectionPool object that creates a new connection each time. > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-13159: - Assignee: Alan Gates > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Alan Gates > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13249: - Attachment: HIVE-13249.4.patch patch 4 for test. > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13565) thrift change
[
https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257306#comment-15257306
]
Hive QA commented on HIVE-13565:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800364/HIVE-13565.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 176 failed/errored test(s), 9961 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_invalidate_column_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_tbl_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_part_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl_dp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_quoting
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_confirm_initial_tbl_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fouter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_parse
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries_with_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit_global_optimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull
[jira] [Commented] (HIVE-13458) Heartbeater doesn't fail query when heartbeat fails
[ https://issues.apache.org/jira/browse/HIVE-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257301#comment-15257301 ] Wei Zheng commented on HIVE-13458: -- Test failures are not related. But I will upload patch 5, which is just a rename from patch 4, to make sure. > Heartbeater doesn't fail query when heartbeat fails > --- > > Key: HIVE-13458 > URL: https://issues.apache.org/jira/browse/HIVE-13458 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13458.1.patch, HIVE-13458.2.patch, > HIVE-13458.3.patch, HIVE-13458.4.patch, HIVE-13458.5.patch > > > When a heartbeat fails to locate a lock, it should fail the current query. > That doesn't happen, which is a bug. > Another thing is, we need to make sure stopHeartbeat really stops the > heartbeat, i.e. no additional heartbeat will be sent, since that will break > the assumption and cause the query to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13458) Heartbeater doesn't fail query when heartbeat fails
[ https://issues.apache.org/jira/browse/HIVE-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13458: - Attachment: HIVE-13458.5.patch > Heartbeater doesn't fail query when heartbeat fails > --- > > Key: HIVE-13458 > URL: https://issues.apache.org/jira/browse/HIVE-13458 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13458.1.patch, HIVE-13458.2.patch, > HIVE-13458.3.patch, HIVE-13458.4.patch, HIVE-13458.5.patch > > > When a heartbeat fails to locate a lock, it should fail the current query. > That doesn't happen, which is a bug. > Another thing is, we need to make sure stopHeartbeat really stops the > heartbeat, i.e. no additional heartbeat will be sent, since that will break > the assumption and cause the query to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13429) Tool to remove dangling scratch dir
[ https://issues.apache.org/jira/browse/HIVE-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257271#comment-15257271 ] Shannon Ladymon commented on HIVE-13429: Thanks for the edits, [~daijy]. I added the information about multi-user environments not being an option for *hive.start.cleanup.scratchdir* to the wiki as well. > Tool to remove dangling scratch dir > --- > > Key: HIVE-13429 > URL: https://issues.apache.org/jira/browse/HIVE-13429 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13429.1.patch, HIVE-13429.2.patch, > HIVE-13429.3.patch, HIVE-13429.4.patch, HIVE-13429.5.patch, > HIVE-13429.branch-1.patch > > > We have seen in some cases, user will leave the scratch dir behind, and > eventually eat out hdfs storage. This could happen when vm restarts and leave > no chance for Hive to run shutdown hook. This is applicable for both HiveCli > and HiveServer2. Here we provide an external tool to clear dead scratch dir > as needed. > We need a way to identify which scratch dir is in use. We will rely on HDFS > write lock for that. Here is how HDFS write lock works: > 1. A HDFS client open HDFS file for write and only close at the time of > shutdown > 2. Cleanup process can try to open HDFS file for write. If the client holding > this file is still running, we will get exception. Otherwise, we know the > client is dead > 3. If the HDFS client dies without closing the HDFS file, NN will reclaim the > lease after 10 min, ie, the HDFS file hold by the dead client is writable > again after 10 min > So here is how we remove dangling scratch directory in Hive: > 1. HiveCli/HiveServer2 opens a well-named lock file in scratch directory and > only close it when we about to drop scratch directory > 2. A command line tool cleardanglingscratchdir will check every scratch > directory and try open the lock file for write. If it does not get exception, > meaning the owner is dead and we can safely remove the scratch directory > 3. The 10 min window means it is possible a HiveCli/HiveServer2 is dead but > we still cannot reclaim the scratch directory for another 10 min. But this > should be tolerable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13429) Tool to remove dangling scratch dir
[ https://issues.apache.org/jira/browse/HIVE-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shannon Ladymon updated HIVE-13429: --- Labels: (was: TODOC1.3 TODOC2.1) > Tool to remove dangling scratch dir > --- > > Key: HIVE-13429 > URL: https://issues.apache.org/jira/browse/HIVE-13429 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13429.1.patch, HIVE-13429.2.patch, > HIVE-13429.3.patch, HIVE-13429.4.patch, HIVE-13429.5.patch, > HIVE-13429.branch-1.patch > > > We have seen in some cases, user will leave the scratch dir behind, and > eventually eat out hdfs storage. This could happen when vm restarts and leave > no chance for Hive to run shutdown hook. This is applicable for both HiveCli > and HiveServer2. Here we provide an external tool to clear dead scratch dir > as needed. > We need a way to identify which scratch dir is in use. We will rely on HDFS > write lock for that. Here is how HDFS write lock works: > 1. A HDFS client open HDFS file for write and only close at the time of > shutdown > 2. Cleanup process can try to open HDFS file for write. If the client holding > this file is still running, we will get exception. Otherwise, we know the > client is dead > 3. If the HDFS client dies without closing the HDFS file, NN will reclaim the > lease after 10 min, ie, the HDFS file hold by the dead client is writable > again after 10 min > So here is how we remove dangling scratch directory in Hive: > 1. HiveCli/HiveServer2 opens a well-named lock file in scratch directory and > only close it when we about to drop scratch directory > 2. A command line tool cleardanglingscratchdir will check every scratch > directory and try open the lock file for write. If it does not get exception, > meaning the owner is dead and we can safely remove the scratch directory > 3. The 10 min window means it is possible a HiveCli/HiveServer2 is dead but > we still cannot reclaim the scratch directory for another 10 min. But this > should be tolerable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257267#comment-15257267 ] Wei Zheng commented on HIVE-11793: -- I ran TestDbTxnManager locally and nothing fails. Other failures don't seem related. [~ekoifman] Can you take another look? > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257234#comment-15257234
]
Pengcheng Xiong commented on HIVE-13602:
[~nemon], I tried the following query (I think it should be the same as yours)
{code}
select p_brand,p_type, p_size,count(distinct ps_suppkey) as supplier_cnt
from partsupp, part
where p_partkey = ps_partkey and p_brand <> 'Brand#34' and p_type not like
'ECONOMY BRUSHED%' and p_size in (22, 14, 27, 49, 21, 33, 35, 28)
and partsupp.ps_suppkey not in (select s_suppkey from supplier where s_comment
like '%Customer%Complaints%' )
group by p_brand, p_type, p_size
order by supplier_cnt desc, p_brand, p_type, p_size;
{code}
on Postgres. It returns *24585* rows...
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 1.2.2
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: calcite_cbo_bad.out, calcite_cbo_good.out,
> explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out,
> explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4924) Support query timeout for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-4924:
---
Affects Version/s: 2.0.0
> Support query timeout for jdbc2
> ---
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Navis
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.D11787.1.patch,
> HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-4924) Support query timeout for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta reassigned HIVE-4924:
--
Assignee: Vaibhav Gumashta (was: Navis)
> Support query timeout for jdbc2
> ---
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Reporter: Navis
>Assignee: Vaibhav Gumashta
>Priority: Trivial
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.D11787.1.patch,
> HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12634: - Attachment: HIVE-12634.4.patch patch 3 didn't apply to master. Upload patch 4 for test. > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch, > HIVE-12634.3.patch, HIVE-12634.4.patch > > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13608:
-
Status: Patch Available (was: Open)
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13608:
-
Attachment: HIVE-13608.1.patch
cc-ing [~ashutoshc] for review.
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257151#comment-15257151 ] Zach York commented on HIVE-13463: -- None of these test failures appear related to EXIM functionality. > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463-4.patch, HIVE-13463.4.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[
https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257136#comment-15257136
]
Hive QA commented on HIVE-13463:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800351/HIVE-13463.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 9953 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestSSL - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
{noformat}
Test results:
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.09.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-12878.09.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257127#comment-15257127 ] Sergey Shelukhin commented on HIVE-9660: That is pretty much it. There are some more detailed descriptions in the comments. The two complex bits are the integer writers that have their separate caches, so one needs to be aware when accounting for a CB that, even though some RGs might be fully written, their values could still be in the integer writer literals array (or a similar place), and not in this CB. Another is the string writer, which is logically simple (we save index entries as before, only this time we have to make sure when writing stuff out that we maintain a correct set of active RGs for those CB callbacks), but a little bit involved code-wise. I'll look at test failures, I think the last patch was supposed to pass all the tests before rebase, probably some stupid error. > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.patch, HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257124#comment-15257124
]
Lefty Leverenz commented on HIVE-13290:
---
Doc note: The new syntax needs to be documented for release 2.1.0 in the
Create Table section of the DDL doc. A subsection can also be added for usage
notes.
* [DDL -- Create Table |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable]
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-13290:
--
Labels: TODOC2.1 (was: )
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13596) HS2 should refresh UDFs more frequently(?), at least in multi-HS2 case
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13596: Summary: HS2 should refresh UDFs more frequently(?), at least in multi-HS2 case (was: HS2 should refresh UDFs more frequently, at least in multi-HS2 case) > HS2 should refresh UDFs more frequently(?), at least in multi-HS2 case > -- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We could change the code to refresh the udf by name if it's missing, similar > to getting a table or whatever; or we could refresh UDFs when a session is > started in multi-HS2 case, or at some other convenient point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13596) HS2 should refresh UDFs more frequently, at least in multi-HS2 case
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13596: Summary: HS2 should refresh UDFs more frequently, at least in multi-HS2 case (was: HS2 should refresh UDFs for each session, at least in multi-HS2 case) > HS2 should refresh UDFs more frequently, at least in multi-HS2 case > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We could change the code to refresh the udf by name if it's missing, similar > to getting a table or whatever; or we could refresh UDFs when a session is > started in multi-HS2 case, or at some other convenient point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2573) Create per-session function registry
[ https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257105#comment-15257105 ] Sergey Shelukhin commented on HIVE-2573: This patch breaks some scenarios for multi-HS2 deployments, due to the code that talks to metastore being removed from getFunctionNames(). [~navis] is there a reason why this code had to be removed? I filed HIVE-13596 before I knew this patch existed (I was assuming the scenario in HIVE-13596 never worked); restoring the metastore-calling code, possibly with a config flag, could be one way to fix the issue described there. Ideally, the metastore call should only happen if the UDF is not found locally; that can be the next step, for now we can just fix the issue by restoring the old behavior. > Create per-session function registry > - > > Key: HIVE-2573 > URL: https://issues.apache.org/jira/browse/HIVE-2573 > Project: Hive > Issue Type: Improvement > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Minor > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2573.D3231.1.patch, > HIVE-2573.1.patch.txt, HIVE-2573.10.patch.txt, HIVE-2573.11.patch.txt, > HIVE-2573.12.patch.txt, HIVE-2573.13.patch.txt, HIVE-2573.14.patch.txt, > HIVE-2573.15.patch.txt, HIVE-2573.2.patch.txt, HIVE-2573.3.patch.txt, > HIVE-2573.4.patch.txt, HIVE-2573.5.patch, HIVE-2573.6.patch, > HIVE-2573.7.patch, HIVE-2573.8.patch.txt, HIVE-2573.9.patch.txt > > > Currently the function registry is shared resource and could be overrided by > other users when using HiveServer. If per-session function registry is > provided, this situation could be prevented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257066#comment-15257066 ] Gunther Hagleitner commented on HIVE-13343: --- The map join list only gets cleaned when the task can run in llap, but should be cleared always - or better yet, initialized each time we run the decider. I think it's working right now, because the compiler initializes a new decider each time a query is compiled. Also, two of the log statements are not very helpful, I think. They repeat stuff that's already logged (mode) or tell you something obvious (reducer not running when mode=map). > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[ https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13607: -- Status: Patch Available (was: Open) NO PRECOMMIT TESTS > Change website references to HQL/HiveQL to SQL > -- > > Key: HIVE-13607 > URL: https://issues.apache.org/jira/browse/HIVE-13607 > Project: Hive > Issue Type: Improvement > Components: Website >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13607.patch > > > When it started Hive's SQL dialect was far enough from standard SQL that the > developers called it HQL or HiveQL. > Over the years Hive's SQL dialect has matured. It still has some oddities > but it is explicitly pushing towards SQL 2011 conformance. Calling the > language anything but SQL now is confusing for users. > In addition to changing the website I propose to make changes in the wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257060#comment-15257060
]
Chaoyu Tang commented on HIVE-13588:
These failed tests are not related the patch. [~ychena] Could you help review
the patch? Thanks
> NPE is thrown from MapredLocalTask.executeInChildVM
> ---
>
> Key: HIVE-13588
> URL: https://issues.apache.org/jira/browse/HIVE-13588
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13588.patch, HIVE-13588.patch
>
>
> NPE was thrown out from MapredLocalTask.executeInChildVM in running some
> queries with CLI, see error below:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:321)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:148)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1868)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1595)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1346)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1117)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1105)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[?:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_45]
> {code}
> It is because the operationLog is only applicable to HS2 but CLI, therefore
> it might not be set (null)
> It is related to HIVE-13183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[ https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13607: -- Attachment: HIVE-13607.patch > Change website references to HQL/HiveQL to SQL > -- > > Key: HIVE-13607 > URL: https://issues.apache.org/jira/browse/HIVE-13607 > Project: Hive > Issue Type: Improvement > Components: Website >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13607.patch > > > When it started Hive's SQL dialect was far enough from standard SQL that the > developers called it HQL or HiveQL. > Over the years Hive's SQL dialect has matured. It still has some oddities > but it is explicitly pushing towards SQL 2011 conformance. Calling the > language anything but SQL now is confusing for users. > In addition to changing the website I propose to make changes in the wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257046#comment-15257046 ] Naveen Gangam commented on HIVE-13502: -- Currently, beeline prompts for credentials even when principal (kerberized credentials) is specified in the URL.This fix should resolve this issue too. It will no longer prompt for credentials when principal is specified in the URL. > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257042#comment-15257042 ] Pengcheng Xiong commented on HIVE-13602: [~ashutoshc], enable the CBO, set hive.optimize.constant.propagation=false; will give correct result > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.3.0, 2.0.0, 1.2.2 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13130: - Fix Version/s: 2.1.0 > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, > HIVE-13130.5.patch, HIVE-13130.6.patch, test-results.txt > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Fix Version/s: 2.1.0
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13591: - Fix Version/s: 2.1.0 > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed to master. Thanks [~ashutoshc] for the review.
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13241) LLAP: Incremental Caching marks some small chunks as "incomplete CB"
[
https://issues.apache.org/jira/browse/HIVE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257031#comment-15257031
]
Prasanth Jayachandran commented on HIVE-13241:
--
+1
> LLAP: Incremental Caching marks some small chunks as "incomplete CB"
>
>
> Key: HIVE-13241
> URL: https://issues.apache.org/jira/browse/HIVE-13241
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13241.01.patch, HIVE-13241.patch
>
>
> Run #3 of a query with 1 node still has cache misses.
> {code}
> LLAP IO Summary
> --
> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS ALLOCATION
> USED TOTAL_IO
> --
> Map 111 1116 01.65GB93.61MB 0B
>0B32.72s
> --
> {code}
> {code}
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x1c44401d(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x1c44401d(2)
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(695)) - Locking
> 0x4e51b032(1) due to reuse
> 2016-03-08T21:05:39,417 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:prepareRangesForCompressedRead(701)) - Adding an
> already-uncompressed buffer 0x4e51b032(2)
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addOneCompressionBuffer(1161)) - Found CB at 1373931,
> chunk length 86587, total 86590, compressed
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:addIncompleteCompressionBuffer(1241)) - Replacing
> data range [1373931, 1408408), size: 34474(!) type: direct (and 0 previous
> chunks) with incomplete CB start: 1373931 end: 1408408 in the buffers
> 2016-03-08T21:05:39,418 INFO
> [IO-Elevator-Thread-9[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.EncodedReaderImpl
> (EncodedReaderImpl.java:createRgColumnStreamData(441)) - Getting data for
> column 7 RG 14 stream DATA at 1460521, 319811 index position 0: compressed
> [1626961, 1780332)
> {code}
> {code}
> 2016-03-08T21:05:38,925 INFO
> [IO-Elevator-Thread-7[attempt_1455662455106_2688_3_00_01_0]]:
> encoded.OrcEncodedDataReader (OrcEncodedDataReader.java:readFileData(878)) -
> Disk ranges after disk read (file 5372745, base offset 3): [{start: 18986
> end: 20660 cache buffer: 0x660faf7c(1)}, {start: 20660 end: 35775 cache
> buffer: 0x1dcb1d97(1)}, {start: 318852 end: 422353 cache buffer:
> 0x6c7f9a05(1)}, {start: 1148616 end: 1262468 cache buffer: 0x196e1d41(1)},
> {start: 1262468 end: 1376342 cache buffer: 0x201255f(1)}, {data range
> [1376342, 1410766), size: 34424 type: direct}, {start: 1631359 end: 1714694
> cache buffer: 0x47e3a72d(1)}, {start: 1714694 end: 1785770 cache buffer:
> 0x57dca266(1)}, {start: 4975035 end: 5095215 cache buffer: 0x3e3139c9(1)},
> {start: 5095215 end: 5197863 cache buffer: 0x3511c88d(1)}, {start: 7448387
> end: 7572268 cache buffer: 0x6f11dbcd(1)}, {start: 7572268 end: 7696182 cache
> buffer: 0x5d6c9bdb(1)}, {data range [7696182, 7710537), size: 14355 type:
> direct}, {start: 8235756 end: 8345367 cache buffer: 0x6a241ece(1)}, {start:
> 8345367 end: 8455009 cache buffer: 0x51caf6a7(1)}, {data range [8455009,
> 8497906), size: 42897 type: direct}, {start: 9035815 end: 9159708 cache
> buffer: 0x306480e0(1)}, {start: 9159708 end: 9283629 cache buffer:
> 0x9ef7774(1)}, {data range [9283629, 9297965), size: 14336 type: direct},
> {start: 9989884 end: 10113731 cache buffer: 0x43f7cae9(1)}, {start: 10113731
> end: 10237589 cache buffer: 0x458e63fe(1)}, {data range [10237589, 10252034),
> size: 14445 type: direct}, {start: 11897896 end: 12021787 cache buffer:
> 0x51f9982f(1)}, {start: 12021787 end: 12145656 cache buffer: 0x23df01b3(1)},
> {data range [12145656, 12160046), size: 14390 type: direct}, {start: 12851928
>
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257026#comment-15257026 ] Gunther Hagleitner commented on HIVE-13342: --- I don't think we need to log each time we look at an operator or vertex. At least not on info level. That will be too noisy. But we should add logging each place in the code where we decide LLAP isn't an option. And then with a reason (which I think is missing in this patch still). > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Attachment: test-results.txt
HIVE-13290.final.patch
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Status: Patch Available (was: Open) With the attached patch beeline will now honor URL such as beeline> !connect "jdbc:hive2://localhost:1/default/;user=hive;password=hive" or $ beeline -u "jdbc:hive2://localhost:1/default/;user=hive;password=hive" which is equivalent to $ beeline -u "jdbc:hive2://localhost:1/default/" -n hive -p hive > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256985#comment-15256985
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13290:
--
The only failure that happened locally and was fixed while testing locally :
{code}
Failed tests:
TestHiveDecimalParse.testDecimalType7:107 FAILED: ParseException line 1:32
missing ) at ',' near ',' in column name or primary key or foreign key
line 1:33 cannot recognize input near '4' ')' ')' in column name or primary key
or foreign key
Tests run: 3404, Failures: 1, Errors: 0, Skipped: 2
{code}
I have attached test-results.
Thanks
Hari
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4997) HCatalog doesn't allow multiple input tables
[ https://issues.apache.org/jira/browse/HIVE-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256965#comment-15256965 ] Sushanth Sowmyan commented on HIVE-4997: Using Hive tables directly through mapreduce - HCat is the primary way to do it. As for reading multiple tables simultaneously through HCat, to do your own custom join, HCat does not support that feature yet, but from the activity on this jira, is clearly a capability people want. > HCatalog doesn't allow multiple input tables > > > Key: HIVE-4997 > URL: https://issues.apache.org/jira/browse/HIVE-4997 > Project: Hive > Issue Type: Improvement > Components: HCatalog >Affects Versions: 0.13.0 >Reporter: Daniel Intskirveli > Attachments: HIVE-4997.2.patch, HIVE-4997.3.patch, HIVE-4997.4.patch > > > HCatInputFormat does not allow reading from multiple hive tables in the same > MapReduce job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Attachment: HIVE-13502.patch > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13348) Add Event Nullification support for Replication
[ https://issues.apache.org/jira/browse/HIVE-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256953#comment-15256953 ] Sushanth Sowmyan commented on HIVE-13348: - Sorry, to clarify, the idea is not to nullify the events in the main eventlog itself - we will still maintain those, and they are under the purview of the metastore currently - the idea is that when a program calls HCatClient.getReplicationTasks which exposes an Iterator, where currently, there is a 1:1 map from Event to ReplicationTask, and we should ideally have a many-one. Thus, this filtering would be downstream of the actual collection of events, it would be in-stream for the processing of replication events. Or are you suggesting that even for replication, we should allow the capability to send along noop-replication-tasks as marker tasks for those events which were nullified, so we can have an audit on the destination? That could be done too, and would be performant as well. > Add Event Nullification support for Replication > --- > > Key: HIVE-13348 > URL: https://issues.apache.org/jira/browse/HIVE-13348 > Project: Hive > Issue Type: Sub-task > Components: Import/Export >Reporter: Sushanth Sowmyan > Labels: gsoc2016 > > Replication, as implemented by HIVE-7973 works as follows: > a) For every singly modification to the hive metastore, an event gets > triggered that logs a notification object. > b) Replication tools such as falcon can consume these notification objects as > a HCatReplicationTaskIterator from > HCatClient.getReplicationTasks(lastEventId, maxEvents, dbName, tableName). > c) For each event, we generate statements and distcp requirements for falcon > to export, distcp and import to do the replication (along with requisite > changes to export and import that would allow state management). > The big thing missing from this picture is that while it works, it is pretty > dumb about how it works in that it will exhaustively process every single > event generated, and will try to do the export-distcp-import cycle for all > modifications, irrespective of whether or not that will actually get used at > import time. > We need to build some sort of filtering logic which can process a batch of > events to identify events that will result in effective no-ops, and to > nullify those events from the stream before passing them on. The goal is to > minimize the number of events that the tools like Falcon would actually have > to process. > Examples of cases where event nullification would take place: > a) CREATE-DROP cases: If an object is being created in event#34 that will > eventually get dropped in event#47, then there is no point in replicating > this along. We simply null out both these events, and also, any other event > that references this object between event#34 and event#47. > b) APPEND-APPEND : Some objects are replicated wholesale, which means every > APPEND that occurs would cause a full export of the object in question. At > this point, the prior APPENDS would all be supplanted by the last APPEND. > Thus, we could nullify all the prior such events. > Additional such cases can be inferred by analysis of the Export-Import relay > protocol definition at > https://issues.apache.org/jira/secure/attachment/12725999/EXIMReplicationReplayProtocol.pdf > or by reasoning out various event processing orders possible. > Replication, as implemented by HIVE-7973 is merely a first step for > functional support. This work is needed for replication to be efficient at > all, and thus, usable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256936#comment-15256936
]
Mithun Radhakrishnan commented on HIVE-13509:
-
Sorry for delaying you on this. If I don't have feedback for you tomorrow,
please go ahead and check in as is. I'll trust [~szehon]'s review. :] Thanks
for keeping the default behavior.
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256893#comment-15256893 ] Prasanth Jayachandran commented on HIVE-13536: -- .3 patch address [~sseth] review comment about missing metrics update when removing from pre-emption queue. > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.09.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Status: Patch Available (was: In Progress) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-12878.09.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13445) LLAP: token should encode application and cluster ids
[
https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256866#comment-15256866
]
Hive QA commented on HIVE-13445:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800359/HIVE-13445.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 9947 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/75/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/75/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-75/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800359 - PreCommit-HIVE-MASTER-Build
> LLAP: token should encode application and cluster ids
> -
>
> Key: HIVE-13445
> URL:
[jira] [Updated] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13591: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, Thanks [~ashutoshc] for the review. > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256831#comment-15256831 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-13591: -- Tested the fix locally. This is a metastore script change and the entire unit tests are irrelevant. Will commit this fix into master. Thanks Hari > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13130: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, Thanks [~ashutoshc] for the review. > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, > HIVE-13130.5.patch, HIVE-13130.6.patch, test-results.txt > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[
https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256795#comment-15256795
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13130:
--
Did the tests locally and confirmed that there are no issues. Sample outputs
from the tests:
{code}
Tests run: 3404, Failures: 0, Errors: 0, Skipped: 2
{code}
I am attaching the results as an attachment.
The jira is in commit stage.
Thanks
Hari
> HS2 changes : API calls for retrieving primary keys and foreign keys
> information
> -
>
> Key: HIVE-13130
> URL: https://issues.apache.org/jira/browse/HIVE-13130
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch,
> HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch,
> HIVE-13130.5.patch, HIVE-13130.6.patch, test-results.txt
>
>
> ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes
> getPrimaryKeys and getCrossReference API calls. We need to provide these
> interfaces as part of PK/FK implementation in Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13130: - Attachment: test-results.txt > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, > HIVE-13130.5.patch, HIVE-13130.6.patch, test-results.txt > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12637:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-12637.1.patch, HIVE-12637.2.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256729#comment-15256729
]
Wei Zheng commented on HIVE-12637:
--
Test failures cannot be repro'd locally and don't seem related.
Committed to master and branch-1.
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-12637.1.patch, HIVE-12637.2.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256728#comment-15256728 ] Prasanth Jayachandran commented on HIVE-12159: -- Merging of the readers look good to me. +1. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256720#comment-15256720
]
Chaoyu Tang commented on HIVE-13509:
[~mithun] Do you have a chance to review the new patch revised based on your
requests? Otherwise, I will go ahead to commit the patch since [~szehon] has
already +1 on the fix.
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256712#comment-15256712 ] Gunther Hagleitner commented on HIVE-13568: --- Left some comments on review board. As far as test run goes: You'll have to include .q.out files and you will have to update show_functions.q.out to get clean runs. > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13586) Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion
[ https://issues.apache.org/jira/browse/HIVE-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256664#comment-15256664 ] Ashutosh Chauhan commented on HIVE-13586: - I wonder instead of adding new flag whether can we try to solve issues you outlined above. For udf: * We can check for values of getRequiredFiles() & getRequiredJars() to determine whether they may need those. * For configure, we can manufacture MapRedContext for local task to solve that. For UDTF also its a matter of fixing lifecycle of close() Also, without fixing that we can't enable local fetch task conversion for UDTF even when proposed flag says so. > Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion > > > Key: HIVE-13586 > URL: https://issues.apache.org/jira/browse/HIVE-13586 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13586.1.patch > > > Some UDFs/UDTFs cannot run properly with fetch task conversion. > For UDFs: > - The UDF may be depending on files/archives added to the distributed cache, > which may not be available in the local fetch task. > - The UDF may be depending on GenericUDF.configure() to be called - this is > not called for locally run UDFs. > For UDTFs, one issue is HIVE-11892. > A bit of a workaround, but for such UDFs/UDTFs we can have a flag to indicate > whether the UDF should be allowed to run in a local fetch task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256645#comment-15256645
]
Ashutosh Chauhan commented on HIVE-10176:
-
I like new patch. It makes sense to deal with files containing header/footer
separately then trying to conflate logic with files not containing those. Had a
question on RB. Other than that looks good.
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch,
> HIVE-10176.2.patch, HIVE-10176.3.patch, HIVE-10176.4.patch,
> HIVE-10176.5.patch, HIVE-10176.6.patch, HIVE-10176.7.patch,
> HIVE-10176.8.patch, HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Attachment: HIVE-13536.3.patch > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladyslav Pavlenko updated HIVE-10176:
--
Attachment: HIVE-10176.13.patch
Forget delete file if it in hdfs.
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch,
> HIVE-10176.2.patch, HIVE-10176.3.patch, HIVE-10176.4.patch,
> HIVE-10176.5.patch, HIVE-10176.6.patch, HIVE-10176.7.patch,
> HIVE-10176.8.patch, HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256600#comment-15256600
]
Vladyslav Pavlenko commented on HIVE-10176:
---
[~ashutoshc]
I created one before: https://reviews.apache.org/r/46020. Yes, I agree with you
that my deŃision wasn't optimal. I had faced several problems in the beginning
that is why I decided to create temp file for this issue. I know now how to
solve it. That is why I do not create temp file in the new patch.
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.2.patch, HIVE-10176.3.patch,
> HIVE-10176.4.patch, HIVE-10176.5.patch, HIVE-10176.6.patch,
> HIVE-10176.7.patch, HIVE-10176.8.patch, HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladyslav Pavlenko updated HIVE-10176:
--
Attachment: HIVE-10176.12.patch
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.2.patch,
> HIVE-10176.3.patch, HIVE-10176.4.patch, HIVE-10176.5.patch,
> HIVE-10176.6.patch, HIVE-10176.7.patch, HIVE-10176.8.patch,
> HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13551) Make cleardanglingscratchdir work on Windows
[
https://issues.apache.org/jira/browse/HIVE-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256577#comment-15256577
]
Hive QA commented on HIVE-13551:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800343/HIVE-13551.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 9943 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cbo_windowing.q-tez_join.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-vectorization_10.q-orc_merge5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/74/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/74/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-74/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800343 - PreCommit-HIVE-MASTER-Build
> Make cleardanglingscratchdir work on Windows
>
>
> Key: HIVE-13551
> URL: https://issues.apache.org/jira/browse/HIVE-13551
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee:
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256560#comment-15256560 ] Owen O'Malley commented on HIVE-9660: - I guess my assumption was that you would make a callback from the underlying stream and when a compression buffer finished, you would record a length for any pending RG. > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.patch, HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256543#comment-15256543
]
Ashutosh Chauhan commented on HIVE-13602:
-
[~nemon] If HIVE-11104 is the culprit then turning off constant propagation
should yield correct results. Did you try turning that off by {{set
hive.optimize.constant.propagation=false;}} and running the query? Does that
give correct results?
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 1.2.2
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: calcite_cbo_bad.out, calcite_cbo_good.out,
> explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out,
> explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13602: Affects Version/s: 2.0.0 > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.3.0, 2.0.0, 1.2.2 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256542#comment-15256542 ] Owen O'Malley commented on HIVE-9660: - I don't think we need to bump up the writer version for this change, because the reader can tell if the protobuf has the field or not. WriterVersions are typically reserved for bugs in the writer where the reader needs to work around bugs. Can you give a top level view on how you are approaching adding the lengths? > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.patch, HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4997) HCatalog doesn't allow multiple input tables
[ https://issues.apache.org/jira/browse/HIVE-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256478#comment-15256478 ] Ben Anderson commented on HIVE-4997: any update on this? Am I correct in thinking this is the only way to join hive tables using mapreduce apis directly? > HCatalog doesn't allow multiple input tables > > > Key: HIVE-4997 > URL: https://issues.apache.org/jira/browse/HIVE-4997 > Project: Hive > Issue Type: Improvement > Components: HCatalog >Affects Versions: 0.13.0 >Reporter: Daniel Intskirveli > Attachments: HIVE-4997.2.patch, HIVE-4997.3.patch, HIVE-4997.4.patch > > > HCatInputFormat does not allow reading from multiple hive tables in the same > MapReduce job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
