[jira] [Commented] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[
https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263590#comment-15263590
]
Hive QA commented on HIVE-13607:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800871/HIVE-13607.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/114/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/114/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-114/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-114/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 076f365 HIVE-13572 : Redundant setting full file status in
Hive::copyFiles (Rui Li via Ashutosh Chauhan)
+ git clean -f -d
Removing
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig
Removing
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStoreGetMetaConf.java
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 076f365 HIVE-13572 : Redundant setting full file status in
Hive::copyFiles (Rui Li via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800871 - PreCommit-HIVE-MASTER-Build
> Change website references to HQL/HiveQL to SQL
> --
>
> Key: HIVE-13607
> URL: https://issues.apache.org/jira/browse/HIVE-13607
> Project: Hive
> Issue Type: Improvement
> Components: Website
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-13607.2.patch, HIVE-13607.patch
>
>
> When it started Hive's SQL dialect was far enough from standard SQL that the
> developers called it HQL or HiveQL.
> Over the years Hive's SQL dialect has matured. It still has some oddities
> but it is explicitly pushing towards SQL 2011 conformance. Calling the
> language anything but SQL now is confusing for users.
> In addition to changing the website I propose to make changes in the wiki.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13510) Dynamic partitioning doesn’t work when remote metastore is used
[
https://issues.apache.org/jira/browse/HIVE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263587#comment-15263587
]
Hive QA commented on HIVE-13510:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800855/HIVE-13510.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 9960 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkClient - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263585#comment-15263585
]
Lefty Leverenz commented on HIVE-13509:
---
Documented here:
* [HCatalog Configuration Properties -- Storage Directives |
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Configuration+Properties#HCatalogConfigurationProperties-StorageDirectives]
Thanks, [~ctang.ma]. I added version information and a link to this JIRA
issue.
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 2.1.0
>
> Attachments: HIVE-13509.1.patch, HIVE-13509.2.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12063) Pad Decimal numbers with trailing zeros to the scale of the column
[ https://issues.apache.org/jira/browse/HIVE-12063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12063: - Hadoop Flags: Incompatible change > Pad Decimal numbers with trailing zeros to the scale of the column > -- > > Key: HIVE-12063 > URL: https://issues.apache.org/jira/browse/HIVE-12063 > Project: Hive > Issue Type: Improvement > Components: Types >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 0.13 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12063.1.patch, HIVE-12063.2.patch, > HIVE-12063.3.patch, HIVE-12063.patch > > > HIVE-7373 was to address the problems of trimming tailing zeros by Hive, > which caused many problems including treating 0.0, 0.00 and so on as 0, which > has different precision/scale. Please refer to HIVE-7373 description. > However, HIVE-7373 was reverted by HIVE-8745 while the underlying problems > remained. HIVE-11835 was resolved recently to address one of the problems, > where 0.0, 0.00, and so on cannot be read into decimal(1,1). > However, HIVE-11835 didn't address the problem of showing as 0 in query > result for any decimal values such as 0.0, 0.00, etc. This causes confusion > as 0 and 0.0 have different precision/scale than 0. > The proposal here is to pad zeros for query result to the type's scale. This > not only removes the confusion described above, but also aligns with many > other DBs. Internal decimal number representation doesn't change, however. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13620: -- Attachment: HIVE-13620.3.patch Attaching patch v3, containing the llap branch diff, plus the changes based on [~hagleitn]'s review > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[
https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263429#comment-15263429
]
Hive QA commented on HIVE-13596:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801156/HIVE-13596.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 9961 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/111/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/111/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-111/
Messages:
{noformat}
Executing
[jira] [Commented] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263427#comment-15263427
]
Rui Li commented on HIVE-13572:
---
Thanks [~ashutoshc] for the review.
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.1.0
>
> Attachments: HIVE-13572.1.patch, HIVE-13572.2.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13572:
Affects Version/s: 2.1.0
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.1.0
>
> Attachments: HIVE-13572.1.patch, HIVE-13572.2.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13572:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, [~lirui]
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.1.0
>
> Attachments: HIVE-13572.1.patch, HIVE-13572.2.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-13588:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Committed to 2.1.0. Thanks [~ychena] for reviewing the patch.
> NPE is thrown from MapredLocalTask.executeInChildVM
> ---
>
> Key: HIVE-13588
> URL: https://issues.apache.org/jira/browse/HIVE-13588
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 2.1.0
>
> Attachments: HIVE-13588.1.patch, HIVE-13588.patch, HIVE-13588.patch
>
>
> NPE was thrown out from MapredLocalTask.executeInChildVM in running some
> queries with CLI, see error below:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:321)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:148)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1868)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1595)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1346)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1117)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1105)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[?:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_45]
> {code}
> It is because the operationLog is only applicable to HS2 but CLI, therefore
> it might not be set (null)
> It is related to HIVE-13183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-13509:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 2.1.0
>
> Attachments: HIVE-13509.1.patch, HIVE-13509.2.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263418#comment-15263418
]
Chaoyu Tang commented on HIVE-13509:
Committed to 2.1.0. Thanks [~szehon] and [~mithun] for reviewing the patch. The
new configuration hcat.input.ignore.invalid.path needs to be documented.
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.2.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263406#comment-15263406
]
Chaoyu Tang commented on HIVE-13588:
The failed tests are not related to this patch.
> NPE is thrown from MapredLocalTask.executeInChildVM
> ---
>
> Key: HIVE-13588
> URL: https://issues.apache.org/jira/browse/HIVE-13588
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13588.1.patch, HIVE-13588.patch, HIVE-13588.patch
>
>
> NPE was thrown out from MapredLocalTask.executeInChildVM in running some
> queries with CLI, see error below:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:321)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:148)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1868)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1595)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1346)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1117)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1105)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[?:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_45]
> {code}
> It is because the operationLog is only applicable to HS2 but CLI, therefore
> it might not be set (null)
> It is related to HIVE-13183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263390#comment-15263390
]
Rui Li commented on HIVE-13572:
---
[~ashutoshc] - Yeah I think we can check this one in. Test failures either
cannot be reproduced or fail in other runs, and therefore are not related to
the patch here.
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13572.1.patch, HIVE-13572.2.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263386#comment-15263386
]
Yongzhi Chen commented on HIVE-13632:
-
Each serde is different, for example the avro, avro record has a map:
record GenericData$Record (id=8454)
{"key": "abcd", "arrayvalues": [], "mapvalues": {}}
Each column name map to a value object. For empty array, it is an empty List
object.
The Serializing value is just convert the record directly by encoding:
BinaryEncoder be =
EncoderFactory.get().directBinaryEncoder((DataOutputStream)out, null);
So it is very easy to translate.
Our problem is, although hive know it is an empty list before Serializing to
Parque, but it does not know how to tell Parque it is an empty.
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263366#comment-15263366 ] Matt McCline commented on HIVE-13178: - Last queued build #141 or #142 > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch, HIVE-13178.04.patch, HIVE-13178.05.patch, > HIVE-13178.06.patch, HIVE-13178.07.patch, HIVE-13178.08.patch, > HIVE-13178.09.patch, HIVE-13178.091.patch, HIVE-13178.092.patch, > HIVE-13178.093.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.093.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch, HIVE-13178.04.patch, HIVE-13178.05.patch, > HIVE-13178.06.patch, HIVE-13178.07.patch, HIVE-13178.08.patch, > HIVE-13178.09.patch, HIVE-13178.091.patch, HIVE-13178.092.patch, > HIVE-13178.093.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: Patch Available (was: In Progress) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch, HIVE-13178.04.patch, HIVE-13178.05.patch, > HIVE-13178.06.patch, HIVE-13178.07.patch, HIVE-13178.08.patch, > HIVE-13178.09.patch, HIVE-13178.091.patch, HIVE-13178.092.patch, > HIVE-13178.093.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: In Progress (was: Patch Available) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch, HIVE-13178.04.patch, HIVE-13178.05.patch, > HIVE-13178.06.patch, HIVE-13178.07.patch, HIVE-13178.08.patch, > HIVE-13178.09.patch, HIVE-13178.091.patch, HIVE-13178.092.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263364#comment-15263364
]
Prasanth Jayachandran commented on HIVE-13447:
--
+1
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13648) Look at ORC data type conversion edge cases (CHAR, VARCHAR, DECIMAL)
[ https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13648: Description: New ORC data type conversion has some edges cases like CHAR/VAR different max lengths and DECIMAL different scale/precision that need attention. Also, wondering if some of the orc_* Q test failures (HIVE-13178 patch 092) were due to DECIMAL data type precision not being copied correctly? was: New ORC data type conversion has some edges cases like CHAR/VAR different max lengths and DECIMAL different scale/precision that need attention. Also, wondering if some of the orc_* Q test failures were due to DECIMAL data type precision not being copied correctly? > Look at ORC data type conversion edge cases (CHAR, VARCHAR, DECIMAL) > > > Key: HIVE-13648 > URL: https://issues.apache.org/jira/browse/HIVE-13648 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > New ORC data type conversion has some edges cases like CHAR/VAR different max > lengths and DECIMAL different scale/precision that need attention. > Also, wondering if some of the orc_* Q test failures (HIVE-13178 patch 092) > were due to DECIMAL data type precision not being copied correctly? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13648) Look at ORC data type conversion edge cases (CHAR, VARCHAR, DECIMAL)
[ https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13648: Description: New ORC data type conversion has some edges cases like CHAR/VAR different max lengths and DECIMAL different scale/precision that need attention. Also, wondering if some of the orc_* Q test failures were due to DECIMAL data type precision not being copied correctly? was:New ORC data type conversion has some edges cases like CHAR/VAR different max lengths and DECIMAL different scale/precision that need attention. > Look at ORC data type conversion edge cases (CHAR, VARCHAR, DECIMAL) > > > Key: HIVE-13648 > URL: https://issues.apache.org/jira/browse/HIVE-13648 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > New ORC data type conversion has some edges cases like CHAR/VAR different max > lengths and DECIMAL different scale/precision that need attention. > Also, wondering if some of the orc_* Q test failures were due to DECIMAL data > type precision not being copied correctly? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Attachment: HIVE-13442.patch > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.nogen.patch, HIVE-13442.patch, > HIVE-13442.patch, HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Status: Patch Available (was: Open) > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.nogen.patch, HIVE-13442.patch, > HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Attachment: HIVE-13442.patch The patch. > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.nogen.patch, HIVE-13442.patch, > HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263310#comment-15263310 ] Gunther Hagleitner commented on HIVE-13620: --- comments on rb. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263277#comment-15263277 ] Lefty Leverenz commented on HIVE-13536: --- Doc note: This adds *hive.llap.daemon.task.preemption.metrics.intervals* to HiveConf.java, so it will need to be documented in the LLAP section of Configuration Properties for release 2.1.0. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC2.1 label. > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch, HIVE-13536.4.patch, HIVE-13536.5.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13536: -- Labels: TODOC2.1 (was: ) > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch, HIVE-13536.4.patch, HIVE-13536.5.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263255#comment-15263255 ] Lefty Leverenz commented on HIVE-13249: --- Thanks [~wzheng], looks good. +1 for the parameter descriptions. > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[
https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263247#comment-15263247
]
Hive QA commented on HIVE-11417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800872/HIVE-11417.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 63 failed/errored test(s), 9956 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_casts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_timestamp_ints_casts
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_casts
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparator
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263146#comment-15263146
]
Sergio Peña commented on HIVE-13632:
[~ychena] I think we should try to be consistent with other file formats before
committing this patch.
Could you investigate what other format serde or object inspectors do in order
to translate this to empty lists?
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13620: -- Attachment: HIVE-13620.2.patch Sorry, looks like when I did the diff with master, master probably had a commit or two which hadn't yet been merged into the llap branch. Attaching patch v2 where they should be more in sync. RB at https://reviews.apache.org/r/46802/ > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Attachment: HIVE-13615.1.patch > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Attachment: (was: HIVE-13615.1.patch) > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263134#comment-15263134
]
Yongzhi Chen commented on HIVE-13632:
-
[~spena], I think I tried it, the result is [null]
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263126#comment-15263126 ] Ashutosh Chauhan commented on HIVE-13637: - Wondering if it were better to do this on Calcite tree instead? May be HiveRexUtil::simplify() > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263111#comment-15263111 ] Gunther Hagleitner commented on HIVE-13620: --- The logging in HivePassThroughRecordWriter looks to be debug logging and should be removed. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263113#comment-15263113 ] Gunther Hagleitner commented on HIVE-13620: --- Change in ColumnPrunerFactory looks wrong. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263107#comment-15263107 ] Gunther Hagleitner commented on HIVE-13620: --- There's TODOs in HiveSplitGenerator that should be looked at, I think. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263103#comment-15263103 ] Gunther Hagleitner commented on HIVE-13620: --- There's a diff in Driver.java that looks incorrect (accidental) also. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[
https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263100#comment-15263100
]
Gunther Hagleitner commented on HIVE-13620:
---
Should remove:
{noformat}
LOG.warn("ZZZ: DBG: usingTezAm=" + withTezAm);
{noformat}
> Merge llap branch work to master
>
>
> Key: HIVE-13620
> URL: https://issues.apache.org/jira/browse/HIVE-13620
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13620.1.patch, llap_master_diff.txt
>
>
> Would like to try to merge the llap branch work for HIVE-12991 into the
> master branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263088#comment-15263088 ] Gunther Hagleitner commented on HIVE-13620: --- Are the changes in TestHiveAuthorizerCheckInvocation.java intentional? > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263081#comment-15263081 ] Gunther Hagleitner commented on HIVE-13620: --- Can you open a review board request for this? > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263064#comment-15263064 ] Gunther Hagleitner commented on HIVE-13518: --- [~vikram.dixit] is this ready? > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263058#comment-15263058 ] Gunther Hagleitner commented on HIVE-13619: --- The log statements look like debug leftover, can you remove them? Otherwise +1. > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13619.1.patch > > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263054#comment-15263054
]
Sergio Peña commented on HIVE-13632:
I think this should work in situations where the list does not contain values.
Try it and see what it does.
recordConsumer.startField(repeatedGroupName, 0);
recordConsumer.startGroup()
recordConsumer.endGroup()
recordConsumer.endField(repeatedGroupName, 0);
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263052#comment-15263052
]
Gunther Hagleitner commented on HIVE-13621:
---
Should be able to use:
{noformat}
com.google.protobuf.ByteString.EMPTY
{noformat}
instead of your 1 byte array.
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13645) Beeline needs null-guard around hiveVars and hiveConfVars read
[
https://issues.apache.org/jira/browse/HIVE-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263039#comment-15263039
]
Thejas M Nair commented on HIVE-13645:
--
+1
> Beeline needs null-guard around hiveVars and hiveConfVars read
> --
>
> Key: HIVE-13645
> URL: https://issues.apache.org/jira/browse/HIVE-13645
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Minor
> Attachments: HIVE-13645.patch
>
>
> Beeline has a bug wherein if a user does a !save ever, then on next load, if
> beeline.hiveVariables or beeline.hiveconfvariables are empty, i.e. \{\} or
> unspecified, then it loads it as null, and then, on next connect, there is no
> null-check on these variables leading to an NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[
https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263037#comment-15263037
]
Hive QA commented on HIVE-13178:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800832/HIVE-13178.092.patch
{color:green}SUCCESS:{color} +1 due to 30 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 9954 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-transform_ppr2.q-vector_outer_join0.q-vector_bround.q-and-10-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_split_elimination
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion3_acid
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
[jira] [Commented] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263036#comment-15263036
]
Gunther Hagleitner commented on HIVE-13621:
---
The explain plan in the test shows this:
{noformat}
Select Operator [SEL_9] (rows=9223372036854775807 width=1)
{noformat}
This seems horribly wrong, no? (might be unrelated, but looks serious).
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263032#comment-15263032
]
Yongzhi Chen commented on HIVE-13632:
-
They show empty. But I can not find a way in parquet.
writeNullForMissingFieldsAtCurrentLevel(); in the
java/parquet/io/MessageColumnIO.java may cause this.
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263023#comment-15263023 ] Gunther Hagleitner commented on HIVE-13568: --- +1 assuming tests pass. > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch, > HIVE-13568.2.patch, HIVE-13568.3.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263018#comment-15263018 ] Gunther Hagleitner commented on HIVE-13343: --- Couple of nits. Otherwise +1. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Attachment: HIVE-13637.patch > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Status: Patch Available (was: In Progress) > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262997#comment-15262997 ] Sergey Shelukhin commented on HIVE-13628: - findClassNotFoundException doesn't need recursion... ke = ke.getCause until it's null > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13637 started by Jesus Camacho Rodriguez. -- > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13627) When running under LLAP, for regular map joins, throw an error if memory utilization goes above what is allocated to the task
[ https://issues.apache.org/jira/browse/HIVE-13627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262990#comment-15262990 ] Gunther Hagleitner commented on HIVE-13627: --- Nope that's the patch for HIVE-13628. > When running under LLAP, for regular map joins, throw an error if memory > utilization goes above what is allocated to the task > - > > Key: HIVE-13627 > URL: https://issues.apache.org/jira/browse/HIVE-13627 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch > > > When running under LLAP, for regular map joins, throw an error if memory > utilization goes above what is allocated to the task. This way, the rest of > the dependent tasks can fail sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13628) Support for permanent functions - error handling if no restart
[
https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262982#comment-15262982
]
Gunther Hagleitner commented on HIVE-13628:
---
Cool. Couple of minor recommendations:
a.)
{noformat}
if (LlapProxy.isDaemon()) { // daemon
{noformat}
might be a better check than just configuration.
b.)
{noformat}
import sun.tools.java.ClassNotFound;
{noformat}
is not needed, is it?
c.)
The message should be qualified by a "maybe" i think. It's still possible that
there's something else that is wrong. I think it should say something like.
"Missing class "XXX". If this a UDF and you're running llap, you might require
..."
Otherwise +1 pending tests.
> Support for permanent functions - error handling if no restart
> --
>
> Key: HIVE-13628
> URL: https://issues.apache.org/jira/browse/HIVE-13628
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13628.1.patch
>
>
> Support for permanent functions - error handling if no restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13623) Hive on tez produce wrong results when withClause and (outer) joins
[
https://issues.apache.org/jira/browse/HIVE-13623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivashankar reassigned HIVE-13623:
--
Assignee: Sivashankar
> Hive on tez produce wrong results when withClause and (outer) joins
> ---
>
> Key: HIVE-13623
> URL: https://issues.apache.org/jira/browse/HIVE-13623
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0
>Reporter: JinsuKim
>Assignee: Sivashankar
>Priority: Trivial
>
> Hive on mr produce correct results when (outer) joins and withClause. But tez
> produce wrong results
> {code:sql|title=Case1}
> with a as (
> select 1 as c1
> union all
> select 2 as c1
> union all
> select 3 as c1
> ),
> b as (
> select 1 as c1
> )
> select *
> from ( select a.c1 as ac1, b.c1 as bc1
> from a left outer join b
> on a.c1 = b.c1
>) c;
> where c.bc1 is null
> {code}
> {code:title=case1 result}
> mr :
> +++--+
> | c.ac1 | c.bc1 |
> +++--+
> | 2 | NULL |
> | 3 | NULL |
> +++--+
> tez :
> +++--+
> | c.ac1 | c.bc1 |
> +++--+
> +++--+
> {code}
> Case2 as similar to Case1 produces same results.
> {code:sql|title=Case2}
> with a as (
> select * from j1
> ),
> b as (
> select * from j2
> )
> select *
> from ( select a.c1 as ac1, b.c1 as bc1
> from a left outer join b
> on a.c1 = b.c1
>) c
> where c.bc1 is null;
>
> drop table j1;
> create table j1 as select c1 from
> ( select 1 as c1 from default.dual
> union all
> select 2 as c1 from default.dual
> union all
> select 3 as c1 from default.dual
> ) t1;
> drop table j2;
> create table j2 as
> select 1 as c1 from default.dual;
> {code}
> {code:title=case2 result}
> mr :
> +++--+
> | c.ac1 | c.bc1 |
> +++--+
> | 2 | NULL |
> | 3 | NULL |
> +++--+
> tez :
> +++--+
> | c.ac1 | c.bc1 |
> +++--+
> | 2 | NULL |
> | 3 | NULL |
> +++--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262955#comment-15262955
]
Ashutosh Chauhan commented on HIVE-13572:
-
I think setting perms in FS might be a bigger change. [~ruili] Shall we check
this one in while you work on FS changes. I am +1 on current patch.
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13572.1.patch, HIVE-13572.2.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10294) ParseException issue (Failed to recognize predicate 'user') [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivashankar reassigned HIVE-10294:
--
Assignee: Sivashankar
> ParseException issue (Failed to recognize predicate 'user') [Spark Branch]
> --
>
> Key: HIVE-10294
> URL: https://issues.apache.org/jira/browse/HIVE-10294
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xin Hao
>Assignee: Sivashankar
>
> ParseException (Failed to recognize predicate 'user') occurred when we run
> some Hive query (e.g. BigBench case Q03).
> By the way, if we change all the 'user' to 'user2', the parsing could be
> passed.
> (1) Exception Details:
> {noformat}
> FailedPredicateException(identifier,{useSQL11ReservedKeywordsForIdentifier()}?)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10912)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:45479)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2933)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1367)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1122)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45475)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41136)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41043)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40059)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5295)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3731)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1865)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1512)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45556)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.singleFromStatement(HiveParser.java:40666)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromStatement(HiveParser.java:40389)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40044)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5295)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3731)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1865)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1512)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45556)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41157)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40876)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40059)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1574)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.explainStatement(HiveParser.java:1269)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1070)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:397)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:309)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1115)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1163)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1052)
> at
[jira] [Assigned] (HIVE-13646) make hive.optimize.sort.dynamic.partition compatible with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-13646: - Assignee: Eugene Koifman > make hive.optimize.sort.dynamic.partition compatible with ACID tables > - > > Key: HIVE-13646 > URL: https://issues.apache.org/jira/browse/HIVE-13646 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries. > dynamic inserts are common in ACID and this leaves users with few options if > they are seeing OutOfMemory errors due to too many writers. > hive.optimize.sort.dynamic.partition sorts data by partition col/bucket > col/sort col to ensure each reducer only needs 1 writer. > Acid requires data in each bucket file to be sorted by ROW__ID and thus > doesn't allow end user to determine sorting. > So we should be able to support hive.optimize.sort.dynamic.partition with > sort on partition col/bucket col/ROW__ID -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Attachment: HIVE-13615.1.patch cc-ing [~ashutoshc] and [~pxiong] for review. > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Status: Patch Available (was: Open) > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[
https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262893#comment-15262893
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13615:
--
[~ashutoshc] Here is the problem:
{code}
identifier
:
Identifier
| nonReserved -> Identifier[$nonReserved.text]
// If it decides to support SQL11 reserved keywords, i.e.,
useSQL11ReservedKeywordsForIdentifier()=false,
// the sql11keywords in existing q tests will NOT be added back.
| {useSQL11ReservedKeywordsForIdentifier()}?
sql11ReservedKeywordsUsedAsIdentifier ->
Identifier[$sql11ReservedKeywordsUsedAsIdentifier.text]
;
{code}
If you at the grammar files we use "Identifier" and "identifier"
interchangeably. I think we should use "identifier" everywhere except for the
above defiinition.
cc-ing [~pxiong]
Thanks
Hari
> nomore_ambiguous_table_col.q is failing on master
> -
>
> Key: HIVE-13615
> URL: https://issues.apache.org/jira/browse/HIVE-13615
> Project: Hive
> Issue Type: Test
> Components: Parser
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> Fails with:
> FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key'
> 'INSERT' in from source 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-13615: Assignee: Hari Sankar Sivarama Subramaniyan > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262859#comment-15262859
]
Sergio Peña commented on HIVE-13632:
What do other file formats show with empty arrays? null or empty?
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262847#comment-15262847
]
Sergio Peña commented on HIVE-13632:
Looks good
+1
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13645) Beeline needs null-guard around hiveVars and hiveConfVars read
[
https://issues.apache.org/jira/browse/HIVE-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-13645:
Priority: Minor (was: Major)
> Beeline needs null-guard around hiveVars and hiveConfVars read
> --
>
> Key: HIVE-13645
> URL: https://issues.apache.org/jira/browse/HIVE-13645
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Minor
> Attachments: HIVE-13645.patch
>
>
> Beeline has a bug wherein if a user does a !save ever, then on next load, if
> beeline.hiveVariables or beeline.hiveconfvariables are empty, i.e. \{\} or
> unspecified, then it loads it as null, and then, on next connect, there is no
> null-check on these variables leading to an NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13645) Beeline needs null-guard around hiveVars and hiveConfVars read
[
https://issues.apache.org/jira/browse/HIVE-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262815#comment-15262815
]
Sushanth Sowmyan commented on HIVE-13645:
-
[~thejas], could you please take a look - this is a near-trivial patch.
> Beeline needs null-guard around hiveVars and hiveConfVars read
> --
>
> Key: HIVE-13645
> URL: https://issues.apache.org/jira/browse/HIVE-13645
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13645.patch
>
>
> Beeline has a bug wherein if a user does a !save ever, then on next load, if
> beeline.hiveVariables or beeline.hiveconfvariables are empty, i.e. \{\} or
> unspecified, then it loads it as null, and then, on next connect, there is no
> null-check on these variables leading to an NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13645) Beeline needs null-guard around hiveVars and hiveConfVars read
[
https://issues.apache.org/jira/browse/HIVE-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-13645:
Attachment: HIVE-13645.patch
Attaching patch.
> Beeline needs null-guard around hiveVars and hiveConfVars read
> --
>
> Key: HIVE-13645
> URL: https://issues.apache.org/jira/browse/HIVE-13645
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13645.patch
>
>
> Beeline has a bug wherein if a user does a !save ever, then on next load, if
> beeline.hiveVariables or beeline.hiveconfvariables are empty, i.e. \{\} or
> unspecified, then it loads it as null, and then, on next connect, there is no
> null-check on these variables leading to an NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13645) Beeline needs null-guard around hiveVars and hiveConfVars read
[
https://issues.apache.org/jira/browse/HIVE-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-13645:
Status: Patch Available (was: Open)
> Beeline needs null-guard around hiveVars and hiveConfVars read
> --
>
> Key: HIVE-13645
> URL: https://issues.apache.org/jira/browse/HIVE-13645
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13645.patch
>
>
> Beeline has a bug wherein if a user does a !save ever, then on next load, if
> beeline.hiveVariables or beeline.hiveconfvariables are empty, i.e. \{\} or
> unspecified, then it loads it as null, and then, on next connect, there is no
> null-check on these variables leading to an NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13632:
Status: Patch Available (was: Open)
[~spena], could you review the code?
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13632:
Attachment: HIVE-13632.1.patch
patch 1 fixes the exception by avoid creating empty field.
But parquet api seems have limitations, empty array and map are stored as null.
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Attachment: HIVE-13442.protobuf.patch [~sseth] can you take a look at the API (based on the discussion) before I rebuild and start refactoring the usage. > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks [~sseth] for the reviews! > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch, HIVE-13536.4.patch, HIVE-13536.5.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262769#comment-15262769 ] Prasanth Jayachandran commented on HIVE-13536: -- I ran all the 28 llap test failures locally on jdk8 and all of them ran successfully. Not sure why it failed in the last precommit run. Other test failures are not related. > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch, > HIVE-13536.3.patch, HIVE-13536.4.patch, HIVE-13536.5.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[
https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262734#comment-15262734
]
Sergey Shelukhin commented on HIVE-13643:
-
Nit: is it possible to capitalize LLAP in log lines?
{noformat}
if (StringUtils.isEmpty(appName)) {
+appName = HiveConf.getVar(conf,
HiveConf.ConfVars.LLAP_DAEMON_SERVICE_HOSTS);
+if (appName.startsWith("@") && appName.length() > 1) {
+ appName = appName.substring(1);
+}
+ }
+ if (StringUtils.isEmpty(appName) || (appName.startsWith("@") ||
appName.length() == 1)) {
+String message = "Invalid app name. This must be setup via config or
passed in as a parameter";
+LOG.info(message);
+return ExitCode.INCORRECT_USAGE.getInt();
+ }
{noformat}
Would an app name being a list of hosts with dots, commas and stuff actually
work? And make sense?
Also why cannot the length be 1
> Various enhancements / fixes to llap cli tools
> --
>
> Key: HIVE-13643
> URL: https://issues.apache.org/jira/browse/HIVE-13643
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13643.01.patch
>
>
> - BUG - llapstatus does not always produce output (PrintWriter.close)
> - llapstatus should parse hiveconf
> - package llap-cli-log4j2.template
> - Log at the start and end of the script to include parameters
> - Generate logs to a file different from hive.log
> - hive --service llap --name does not propagate the name to the daemons
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[
https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262691#comment-15262691
]
Hive QA commented on HIVE-13560:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801086/HIVE-13560.4.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 104 failed/errored test(s), 9918 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-auto_sortmerge_join_7.q-orc_merge9.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_empty_dir_in_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_external_table_with_space_in_location_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_file_with_header_footer
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_gen_udf_example_add10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_import_exported_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_input16_cc
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_leftsemijoin_mr
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_fs2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge8
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_quotedid_smb
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262677#comment-15262677 ] Sergey Shelukhin commented on HIVE-12963: - Sorry, forgot about this... the test failed in the above QA run, and it passes for other JIRAs. I'll run it locally to see if it passes and commit if it does. > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13641) LLAP fails to start if llap.io.enabled is set to true, and cache size is set to 0
[ https://issues.apache.org/jira/browse/HIVE-13641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-13641. - Resolution: Not A Problem hive.llap.io.memory.mode can be set to "none". The perf may suffer, though... > LLAP fails to start if llap.io.enabled is set to true, and cache size is set > to 0 > - > > Key: HIVE-13641 > URL: https://issues.apache.org/jira/browse/HIVE-13641 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > > With this configuration - I'd expect the cache to be automatically disabled / > or just the elevator used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262629#comment-15262629 ] Sergey Shelukhin commented on HIVE-12878: - +1 pending tests. I'd rather that the public fields were changed to have getters/setters... > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch, HIVE-12878.091.patch, HIVE-12878.092.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Status: Patch Available (was: Open) > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0, 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.6.patch, HIVE-13395.7.patch, > HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T ~2~: w ~2~\[x] -> w ~2~\[y] -> c ~2~ > Suppose the
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Attachment: HIVE-13395.13.patch patch 13 is the same as 12 > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.6.patch, HIVE-13395.7.patch, > HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T ~2~: w ~2~\[x] -> w ~2~\[y] ->
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Status: Open (was: Patch Available) > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0, 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.6.patch, HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T ~2~: w ~2~\[x] -> w ~2~\[y] -> c ~2~ > Suppose the order of operations is r
[jira] [Updated] (HIVE-13609) Fix UDTFs to allow local fetch task to fetch rows forwarded by GenericUDTF.close()
[ https://issues.apache.org/jira/browse/HIVE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13609: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master > Fix UDTFs to allow local fetch task to fetch rows forwarded by > GenericUDTF.close() > -- > > Key: HIVE-13609 > URL: https://issues.apache.org/jira/browse/HIVE-13609 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.1.0 > > Attachments: HIVE-13609.1.patch > > > From [~ashutoshc]'s comments in HIVE-13586, attempt to fix whatever is > causing the local fetch task to not get the rows forwarded by UDTF close(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13249: - Attachment: HIVE-13249.6.patch patch 6 addresses [~leftylev]'s comments. Thanks! > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13644:
---
Attachment: HIVE-13644.1.patch
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13644:
---
Status: Patch Available (was: Open)
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[ https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262442#comment-15262442 ] Alan Gates commented on HIVE-13607: --- I think it would be good to have a more full description of hive, either later on the page after all the links or perhaps on the first page of the wiki (which already has a more complete description, and which I plan to edit next). I'm going to keep this fix just to the current changes (thanks for the review btw), but we could file a JIRA for further improvements. > Change website references to HQL/HiveQL to SQL > -- > > Key: HIVE-13607 > URL: https://issues.apache.org/jira/browse/HIVE-13607 > Project: Hive > Issue Type: Improvement > Components: Website >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13607.2.patch, HIVE-13607.patch > > > When it started Hive's SQL dialect was far enough from standard SQL that the > developers called it HQL or HiveQL. > Over the years Hive's SQL dialect has matured. It still has some oddities > but it is explicitly pushing towards SQL 2011 conformance. Calling the > language anything but SQL now is confusing for users. > In addition to changing the website I propose to make changes in the wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11339) org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out) makes incorrect cast
[
https://issues.apache.org/jira/browse/HIVE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262393#comment-15262393
]
Arnaud Linz commented on HIVE-11339:
Ok for me.
> org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out)
> makes incorrect cast
> -
>
> Key: HIVE-11339
> URL: https://issues.apache.org/jira/browse/HIVE-11339
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.14.0
>Reporter: Arnaud Linz
>Assignee: Zoltan Haindrich
> Labels: easyfix, newbie
> Attachments: HIVE-11339.patch
>
>
> Hi, it's my first Jira and I don't know how to make patches, so I'll explain
> the issue in the description as it is rather simple.
> I have a problem serializing "DefaultHCatRecord" using Apache Flink when
> those records include Timestamps because of an incorrect class cast in
> org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out). It
> is implemented using a cast to Outputstream :
> public void write(DataOutput out) throws IOException {
> write((OutputStream) out);
> }
> but nothing says that a DataOutput object is an OutputStream, (and it's not
> the case in Flink) it should rather be implmented using the same code as
> write(OutputStream) :
> {
> checkBytes();
> out.write(currentBytes, offset, getTotalLength());
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262331#comment-15262331
]
Hive QA commented on HIVE-13588:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800818/HIVE-13588.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 9964 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Fix Version/s: 2.0.1 > Include Apache Phoenix support in HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1, 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Fix For: 2.0.1 > > Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > in Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Status: Open (was: Patch Available) > Include Apache Phoenix support in HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 2.0.0, 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > in Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Status: Patch Available (was: Open) > Include Apache Phoenix support in HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 2.0.0, 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > in Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Fix Version/s: 2.0.1 Status: Patch Available (was: Open) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Fix For: 2.0.1 > > Attachments: HIVE-13584-branch-2.0.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Summary: Various enhancements / fixes to llap cli tools (was: Various enhancements to llap cli tools) > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Status: Patch Available (was: Open) > Various enhancements to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Attachment: HIVE-13643.01.patch Patch to address the issues mentioned, except the last one. Will upload another one for that here or in a separate jira. [~prasanth_j], [~sershe] - please review. This is a fairly simple set of changes. > Various enhancements to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
