[jira] [Commented] (HIVE-17277) HiveMetastoreClient Log name is wrong
[
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131761#comment-16131761
]
Hive QA commented on HIVE-17277:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882487/HIVE-17277.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6450/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6450/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6450/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-08-18 05:51:50.190
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6450/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-08-18 05:51:50.193
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
48be6ae..c9e0940 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 48be6ae HIVE-17347:
TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is
failing every time (Peter Vary reviewed by Xuefu Zhang)
+ git clean -f -d
Removing ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsBase.java
Removing ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java
Removing ql/src/test/queries/clientpositive/acid_no_buckets.q
Removing ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at c9e0940 HIVE-17194: JDBC: Implement Gzip compression for HTTP
mode (Gopal V, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-08-18 05:51:53.202
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer : Classpath
>> /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) :
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList
ENHANCED (Persistable) :
org.apache.hadoop.hive.metastore.model.MStorageDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartition
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MIndex
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRole
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRoleMap
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MGlobalPrivilege
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDBPrivilege
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTablePrivilege
ENHANCED
[jira] [Commented] (HIVE-17205) add functional support
[
https://issues.apache.org/jira/browse/HIVE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131757#comment-16131757
]
Hive QA commented on HIVE-17205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882486/HIVE-17205.10.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10982 tests
executed
*Failed tests:*
{noformat}
TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out)
(batchId=281)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_not_acid]
(batchId=90)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=206)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTransactionalValidation
(batchId=208)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTransactionalValidation
(batchId=205)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTransactionalValidation
(batchId=203)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01
(batchId=284)
org.apache.hadoop.hive.ql.TestTxnCommands.testTimeOutReaper (batchId=284)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=271)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBucketsTableCreate
(batchId=271)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6449/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6449/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6449/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882486 - PreCommit-HIVE-Build
> add functional support
> --
>
> Key: HIVE-17205
> URL: https://issues.apache.org/jira/browse/HIVE-17205
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-17205.01.patch, HIVE-17205.02.patch,
> HIVE-17205.03.patch, HIVE-17205.09.patch, HIVE-17205.10.patch
>
>
> make sure unbucketed tables can be marked transactional=true
> make insert/update/delete/compaction work
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17194) JDBC: Implement Gzip compression for HTTP mode
[
https://issues.apache.org/jira/browse/HIVE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-17194:
---
Summary: JDBC: Implement Gzip compression for HTTP mode (was: JDBC:
Implement Gzip servlet filter)
> JDBC: Implement Gzip compression for HTTP mode
> --
>
> Key: HIVE-17194
> URL: https://issues.apache.org/jira/browse/HIVE-17194
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-17194.1.patch, HIVE-17194.2.patch,
> HIVE-17194.3.patch
>
>
> {code}
> POST /cliservice HTTP/1.1
> Content-Type: application/x-thrift
> Accept: application/x-thrift
> User-Agent: Java/THttpClient/HC
> Authorization: Basic YW5vbnltb3VzOmFub255bW91cw==
> Content-Length: 71
> Host: localhost:10007
> Connection: Keep-Alive
> Accept-Encoding: gzip,deflate
> X-XSRF-HEADER: true
> {code}
> The Beeline client clearly sends out HTTP compression headers which are
> ignored by the HTTP service layer in HS2.
> After patch, result looks like
> {code}
> HTTP/1.1 200 OK
> Date: Tue, 01 Aug 2017 01:47:23 GMT
> Content-Type: application/x-thrift
> Vary: Accept-Encoding, User-Agent
> Content-Encoding: gzip
> Transfer-Encoding: chunked
> Server: Jetty(9.3.8.v20160314)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-13989) Extended ACLs are not handled according to specification
[
https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Drome updated HIVE-13989:
---
Attachment: HIVE-13989.4-branch-2.2.patch
Uploaded new version of branch-2.2 patch.
> Extended ACLs are not handled according to specification
>
>
> Key: HIVE-13989
> URL: https://issues.apache.org/jira/browse/HIVE-13989
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-13989.1-branch-1.patch, HIVE-13989.1.patch,
> HIVE-13989.4-branch-2.2.patch, HIVE-13989-branch-1.patch,
> HIVE-13989-branch-2.2.patch, HIVE-13989-branch-2.2.patch,
> HIVE-13989-branch-2.2.patch
>
>
> Hive takes two approaches to working with extended ACLs depending on whether
> data is being produced via a Hive query or HCatalog APIs. A Hive query will
> run an FsShell command to recursively set the extended ACLs for a directory
> sub-tree. HCatalog APIs will attempt to build up the directory sub-tree
> programmatically and runs some code to set the ACLs to match the parent
> directory.
> Some incorrect assumptions were made when implementing the extended ACLs
> support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the
> design documents of extended ACLs in HDFS. These documents model the
> implementation after the POSIX implementation on Linux, which can be found at
> http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html.
> The code for setting extended ACLs via HCatalog APIs is found in
> HdfsUtils.java:
> {code}
> if (aclEnabled) {
> aclStatus = sourceStatus.getAclStatus();
> if (aclStatus != null) {
> LOG.trace(aclStatus.toString());
> aclEntries = aclStatus.getEntries();
> removeBaseAclEntries(aclEntries);
> //the ACL api's also expect the tradition user/group/other permission
> in the form of ACL
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER,
> sourcePerm.getUserAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP,
> sourcePerm.getGroupAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER,
> sourcePerm.getOtherAction()));
> }
> }
> {code}
> We found that DEFAULT extended ACL rules were not being inherited properly by
> the directory sub-tree, so the above code is incomplete because it
> effectively drops the DEFAULT rules. The second problem is with the call to
> {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended
> ACLs. When extended ACLs are used the GROUP permission is replaced with the
> extended ACL mask. So the above code will apply the wrong permissions to the
> GROUP. Instead the correct GROUP permissions now need to be pulled from the
> AclEntry as returned by {{getAclStatus().getEntries()}}. See the
> implementation of the new method {{getDefaultAclEntries}} for details.
> Similar issues exist with the HCatalog API. None of the API accounts for
> setting extended ACLs on the directory sub-tree. The changes to the HCatalog
> API allow the extended ACLs to be passed into the required methods similar to
> how basic permissions are passed in. When building the directory sub-tree the
> extended ACLs of the table directory are inherited by all sub-directories,
> including the DEFAULT rules.
> Replicating the problem:
> Create a table to write data into (I will use acl_test as the destination and
> words_text as the source) and set the ACLs as follows:
> {noformat}
> $ hdfs dfs -setfacl -m
> default:user::rwx,default:group::r-x,default:mask::rwx,default:user:hdfs:rwx,group::r-x,user:hdfs:rwx
> /user/cdrome/hive/acl_test
> $ hdfs dfs -ls -d /user/cdrome/hive/acl_test
> drwxrwx---+ - cdrome hdfs 0 2016-07-13 20:36
> /user/cdrome/hive/acl_test
> $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test
> # file: /user/cdrome/hive/acl_test
> # owner: cdrome
> # group: hdfs
> user::rwx
> user:hdfs:rwx
> group::r-x
> mask::rwx
> other::---
> default:user::rwx
> default:user:hdfs:rwx
> default:group::r-x
> default:mask::rwx
> default:other::---
> {noformat}
> Note that the basic GROUP permission is set to {{rwx}} after setting the
> ACLs. The ACLs explicitly set the DEFAULT rules and a rule specifically for
> the {{hdfs}} user.
> Run the following query to populate the table:
> {noformat}
> insert into acl_test partition (dt='a', ds='b') select a, b from words_text
> where dt = 'c';
> {noformat}
> Note that words_text only has a single partition key.
> Now examine the ACLs for the resulting directories:
> {noformat}
> $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test
> # file: /user/cdrome/hive/acl_test
> # owner: cdrome
> # group: hdfs
>
[jira] [Commented] (HIVE-17354) Fix "alter view" for incremental replication
[
https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131712#comment-16131712
]
Hive QA commented on HIVE-17354:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882482/HIVE-17354.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10976 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6448/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6448/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6448/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882482 - PreCommit-HIVE-Build
> Fix "alter view" for incremental replication
>
>
> Key: HIVE-17354
> URL: https://issues.apache.org/jira/browse/HIVE-17354
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17354.1.patch
>
>
> There is a bug that "alter view" operation is resulting in a view creation
> operation instead of a overwriting/replacement operation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-16948:
Attachment: HIVE-16948.7.patch
> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch,
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
> in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all
> select distinct(ds) as ds from srcpart) s where s.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
> DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
> Vertices:
> Map 10
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: max(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: min(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11
> Reduce Operator Tree:
> Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: _col0 is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: _col0 (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
>
[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131675#comment-16131675
]
liyunzhang_intel commented on HIVE-16948:
-
[~pvary]: I saw your comment on review board. Thanks for your tracking
HIVE-17346.
> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch,
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
> in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all
> select distinct(ds) as ds from srcpart) s where s.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
> DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
> Vertices:
> Map 10
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: max(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: min(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11
> Reduce Operator Tree:
> Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: _col0 is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: _col0 (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
> COMPLETE Column stats: NONE
>
[jira] [Commented] (HIVE-17265) Cache merged column stats from retrieved partitions
[
https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131668#comment-16131668
]
Hive QA commented on HIVE-17265:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882477/HIVE-17265.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 10976 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[filter_join_breaktask2]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[filter_join_breaktask]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_shared_scan]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_empty]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mr_diff_schema_alias]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez-tag]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6447/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6447/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6447/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882477 - PreCommit-HIVE-Build
> Cache merged column stats from retrieved partitions
> ---
>
> Key: HIVE-17265
> URL: https://issues.apache.org/jira/browse/HIVE-17265
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17265.02.patch, HIVE-17265.03.patch,
> HIVE-17265.patch
>
>
> Currently when we retrieve stats from the metastore for a column in a
> partitioned table, we will execute the logic to merge the column stats coming
> from each partition multiple times.
> Even though we avoid multiple calls to metastore if the cache for the stats
> in enabled, merging the stats for a given column can take a large amount of
> time if there is a large number of partitions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong
[ https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou updated HIVE-17277: Attachment: (was: HIVE-17277.patch) > HiveMetastoreClient Log name is wrong > - > > Key: HIVE-17277 > URL: https://issues.apache.org/jira/browse/HIVE-17277 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Minor > > The name of Log for HiveMetastoreClient is "hive.metastore". It's confused > for users to trace hive log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17347: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Peter and Xuefu! > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-17346:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Peter and Vlad.
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement
[
https://issues.apache.org/jira/browse/HIVE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131625#comment-16131625
]
Hive QA commented on HIVE-17353:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882472/HIVE-17353.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6446/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6446/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6446/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882472 - PreCommit-HIVE-Build
> The ResultSets are not accessible if running multiple queries within the same
> HiveStatement
>
>
> Key: HIVE-17353
> URL: https://issues.apache.org/jira/browse/HIVE-17353
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17353.1.patch
>
>
> The following queries would fail,
> {noformat}
> ResultSet rs1 =
> stmt.executeQuery("select * from testMultipleResultSets1");
> ResultSet rs2 =
> stmt.executeQuery("select * from testMultipleResultSets2");
> rs1.next();
> rs2.next();
> {noformat}
> with the exception:
> {noformat}
> [HiveServer2-Handler-Pool: Thread-208]: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131618#comment-16131618
]
Rui Li edited comment on HIVE-17292 at 8/18/17 2:22 AM:
Patch LGTM. Let's wait a little bit to get HIVE-17346 and HIVE-17347 in first.
was (Author: lirui):
Path LGTM. Let's wait a little bit to get HIVE-17346 and HIVE-17347 in first.
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131618#comment-16131618
]
Rui Li commented on HIVE-17292:
---
Path LGTM. Let's wait a little bit to get HIVE-17346 and HIVE-17347 in first.
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong
[ https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou updated HIVE-17277: Attachment: HIVE-17277.patch Thanks Alan for your comments. Resubmitting the patch to run unit tests > HiveMetastoreClient Log name is wrong > - > > Key: HIVE-17277 > URL: https://issues.apache.org/jira/browse/HIVE-17277 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Minor > Attachments: HIVE-17277.patch > > > The name of Log for HiveMetastoreClient is "hive.metastore". It's confused > for users to trace hive log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131614#comment-16131614 ] Rui Li commented on HIVE-17347: --- +1 > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131613#comment-16131613
]
Rui Li commented on HIVE-17346:
---
Thanks [~pvary] for working on this. +1
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17205) add functional support
[ https://issues.apache.org/jira/browse/HIVE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17205: -- Attachment: HIVE-17205.10.patch > add functional support > -- > > Key: HIVE-17205 > URL: https://issues.apache.org/jira/browse/HIVE-17205 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17205.01.patch, HIVE-17205.02.patch, > HIVE-17205.03.patch, HIVE-17205.09.patch, HIVE-17205.10.patch > > > make sure unbucketed tables can be marked transactional=true > make insert/update/delete/compaction work -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17336) Missing class 'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from Hive on Spark when inserting into hbase based table
[
https://issues.apache.org/jira/browse/HIVE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131600#comment-16131600
]
Xuefu Zhang commented on HIVE-17336:
Sorry for getting to this late. Thanks for fixing this, [~aihuaxu]. I think
making a copy of jobconf is a conscious decision, though I forgot the reason.
maybe you can find out from the change history.
> Missing class 'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from
> Hive on Spark when inserting into hbase based table
> ---
>
> Key: HIVE-17336
> URL: https://issues.apache.org/jira/browse/HIVE-17336
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 3.0.0
>
> Attachments: HIVE-17336.1.patch
>
>
> When inserting into a hbase based table from hive on spark, the following
> exception is thrown
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
> Serialization trace:
> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
> tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:183)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:201)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
> at
>
[jira] [Updated] (HIVE-17354) Fix "alter view" for incremental replication
[ https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17354: -- Status: Patch Available (was: Open) > Fix "alter view" for incremental replication > > > Key: HIVE-17354 > URL: https://issues.apache.org/jira/browse/HIVE-17354 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17354.1.patch > > > There is a bug that "alter view" operation is resulting in a view creation > operation instead of a overwriting/replacement operation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17354) Fix "alter view" for incremental replication
[ https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17354: -- Attachment: HIVE-17354.1.patch > Fix "alter view" for incremental replication > > > Key: HIVE-17354 > URL: https://issues.apache.org/jira/browse/HIVE-17354 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17354.1.patch > > > There is a bug that "alter view" operation is resulting in a view creation > operation instead of a overwriting/replacement operation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17354) Fix "alter view" for incremental replication
[ https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17354: -- Description: There is a bug that "alter view" operation is resulting in a view creation operation instead of a overwriting/replacement operation. > Fix "alter view" for incremental replication > > > Key: HIVE-17354 > URL: https://issues.apache.org/jira/browse/HIVE-17354 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > > There is a bug that "alter view" operation is resulting in a view creation > operation instead of a overwriting/replacement operation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17354) Fix "alter view" for incremental replication
[ https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17354: -- Component/s: repl > Fix "alter view" for incremental replication > > > Key: HIVE-17354 > URL: https://issues.apache.org/jira/browse/HIVE-17354 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17354) Fix "alter view" for incremental replication
[ https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li reassigned HIVE-17354: - > Fix "alter view" for incremental replication > > > Key: HIVE-17354 > URL: https://issues.apache.org/jira/browse/HIVE-17354 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"
[
https://issues.apache.org/jira/browse/HIVE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131567#comment-16131567
]
Hive QA commented on HIVE-17352:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882461/HIVE-17352.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10962 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=129)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6445/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6445/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6445/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882461 - PreCommit-HIVE-Build
> HiveSever2 error with "Illegal Operation state transition from CLOSED to
> FINISHED"
> --
>
> Key: HIVE-17352
> URL: https://issues.apache.org/jira/browse/HIVE-17352
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-17352.1.patch
>
>
> HiveSever2 error with "Illegal Operation state transition from CLOSED to
> FINISHED"
> Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle
> FINISHED in runQuery() method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17265: --- Attachment: HIVE-17265.03.patch > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-17265.02.patch, HIVE-17265.03.patch, > HIVE-17265.patch > > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement
[
https://issues.apache.org/jira/browse/HIVE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17353:
Status: Patch Available (was: Open)
The executing query is closed when a new query is run in the same statement.
That seems not necessary since we can run multiple queries in the same session.
> The ResultSets are not accessible if running multiple queries within the same
> HiveStatement
>
>
> Key: HIVE-17353
> URL: https://issues.apache.org/jira/browse/HIVE-17353
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17353.1.patch
>
>
> The following queries would fail,
> {noformat}
> ResultSet rs1 =
> stmt.executeQuery("select * from testMultipleResultSets1");
> ResultSet rs2 =
> stmt.executeQuery("select * from testMultipleResultSets2");
> rs1.next();
> rs2.next();
> {noformat}
> with the exception:
> {noformat}
> [HiveServer2-Handler-Pool: Thread-208]: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement
[
https://issues.apache.org/jira/browse/HIVE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17353:
Attachment: HIVE-17353.1.patch
> The ResultSets are not accessible if running multiple queries within the same
> HiveStatement
>
>
> Key: HIVE-17353
> URL: https://issues.apache.org/jira/browse/HIVE-17353
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17353.1.patch
>
>
> The following queries would fail,
> {noformat}
> ResultSet rs1 =
> stmt.executeQuery("select * from testMultipleResultSets1");
> ResultSet rs2 =
> stmt.executeQuery("select * from testMultipleResultSets2");
> rs1.next();
> rs2.next();
> {noformat}
> with the exception:
> {noformat}
> [HiveServer2-Handler-Pool: Thread-208]: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement
[
https://issues.apache.org/jira/browse/HIVE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-17353:
---
> The ResultSets are not accessible if running multiple queries within the same
> HiveStatement
>
>
> Key: HIVE-17353
> URL: https://issues.apache.org/jira/browse/HIVE-17353
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> The following queries would fail,
> {noformat}
> ResultSet rs1 =
> stmt.executeQuery("select * from testMultipleResultSets1");
> ResultSet rs2 =
> stmt.executeQuery("select * from testMultipleResultSets2");
> rs1.next();
> rs2.next();
> {noformat}
> with the exception:
> {noformat}
> [HiveServer2-Handler-Pool: Thread-208]: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17351) use new slider package installation command in run.sh
[
https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131510#comment-16131510
]
Hive QA commented on HIVE-17351:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882454/HIVE-17351.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10976 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge6]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6444/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6444/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6444/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882454 - PreCommit-HIVE-Build
> use new slider package installation command in run.sh
> -
>
> Key: HIVE-17351
> URL: https://issues.apache.org/jira/browse/HIVE-17351
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17351.patch
>
>
> The old syntax does not include some perf improvements in newer versions of
> Slider.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"
[ https://issues.apache.org/jira/browse/HIVE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-17352: -- Attachment: HIVE-17352.1.patch Added the case to ignore CLOSED to FINISHED state. > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > -- > > Key: HIVE-17352 > URL: https://issues.apache.org/jira/browse/HIVE-17352 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-17352.1.patch > > > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle > FINISHED in runQuery() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"
[ https://issues.apache.org/jira/browse/HIVE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-17352: -- Status: Patch Available (was: In Progress) > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > -- > > Key: HIVE-17352 > URL: https://issues.apache.org/jira/browse/HIVE-17352 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle > FINISHED in runQuery() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"
[ https://issues.apache.org/jira/browse/HIVE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17352 started by Deepak Jaiswal. - > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > -- > > Key: HIVE-17352 > URL: https://issues.apache.org/jira/browse/HIVE-17352 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle > FINISHED in runQuery() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"
[ https://issues.apache.org/jira/browse/HIVE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-17352: - > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > -- > > Key: HIVE-17352 > URL: https://issues.apache.org/jira/browse/HIVE-17352 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > HiveSever2 error with "Illegal Operation state transition from CLOSED to > FINISHED" > Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle > FINISHED in runQuery() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[
https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131449#comment-16131449
]
Hive QA commented on HIVE-17198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882431/HIVE-17198.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6443/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6443/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6443/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882431 - PreCommit-HIVE-Build
> Flaky test: TestBeeLineDriver [smb_mapjoin_7]
> -
>
> Key: HIVE-17198
> URL: https://issues.apache.org/jira/browse/HIVE-17198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Janaki Lahorani
>Assignee: Peter Vary
> Attachments: HIVE-17198.2.patch, HIVE-17198.patch
>
>
> Error:
> Exception running or analyzing the results of the query file:
> org.apache.hive.beeline.QFile@4f7b68ad
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17351) use new slider package installation command in run.sh
[ https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131441#comment-16131441 ] Gopal V commented on HIVE-17351: [~gsaha]: I assume the old syntax is deprecated since 0.92, but is still there in the docs - https://slider.incubator.apache.org/docs/getting_started.html#load LGTM - +1 > use new slider package installation command in run.sh > - > > Key: HIVE-17351 > URL: https://issues.apache.org/jira/browse/HIVE-17351 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17351.patch > > > The old syntax does not include some perf improvements in newer versions of > Slider. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17266) DecimalColumnVector64: Scaled fixed point column vector format
[
https://issues.apache.org/jira/browse/HIVE-17266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131433#comment-16131433
]
Gopal V commented on HIVE-17266:
{code}
+ public Decimal64ColumnVector() {
+this(VectorizedRowBatch.DEFAULT_SIZE, 38, 10);
+ }
{code}
>From a quick look, I think (38,10) won't be supported by this column type at
>all.
> DecimalColumnVector64: Scaled fixed point column vector format
> --
>
> Key: HIVE-17266
> URL: https://issues.apache.org/jira/browse/HIVE-17266
> Project: Hive
> Issue Type: New Feature
> Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Owen O'Malley
> Attachments: HIVE-17266.patch
>
>
> I think we should make a new type that looks like:
> class Decimal64ColumnVector extends ColumnVector {
> long[] vector;
> int precision;
> int scale;
> }
> It will be extremely fast and provide a fast conduit to ORC.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17286) Avoid expensive String serialization/deserialization for bitvectors
[ https://issues.apache.org/jira/browse/HIVE-17286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17286: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks for reviewing [~ashutoshc]! > Avoid expensive String serialization/deserialization for bitvectors > --- > > Key: HIVE-17286 > URL: https://issues.apache.org/jira/browse/HIVE-17286 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > Attachments: HIVE-17286.01.patch, HIVE-17286.02.patch, > HIVE-17286.03.patch, HIVE-17286.04.patch, HIVE-17286.05.patch, > HIVE-17286.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17351) use new slider package installation command in run.sh
[ https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131412#comment-16131412 ] Gour Saha commented on HIVE-17351: -- looks good to me > use new slider package installation command in run.sh > - > > Key: HIVE-17351 > URL: https://issues.apache.org/jira/browse/HIVE-17351 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17351.patch > > > The old syntax does not include some perf improvements in newer versions of > Slider. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims
[ https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131407#comment-16131407 ] Alan Gates commented on HIVE-17241: --- [~vgumashta], [~thejas], [~vihangk1], if you get a chance could you review this? Thanks. > Change metastore classes to not use the shims > - > > Key: HIVE-17241 > URL: https://issues.apache.org/jira/browse/HIVE-17241 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-17241.2.patch, HIVE-17241.patch > > > As part of moving the metastore into a standalone package, it will no longer > have access to the shims. This means we need to either copy them or access > the underlying Hadoop operations directly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17351) use new slider package installation command in run.sh
[ https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17351: Status: Patch Available (was: Open) > use new slider package installation command in run.sh > - > > Key: HIVE-17351 > URL: https://issues.apache.org/jira/browse/HIVE-17351 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17351.patch > > > The old syntax does not include some perf improvements in newer versions of > Slider. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17351) use new slider package installation command in run.sh
[ https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17351: Attachment: HIVE-17351.patch [~gopalv] can you take a look? [~gsaha] does this make sense? > use new slider package installation command in run.sh > - > > Key: HIVE-17351 > URL: https://issues.apache.org/jira/browse/HIVE-17351 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17351.patch > > > The old syntax does not include some perf improvements in newer versions of > Slider. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17351) use new slider package installation command in run.sh
[ https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17351: --- > use new slider package installation command in run.sh > - > > Key: HIVE-17351 > URL: https://issues.apache.org/jira/browse/HIVE-17351 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > The old syntax does not include some perf improvements in newer versions of > Slider. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17286) Avoid expensive String serialization/deserialization for bitvectors
[
https://issues.apache.org/jira/browse/HIVE-17286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131349#comment-16131349
]
Hive QA commented on HIVE-17286:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882402/HIVE-17286.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10976 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6442/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882402 - PreCommit-HIVE-Build
> Avoid expensive String serialization/deserialization for bitvectors
> ---
>
> Key: HIVE-17286
> URL: https://issues.apache.org/jira/browse/HIVE-17286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17286.01.patch, HIVE-17286.02.patch,
> HIVE-17286.03.patch, HIVE-17286.04.patch, HIVE-17286.05.patch,
> HIVE-17286.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[ https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17198: -- Attachment: HIVE-17198.2.patch Testing it again > Flaky test: TestBeeLineDriver [smb_mapjoin_7] > - > > Key: HIVE-17198 > URL: https://issues.apache.org/jira/browse/HIVE-17198 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: Peter Vary > Attachments: HIVE-17198.2.patch, HIVE-17198.patch > > > Error: > Exception running or analyzing the results of the query file: > org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[
https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131254#comment-16131254
]
Hive QA commented on HIVE-17198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882367/HIVE-17198.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10971 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=242)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6441/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6441/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882367 - PreCommit-HIVE-Build
> Flaky test: TestBeeLineDriver [smb_mapjoin_7]
> -
>
> Key: HIVE-17198
> URL: https://issues.apache.org/jira/browse/HIVE-17198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Janaki Lahorani
>Assignee: Peter Vary
> Attachments: HIVE-17198.patch
>
>
> Error:
> Exception running or analyzing the results of the query file:
> org.apache.hive.beeline.QFile@4f7b68ad
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131171#comment-16131171
]
Sergey Shelukhin commented on HIVE-17344:
-
How are those bbs allocated?
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think
> the correct solution would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17348) Remove unnecessary GenSparkUtils.java.orig file
[
https://issues.apache.org/jira/browse/HIVE-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131041#comment-16131041
]
Hive QA commented on HIVE-17348:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882366/HIVE-17348.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6440/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6440/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6440/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882366 - PreCommit-HIVE-Build
> Remove unnecessary GenSparkUtils.java.orig file
> ---
>
> Key: HIVE-17348
> URL: https://issues.apache.org/jira/browse/HIVE-17348
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17348.patch
>
>
> HIVE-17247 added an extra file, which is most probably not needed :)
> [~stakiar]? :)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131009#comment-16131009
]
Vlad Gudikov commented on HIVE-17346:
-
Yeah, it was an intended change, missed this one. Thanks!
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130932#comment-16130932
]
Hive QA commented on HIVE-17292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882365/HIVE-17292.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6439/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6439/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6439/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882365 - PreCommit-HIVE-Build
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130931#comment-16130931 ] Xuefu Zhang commented on HIVE-17347: This seems simple enough for me to review as well. +1 > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17286) Avoid expensive String serialization/deserialization for bitvectors
[ https://issues.apache.org/jira/browse/HIVE-17286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17286: --- Attachment: HIVE-17286.05.patch > Avoid expensive String serialization/deserialization for bitvectors > --- > > Key: HIVE-17286 > URL: https://issues.apache.org/jira/browse/HIVE-17286 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-17286.01.patch, HIVE-17286.02.patch, > HIVE-17286.03.patch, HIVE-17286.04.patch, HIVE-17286.05.patch, > HIVE-17286.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130851#comment-16130851 ] Peter Vary commented on HIVE-17347: --- [~lirui], [~stakiar]: Could you please review? > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130813#comment-16130813
]
Hive QA commented on HIVE-17347:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882363/HIVE-17347.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6438/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6438/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6438/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882363 - PreCommit-HIVE-Build
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is
> failing every time
>
>
> Key: HIVE-17347
> URL: https://issues.apache.org/jira/browse/HIVE-17347
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17347.patch
>
>
> As [~lirui] identified there was a missing file from this patch: HIVE-17247 -
> HoS DPP: UDFs on the partition column side does not evaluate correctly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17183) Disable rename operations during bootstrap dump
[ https://issues.apache.org/jira/browse/HIVE-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17183: Description: Currently, bootstrap dump shall lead to data loss when any rename happens while dump in progress. *Scenario:* - Fetch table names (T1 and T2) - Dump table T1 - Rename table T2 to T3 generates RENAME event - Dump table T2 is noop as table doesn’t exist. - In target after load, it only have T1. - Apply RENAME event will fail as T2 doesn’t exist in target. This feature can be supported in next phase development as it need proper design to keep track of renamed tables/partitions. So, for time being, we shall disable rename operations when bootstrap dump in progress to avoid any inconsistent state. was: Currently, bootstrap dump shall lead to data loss when any rename happens while dump in progress. This feature can be supported in next phase development as it need proper design to keep track of renamed tables/partitions. So, for time being, we shall disable rename operations when bootstrap dump in progress to avoid any inconsistent state. > Disable rename operations during bootstrap dump > --- > > Key: HIVE-17183 > URL: https://issues.apache.org/jira/browse/HIVE-17183 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17183.01.patch, HIVE-17183.02.patch, > HIVE-17183.03.patch > > > Currently, bootstrap dump shall lead to data loss when any rename happens > while dump in progress. > *Scenario:* > - Fetch table names (T1 and T2) > - Dump table T1 > - Rename table T2 to T3 generates RENAME event > - Dump table T2 is noop as table doesn’t exist. > - In target after load, it only have T1. > - Apply RENAME event will fail as T2 doesn’t exist in target. > This feature can be supported in next phase development as it need proper > design to keep track of renamed tables/partitions. > So, for time being, we shall disable rename operations when bootstrap dump in > progress to avoid any inconsistent state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17336) Missing class 'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from Hive on Spark when inserting into hbase based table
[
https://issues.apache.org/jira/browse/HIVE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17336:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks Vihang for reviewing.
> Missing class 'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from
> Hive on Spark when inserting into hbase based table
> ---
>
> Key: HIVE-17336
> URL: https://issues.apache.org/jira/browse/HIVE-17336
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 3.0.0
>
> Attachments: HIVE-17336.1.patch
>
>
> When inserting into a hbase based table from hive on spark, the following
> exception is thrown
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
> Serialization trace:
> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
> tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:183)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:201)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
>
[jira] [Commented] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130705#comment-16130705
]
Hive QA commented on HIVE-17346:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882359/HIVE-17346.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge7]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6437/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6437/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6437/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882359 - PreCommit-HIVE-Build
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result
[jira] [Commented] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130680#comment-16130680 ] Sergio Peña commented on HIVE-14747: Thanks [~zsombor.klara]. The patch looks great. +1 > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch, > HIVE-14747.06.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[ https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17198: -- Attachment: HIVE-17198.patch Attaching a do-nothing patch to test. > Flaky test: TestBeeLineDriver [smb_mapjoin_7] > - > > Key: HIVE-17198 > URL: https://issues.apache.org/jira/browse/HIVE-17198 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: Peter Vary > Attachments: HIVE-17198.patch > > > Error: > Exception running or analyzing the results of the query file: > org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[ https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17198: -- Status: Patch Available (was: Open) > Flaky test: TestBeeLineDriver [smb_mapjoin_7] > - > > Key: HIVE-17198 > URL: https://issues.apache.org/jira/browse/HIVE-17198 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: Peter Vary > Attachments: HIVE-17198.patch > > > Error: > Exception running or analyzing the results of the query file: > org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[ https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-17198: - Assignee: Peter Vary > Flaky test: TestBeeLineDriver [smb_mapjoin_7] > - > > Key: HIVE-17198 > URL: https://issues.apache.org/jira/browse/HIVE-17198 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: Peter Vary > > Error: > Exception running or analyzing the results of the query file: > org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
[ https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130521#comment-16130521 ] Janaki Lahorani commented on HIVE-17198: Hi [~pvary], I found this test to fail intermittently when tests were run for uploaded patches. I think the results are no longer available. We will have to rerun the test to see if it still reproduces. Thanks, Janaki. > Flaky test: TestBeeLineDriver [smb_mapjoin_7] > - > > Key: HIVE-17198 > URL: https://issues.apache.org/jira/browse/HIVE-17198 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani > > Error: > Exception running or analyzing the results of the query file: > org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17348) Remove unnecessary GenSparkUtils.java.orig file
[ https://issues.apache.org/jira/browse/HIVE-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17348: -- Attachment: HIVE-17348.patch > Remove unnecessary GenSparkUtils.java.orig file > --- > > Key: HIVE-17348 > URL: https://issues.apache.org/jira/browse/HIVE-17348 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17348.patch > > > HIVE-17247 added an extra file, which is most probably not needed :) > [~stakiar]? :) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17348) Remove unnecessary GenSparkUtils.java.orig file
[ https://issues.apache.org/jira/browse/HIVE-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17348: -- Status: Patch Available (was: Open) > Remove unnecessary GenSparkUtils.java.orig file > --- > > Key: HIVE-17348 > URL: https://issues.apache.org/jira/browse/HIVE-17348 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17348.patch > > > HIVE-17247 added an extra file, which is most probably not needed :) > [~stakiar]? :) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17348) Remove unnecessary GenSparkUtils.java.orig file
[ https://issues.apache.org/jira/browse/HIVE-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-17348: - > Remove unnecessary GenSparkUtils.java.orig file > --- > > Key: HIVE-17348 > URL: https://issues.apache.org/jira/browse/HIVE-17348 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > > HIVE-17247 added an extra file, which is most probably not needed :) > [~stakiar]? :) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17292:
--
Attachment: HIVE-17292.6.patch
The patch contains the following changes:
- Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the
requested 2 executors.
- Changing QTestUtil.setSparkSession, so we will wait until every executor is
available, not only the 1st.
- Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided
paralellism in case of local spark.master too.
- Regenerating golden files (numReducers, and number of files changed in the
explain plans)
The change contains 2 golden file changes
(spark_dynamic_partition_pruning_mapjoin_only.q.out,
spark_dynamic_partition_pruning.q.out) which are containing other neccessary
changes for a green run, so this patch should be regenerated after their
corresponding jiras are solved (HIVE-17347, HIVE-17346)
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17347: -- Status: Patch Available (was: Open) > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17347: -- Attachment: HIVE-17347.patch Regenerated golden file > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17347.patch > > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Attachment: HIVE-17319.draft.patch Attaching the first draft of the patch for BoneCP. [~thejas], [~ekoifman], [~pvary] I hope this will clarify a bit what I had in mind. If you have the time please feel free to comment on the patch here or on reviewboard. > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time
[ https://issues.apache.org/jira/browse/HIVE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-17347: - > TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is > failing every time > > > Key: HIVE-17347 > URL: https://issues.apache.org/jira/browse/HIVE-17347 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > > As [~lirui] identified there was a missing file from this patch: HIVE-17247 - > HoS DPP: UDFs on the partition column side does not evaluate correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17346:
--
Affects Version/s: 3.0.0
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17346:
--
Status: Patch Available (was: Open)
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17346:
--
Attachment: HIVE-17346.patch
Here is the patch, if only golden file regeneration is needed
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17346.patch
>
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130420#comment-16130420
]
Peter Vary commented on HIVE-16948:
---
I do not find my comment on the review board, so I leave a comment here too:
I think {{spark_dynamic_partition_pruning.q.out}} changes are caused by
HIVE-17148 - "Incorrect result for Hive join query with COALESCE in WHERE
condition". Created HIVE-17346 to track the progress there
> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch,
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.patch
>
>
> in
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
> in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all
> select distinct(ds) as ds from srcpart) s where s.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
> DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
> Vertices:
> Map 10
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: max(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: min(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11
> Reduce Operator Tree:
> Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: _col0 is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: _col0 (type: string)
>
[jira] [Assigned] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time
[
https://issues.apache.org/jira/browse/HIVE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-17346:
-
> TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing
> every time
> ---
>
> Key: HIVE-17346
> URL: https://issues.apache.org/jira/browse/HIVE-17346
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> The TestMiniSparkOnYarnCliDriver.testCliDriver -
> spark_dynamic_partition_pruning is failing with this diff:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing spark_dynamic_partition_pruning.q
> 714c714
> < filterExpr: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > filterExpr: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 717c717
> < predicate: ((date = '2008-04-08') and abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type:
> boolean)
> ---
> > predicate: ((date = '2008-04-08') and ds is not null)
> > (type: boolean)
> 749c749
> < filterExpr: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > filterExpr: ds is not null (type: boolean)
> 751,752c751,753
> < Filter Operator
> < predicate: abs(((-
> UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null (type:
> boolean)
> ---
> > Select Operator
> > expressions: ds (type: string)
> > outputColumnNames: _col0
> 754,756c755,758
> < Select Operator
> < expressions: ds (type: string)
> < outputColumnNames: _col0
> ---
> > Reduce Output Operator
> > key expressions: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> > sort order: +
> > Map-reduce partition columns: abs(((-
> > UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> 758,762d759
> < Reduce Output Operator
> < key expressions: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < sort order: +
> < Map-reduce partition columns: abs(((-
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
> < Statistics: Num rows: 2000 Data size: 21248 Basic
> stats: COMPLETE Column stats: NONE
> 767c764
> <
> Output was too long and had to be truncated...
> {code}
> I think it is caused by:
> HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE
> condition
> [~allgoodok]: Am I right? Is it an intended change and only the golden file
> regeneration is needed?
> Thanks,
> Peter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130380#comment-16130380
]
Hive QA commented on HIVE-17344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882323/HIVE-17344.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6436/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6436/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6436/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882323 - PreCommit-HIVE-Build
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think
> the correct solution would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16713) Remove wrong judgments of single and double quotes in "--" comments
[ https://issues.apache.org/jira/browse/HIVE-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130323#comment-16130323 ] Alister Roberts edited comment on HIVE-16713 at 8/17/17 12:33 PM: -- This behaviour is a regression - a comment with an embedded apostrophe worked fine in Hive 2.1.0, but fails in Hive 2.3.0 for this reason. It should be noted that the behaviour does not manifest itself when testing using the Hive CLI - I guess it does some processing that masks the problem. was (Author: gatelogic): This behaviour is a regression - a comment with an embedded apostrophe worked fine in Hive 2.1.0, but fails in Hive 2.3.0 for this reason. > Remove wrong judgments of single and double quotes in "--" comments > --- > > Key: HIVE-16713 > URL: https://issues.apache.org/jira/browse/HIVE-16713 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Liu Chunxiao >Assignee: Liu Chunxiao > > single quote in "--" comments will lead to parse exception > e.g. sql files which contain the following: > select --...user's... > count(*) from aa; > will regard the single quote in comment "user's" as a quote in sql statement. > So ";" after aa will be regarded as a ";" in single quote and will not be > split by the parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16713) Remove wrong judgments of single and double quotes in "--" comments
[ https://issues.apache.org/jira/browse/HIVE-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alister Roberts updated HIVE-16713: --- Priority: Major (was: Minor) > Remove wrong judgments of single and double quotes in "--" comments > --- > > Key: HIVE-16713 > URL: https://issues.apache.org/jira/browse/HIVE-16713 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Liu Chunxiao >Assignee: Liu Chunxiao > > single quote in "--" comments will lead to parse exception > e.g. sql files which contain the following: > select --...user's... > count(*) from aa; > will regard the single quote in comment "user's" as a quote in sql statement. > So ";" after aa will be regarded as a ";" in single quote and will not be > split by the parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16713) Remove wrong judgments of single and double quotes in "--" comments
[ https://issues.apache.org/jira/browse/HIVE-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130323#comment-16130323 ] Alister Roberts commented on HIVE-16713: This behaviour is a regression - a comment with an embedded apostrophe worked fine in Hive 2.1.0, but fails in Hive 2.3.0 for this reason. > Remove wrong judgments of single and double quotes in "--" comments > --- > > Key: HIVE-16713 > URL: https://issues.apache.org/jira/browse/HIVE-16713 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Liu Chunxiao >Assignee: Liu Chunxiao >Priority: Minor > > single quote in "--" comments will lead to parse exception > e.g. sql files which contain the following: > select --...user's... > count(*) from aa; > will regard the single quote in comment "user's" as a quote in sql statement. > So ";" after aa will be regarded as a ";" in single quote and will not be > split by the parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130313#comment-16130313
]
Hive QA commented on HIVE-17100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882306/HIVE-17100.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6435/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6435/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6435/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882306 - PreCommit-HIVE-Build
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch,
> HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch
>
>
> It is necessary to log the progress the replication tasks in a structured
> manner as follows.
> *+Bootstrap Dump:+*
> * At the start of bootstrap dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (BOOTSTRAP)
> * (Estimated) Total number of tables/views to dump
> * (Estimated) Total number of functions to dump.
> * Dump Start Time{color}
> * After each table dump, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table dump end time
> * Table dump progress. Format is Table sequence no/(Estimated) Total number
> of tables and views.{color}
> * After each function dump, will add a log as follows
> {color:#59afe1}* Function Name
> * Function dump end time
> * Function dump progress. Format is Function sequence no/(Estimated) Total
> number of functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> dump.
> {color:#59afe1}* Database Name.
> * Dump Type (BOOTSTRAP).
> * Dump End Time.
> * (Actual) Total number of tables/views dumped.
> * (Actual) Total number of functions dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The actual and estimated number of tables/functions may not match if
> any table/function is dropped when dump in progress.
> *+Bootstrap Load:+*
> * At the start of bootstrap load, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump directory
> * Load Type (BOOTSTRAP)
> * Total number of tables/views to load
> * Total number of functions to load.
> * Load Start Time{color}
> * After each table load, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table load completion time
> * Table load progress. Format is Table sequence no/Total number of tables and
> views.{color}
> * After each function load, will add a log as follows
> {color:#59afe1}* Function Name
> * Function load completion time
> * Function load progress. Format is Function sequence no/Total number of
> functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> load.
> {color:#59afe1}* Database Name.
> * Load Type (BOOTSTRAP).
> * Load End Time.
> * Total number of tables/views loaded.
> * Total number of functions loaded.
> * Last Repl ID of the loaded
[jira] [Updated] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HIVE-17344:
-
Description:
Orc footer cache has a calculation of memory usage:
{code:java}
public int getMemoryUsage() {
return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
(semi-arbitrary).
}
{code}
ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think the
correct solution would be bb.capacity.
was:
Orc footer cache has a calculation of memory usage:
{code:java}
public int getMemoryUsage() {
return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
(semi-arbitrary).
}
{code}
ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
allowing this cache to grow until MAXWEIGHT/100. I think the correct solution
would be bb.capacity.
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think
> the correct solution would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130285#comment-16130285
]
Zoltan Haindrich commented on HIVE-17344:
-
+1 pending tests
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache to grow until MAXWEIGHT/100. I think the correct solution
> would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HIVE-17344:
-
Description:
Orc footer cache has a calculation of memory usage:
{code:java}
public int getMemoryUsage() {
return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
(semi-arbitrary).
}
{code}
ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
allowing this cache to grow until MAXWEIGHT/100. I think the correct solution
would be bb.capacity.
was:
Orc footer cache has a calculation of memory usage:
{code:java}
public int getMemoryUsage() {
return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
(semi-arbitrary).
}
{code}
ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
allowing this cache to grow until MAXSIZE/100. I think the correct solution
would be bb.capacity.
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache to grow until MAXWEIGHT/100. I think the correct solution
> would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HIVE-17344:
-
Status: Patch Available (was: Open)
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache to grow until MAXSIZE/100. I think the correct solution
> would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HIVE-17344:
-
Attachment: HIVE-17344.patch
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache to grow until MAXSIZE/100. I think the correct solution
> would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130250#comment-16130250 ] Janos Gub commented on HIVE-16133: -- I think there is an error in the computation of the cache size. I opened a new jira ticket:https://issues.apache.org/jira/browse/HIVE-17344 > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Labels: TODOC2.2 > Fix For: 2.3.0 > > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.04.patch, > HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub reassigned HIVE-17344:
> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
> return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus
> allowing this cache to grow until MAXSIZE/100. I think the correct solution
> would be bb.capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[
https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130242#comment-16130242
]
Hive QA commented on HIVE-14747:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882289/HIVE-14747.06.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6434/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6434/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6434/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882289 - PreCommit-HIVE-Build
> Remove JAVA paths from profiles by sending them from ptest-client
> -
>
> Key: HIVE-14747
> URL: https://issues.apache.org/jira/browse/HIVE-14747
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch,
> HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch,
> HIVE-14747.06.patch
>
>
> Hive ptest uses some properties files per branch that contain information
> about how to execute the tests.
> This profile includes JAVA paths to build and execute the tests. We should
> get rid of these by passing such information from Jenkins to the
> ptest-server. In case a profile needs a different java version, then we can
> create a specific Jenkins job for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17314) LazySimpleSerializeWrite.writeString() contains if with an empty body
[
https://issues.apache.org/jira/browse/HIVE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130154#comment-16130154
]
Hive QA commented on HIVE-17314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882285/HIVE-17314.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10977 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=222)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6433/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6433/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6433/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882285 - PreCommit-HIVE-Build
> LazySimpleSerializeWrite.writeString() contains if with an empty body
> -
>
> Key: HIVE-17314
> URL: https://issues.apache.org/jira/browse/HIVE-17314
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17314.patch
>
>
> Looking at the LazySimpleSerializeWrite.java I found odd 'if':
> Lines 234-235:
> {code:java}
> if (v.equals(nullSequenceBytes)) {
> }
> {code}
> Seems like either something is missed there or this 'if' could be dropped out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-17100:
Status: Patch Available (was: Open)
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch,
> HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch
>
>
> It is necessary to log the progress the replication tasks in a structured
> manner as follows.
> *+Bootstrap Dump:+*
> * At the start of bootstrap dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (BOOTSTRAP)
> * (Estimated) Total number of tables/views to dump
> * (Estimated) Total number of functions to dump.
> * Dump Start Time{color}
> * After each table dump, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table dump end time
> * Table dump progress. Format is Table sequence no/(Estimated) Total number
> of tables and views.{color}
> * After each function dump, will add a log as follows
> {color:#59afe1}* Function Name
> * Function dump end time
> * Function dump progress. Format is Function sequence no/(Estimated) Total
> number of functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> dump.
> {color:#59afe1}* Database Name.
> * Dump Type (BOOTSTRAP).
> * Dump End Time.
> * (Actual) Total number of tables/views dumped.
> * (Actual) Total number of functions dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The actual and estimated number of tables/functions may not match if
> any table/function is dropped when dump in progress.
> *+Bootstrap Load:+*
> * At the start of bootstrap load, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump directory
> * Load Type (BOOTSTRAP)
> * Total number of tables/views to load
> * Total number of functions to load.
> * Load Start Time{color}
> * After each table load, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table load completion time
> * Table load progress. Format is Table sequence no/Total number of tables and
> views.{color}
> * After each function load, will add a log as follows
> {color:#59afe1}* Function Name
> * Function load completion time
> * Function load progress. Format is Function sequence no/Total number of
> functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> load.
> {color:#59afe1}* Database Name.
> * Load Type (BOOTSTRAP).
> * Load End Time.
> * Total number of tables/views loaded.
> * Total number of functions loaded.
> * Last Repl ID of the loaded database.{color}
> *+Incremental Dump:+*
> * At the start of database dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (INCREMENTAL)
> * (Estimated) Total number of events to dump.
> * Dump Start Time{color}
> * After each event dump, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event dump end time
> * Event dump progress. Format is Event sequence no/ (Estimated) Total number
> of events.{color}
> * After completion of all event dumps, will add a log as follows.
> {color:#59afe1}* Database Name.
> * Dump Type (INCREMENTAL).
> * Dump End Time.
> * (Actual) Total number of events dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The estimated number of events can be terribly inaccurate with actual
> number as we don’t have the number of events upfront until we read from
> metastore NotificationEvents table.
> *+Incremental Load:+*
> * At the start of incremental load, will add one log with below details.
> {color:#59afe1}* Target Database Name
> * Dump directory
> * Load Type (INCREMENTAL)
> * Total number of events to load
> * Load Start Time{color}
> * After each event load, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event load end time
> * Event load progress. Format is Event sequence no/ Total number of
> events.{color}
> * After completion of all event loads, will add a log as follows to
> consolidate the load.
> {color:#59afe1}* Target Database Name.
> * Load Type (INCREMENTAL).
> * Load End Time.
> * Total number of events loaded.
> * Last Repl ID of the loaded database.{color}
--
This message was sent by
[jira] [Updated] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-17100:
Attachment: HIVE-17100.05.patch
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch,
> HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch
>
>
> It is necessary to log the progress the replication tasks in a structured
> manner as follows.
> *+Bootstrap Dump:+*
> * At the start of bootstrap dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (BOOTSTRAP)
> * (Estimated) Total number of tables/views to dump
> * (Estimated) Total number of functions to dump.
> * Dump Start Time{color}
> * After each table dump, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table dump end time
> * Table dump progress. Format is Table sequence no/(Estimated) Total number
> of tables and views.{color}
> * After each function dump, will add a log as follows
> {color:#59afe1}* Function Name
> * Function dump end time
> * Function dump progress. Format is Function sequence no/(Estimated) Total
> number of functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> dump.
> {color:#59afe1}* Database Name.
> * Dump Type (BOOTSTRAP).
> * Dump End Time.
> * (Actual) Total number of tables/views dumped.
> * (Actual) Total number of functions dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The actual and estimated number of tables/functions may not match if
> any table/function is dropped when dump in progress.
> *+Bootstrap Load:+*
> * At the start of bootstrap load, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump directory
> * Load Type (BOOTSTRAP)
> * Total number of tables/views to load
> * Total number of functions to load.
> * Load Start Time{color}
> * After each table load, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table load completion time
> * Table load progress. Format is Table sequence no/Total number of tables and
> views.{color}
> * After each function load, will add a log as follows
> {color:#59afe1}* Function Name
> * Function load completion time
> * Function load progress. Format is Function sequence no/Total number of
> functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> load.
> {color:#59afe1}* Database Name.
> * Load Type (BOOTSTRAP).
> * Load End Time.
> * Total number of tables/views loaded.
> * Total number of functions loaded.
> * Last Repl ID of the loaded database.{color}
> *+Incremental Dump:+*
> * At the start of database dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (INCREMENTAL)
> * (Estimated) Total number of events to dump.
> * Dump Start Time{color}
> * After each event dump, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event dump end time
> * Event dump progress. Format is Event sequence no/ (Estimated) Total number
> of events.{color}
> * After completion of all event dumps, will add a log as follows.
> {color:#59afe1}* Database Name.
> * Dump Type (INCREMENTAL).
> * Dump End Time.
> * (Actual) Total number of events dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The estimated number of events can be terribly inaccurate with actual
> number as we don’t have the number of events upfront until we read from
> metastore NotificationEvents table.
> *+Incremental Load:+*
> * At the start of incremental load, will add one log with below details.
> {color:#59afe1}* Target Database Name
> * Dump directory
> * Load Type (INCREMENTAL)
> * Total number of events to load
> * Load Start Time{color}
> * After each event load, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event load end time
> * Event load progress. Format is Event sequence no/ Total number of
> events.{color}
> * After completion of all event loads, will add a log as follows to
> consolidate the load.
> {color:#59afe1}* Target Database Name.
> * Load Type (INCREMENTAL).
> * Load End Time.
> * Total number of events loaded.
> * Last Repl ID of the loaded database.{color}
--
This message was sent by
[jira] [Updated] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-17100:
Attachment: (was: HIVE-17100.05.patch)
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch,
> HIVE-17100.03.patch, HIVE-17100.04.patch
>
>
> It is necessary to log the progress the replication tasks in a structured
> manner as follows.
> *+Bootstrap Dump:+*
> * At the start of bootstrap dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (BOOTSTRAP)
> * (Estimated) Total number of tables/views to dump
> * (Estimated) Total number of functions to dump.
> * Dump Start Time{color}
> * After each table dump, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table dump end time
> * Table dump progress. Format is Table sequence no/(Estimated) Total number
> of tables and views.{color}
> * After each function dump, will add a log as follows
> {color:#59afe1}* Function Name
> * Function dump end time
> * Function dump progress. Format is Function sequence no/(Estimated) Total
> number of functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> dump.
> {color:#59afe1}* Database Name.
> * Dump Type (BOOTSTRAP).
> * Dump End Time.
> * (Actual) Total number of tables/views dumped.
> * (Actual) Total number of functions dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The actual and estimated number of tables/functions may not match if
> any table/function is dropped when dump in progress.
> *+Bootstrap Load:+*
> * At the start of bootstrap load, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump directory
> * Load Type (BOOTSTRAP)
> * Total number of tables/views to load
> * Total number of functions to load.
> * Load Start Time{color}
> * After each table load, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table load completion time
> * Table load progress. Format is Table sequence no/Total number of tables and
> views.{color}
> * After each function load, will add a log as follows
> {color:#59afe1}* Function Name
> * Function load completion time
> * Function load progress. Format is Function sequence no/Total number of
> functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> load.
> {color:#59afe1}* Database Name.
> * Load Type (BOOTSTRAP).
> * Load End Time.
> * Total number of tables/views loaded.
> * Total number of functions loaded.
> * Last Repl ID of the loaded database.{color}
> *+Incremental Dump:+*
> * At the start of database dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (INCREMENTAL)
> * (Estimated) Total number of events to dump.
> * Dump Start Time{color}
> * After each event dump, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event dump end time
> * Event dump progress. Format is Event sequence no/ (Estimated) Total number
> of events.{color}
> * After completion of all event dumps, will add a log as follows.
> {color:#59afe1}* Database Name.
> * Dump Type (INCREMENTAL).
> * Dump End Time.
> * (Actual) Total number of events dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The estimated number of events can be terribly inaccurate with actual
> number as we don’t have the number of events upfront until we read from
> metastore NotificationEvents table.
> *+Incremental Load:+*
> * At the start of incremental load, will add one log with below details.
> {color:#59afe1}* Target Database Name
> * Dump directory
> * Load Type (INCREMENTAL)
> * Total number of events to load
> * Load Start Time{color}
> * After each event load, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event load end time
> * Event load progress. Format is Event sequence no/ Total number of
> events.{color}
> * After completion of all event loads, will add a log as follows to
> consolidate the load.
> {color:#59afe1}* Target Database Name.
> * Load Type (INCREMENTAL).
> * Load End Time.
> * Total number of events loaded.
> * Last Repl ID of the loaded database.{color}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-17100:
Status: Open (was: Patch Available)
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch,
> HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch
>
>
> It is necessary to log the progress the replication tasks in a structured
> manner as follows.
> *+Bootstrap Dump:+*
> * At the start of bootstrap dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (BOOTSTRAP)
> * (Estimated) Total number of tables/views to dump
> * (Estimated) Total number of functions to dump.
> * Dump Start Time{color}
> * After each table dump, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table dump end time
> * Table dump progress. Format is Table sequence no/(Estimated) Total number
> of tables and views.{color}
> * After each function dump, will add a log as follows
> {color:#59afe1}* Function Name
> * Function dump end time
> * Function dump progress. Format is Function sequence no/(Estimated) Total
> number of functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> dump.
> {color:#59afe1}* Database Name.
> * Dump Type (BOOTSTRAP).
> * Dump End Time.
> * (Actual) Total number of tables/views dumped.
> * (Actual) Total number of functions dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The actual and estimated number of tables/functions may not match if
> any table/function is dropped when dump in progress.
> *+Bootstrap Load:+*
> * At the start of bootstrap load, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump directory
> * Load Type (BOOTSTRAP)
> * Total number of tables/views to load
> * Total number of functions to load.
> * Load Start Time{color}
> * After each table load, will add a log as follows
> {color:#59afe1}* Table/View Name
> * Type (TABLE/VIEW/MATERIALIZED_VIEW)
> * Table load completion time
> * Table load progress. Format is Table sequence no/Total number of tables and
> views.{color}
> * After each function load, will add a log as follows
> {color:#59afe1}* Function Name
> * Function load completion time
> * Function load progress. Format is Function sequence no/Total number of
> functions.{color}
> * After completion of all dumps, will add a log as follows to consolidate the
> load.
> {color:#59afe1}* Database Name.
> * Load Type (BOOTSTRAP).
> * Load End Time.
> * Total number of tables/views loaded.
> * Total number of functions loaded.
> * Last Repl ID of the loaded database.{color}
> *+Incremental Dump:+*
> * At the start of database dump, will add one log with below details.
> {color:#59afe1}* Database Name
> * Dump Type (INCREMENTAL)
> * (Estimated) Total number of events to dump.
> * Dump Start Time{color}
> * After each event dump, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event dump end time
> * Event dump progress. Format is Event sequence no/ (Estimated) Total number
> of events.{color}
> * After completion of all event dumps, will add a log as follows.
> {color:#59afe1}* Database Name.
> * Dump Type (INCREMENTAL).
> * Dump End Time.
> * (Actual) Total number of events dumped.
> * Dump Directory.
> * Last Repl ID of the dump.{color}
> *Note:* The estimated number of events can be terribly inaccurate with actual
> number as we don’t have the number of events upfront until we read from
> metastore NotificationEvents table.
> *+Incremental Load:+*
> * At the start of incremental load, will add one log with below details.
> {color:#59afe1}* Target Database Name
> * Dump directory
> * Load Type (INCREMENTAL)
> * Total number of events to load
> * Load Start Time{color}
> * After each event load, will add a log as follows
> {color:#59afe1}* Event ID
> * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc)
> * Event load end time
> * Event load progress. Format is Event sequence no/ Total number of
> events.{color}
> * After completion of all event loads, will add a log as follows to
> consolidate the load.
> {color:#59afe1}* Target Database Name.
> * Load Type (INCREMENTAL).
> * Load End Time.
> * Total number of events loaded.
> * Last Repl ID of the loaded database.{color}
--
This message was sent by
[jira] [Assigned] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17319: -- Assignee: Barna Zsombor Klara > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17319 started by Barna Zsombor Klara. -- > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[
https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130081#comment-16130081
]
Hive QA commented on HIVE-17100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882282/HIVE-17100.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6432/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6432/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6432/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-6432/succeeded/33-TestCliDriver-ql_rewrite_gbtoidx.q-json_serde1.q-constantPropWhen.q-and-27-more,
remoteFile=/home/hiveptest/104.155.155.39-hiveptest-1/logs/,
getExitCode()=255, getException()=null, getUser()=hiveptest,
getHost()=104.155.155.39, getInstance()=1]: 'Warning: Permanently added
'104.155.155.39' (ECDSA) to the list of known hosts.
receiving incremental file list
./
TEST-33-TestCliDriver-ql_rewrite_gbtoidx.q-json_serde1.q-constantPropWhen.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
0 0%0.00kB/s0:00:00
8,686 100%8.28MB/s0:00:00 (xfr#1, to-chk=5/7)
maven-test.txt
0 0%0.00kB/s0:00:00
48,240 100%1.35MB/s0:00:00 (xfr#2, to-chk=4/7)
logs/
logs/derby.log
0 0%0.00kB/s0:00:00
1,004 100% 28.84kB/s0:00:00 (xfr#3, to-chk=1/7)
logs/hive.log
0 0%0.00kB/s0:00:00
48,103,424 2% 45.74MB/s0:00:34
107,479,040 6% 51.05MB/s0:00:30
165,445,632 9% 52.45MB/s0:00:28
225,116,160 13% 53.44MB/s0:00:26
284,950,528 16% 56.29MB/s0:00:24
343,638,016 20% 56.21MB/s0:00:23
402,194,432 23% 56.36MB/s0:00:22
461,766,656 27% 56.35MB/s0:00:21
521,437,184 31% 56.19MB/s0:00:20
578,584,576 34% 55.83MB/s0:00:19
637,534,208 37% 55.91MB/s0:00:18
696,745,984 41% 55.94MB/s0:00:17
755,859,456 44% 55.90MB/s0:00:16
814,743,552 48% 56.32MB/s0:00:15
873,857,024 51% 56.37MB/s0:00:13
933,265,408 55% 56.31MB/s0:00:12
993,099,776 59% 56.34MB/s0:00:11
1,052,639,232 62% 56.38MB/s0:00:10
1,112,211,456 66% 56.36MB/s0:00:09
1,172,144,128 69% 56.46MB/s0:00:08
1,232,011,264 73% 56.49MB/s0:00:07
1,291,583,488 76% 56.50MB/s0:00:06
1,351,581,696 80% 56.56MB/s0:00:05
1,411,383,296 83% 56.56MB/s0:00:04
1,470,857,216 87% 56.48MB/s0:00:03
1,530,396,672 90% 56.47MB/s0:00:02
1,590,296,576 94% 56.46MB/s0:00:01 Timeout, server 104.155.155.39 not
responding.
rsync: connection unexpectedly closed (1621310331 bytes received so far)
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[receiver=3.1.1]
rsync: connection unexpectedly closed (447 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.1]
ssh: connect to host 104.155.155.39 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.155.155.39 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.155.155.39 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.155.155.39 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12882282 - PreCommit-HIVE-Build
> Improve HS2 operation logs for REPL commands.
> -
>
> Key: HIVE-17100
> URL: https://issues.apache.org/jira/browse/HIVE-17100
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>
[jira] [Commented] (HIVE-17313) Potentially possible 'case fall through' in the ObjectInspectorConverters
[
https://issues.apache.org/jira/browse/HIVE-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130079#comment-16130079
]
Zoltan Haindrich commented on HIVE-17313:
-
did it cause any known problems ?
+1
seems like the jenkins logs are already removedthe
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDate2 might be
related
> Potentially possible 'case fall through' in the ObjectInspectorConverters
> -
>
> Key: HIVE-17313
> URL: https://issues.apache.org/jira/browse/HIVE-17313
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-17313.patch
>
>
> Lines 103-110:
> {code:java}
> case STRING:
> if (outputOI instanceof WritableStringObjectInspector) {
> return new PrimitiveObjectInspectorConverter.TextConverter(
> inputOI);
> } else if (outputOI instanceof JavaStringObjectInspector) {
> return new PrimitiveObjectInspectorConverter.StringConverter(
> inputOI);
> }
> case CHAR:
> {code}
> De-facto it should work correctly since outputOI is either an instance of
> WritableStringObjectInspector or JavaStringObjectInspector, but it would be
> better to rewrite this case to avoid possible fall through.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14747: --- Attachment: HIVE-14747.06.patch > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch, > HIVE-14747.06.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17331) Path must be used as key type of the pathToAlises
[
https://issues.apache.org/jira/browse/HIVE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130070#comment-16130070
]
Zoltan Haindrich commented on HIVE-17331:
-
yes...this is really broken by HIVE-12244 - seems like this part does not have
enough coverage...
Could you intern the path the same time you create it - or reuse the path
instance ; this would increase readablily a bit.
note: it would seem to be a good idea to move the interning to
{{addPathToAlias}}...it just makes it more complicated to read all these...and
it only make sense when it is added... - possibly in a follow up
> Path must be used as key type of the pathToAlises
> -
>
> Key: HIVE-17331
> URL: https://issues.apache.org/jira/browse/HIVE-17331
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17331.patch
>
>
> This code uses String instead of Path as key type of the pathToAliases map,
> so seems like get(String) always null.
> +*GenMapRedUtils.java*+
> {code:java}
> for (int pos = 0; pos < size; pos++) {
> String taskTmpDir = taskTmpDirLst.get(pos);
> TableDesc tt_desc = tt_descLst.get(pos);
> MapWork mWork = plan.getMapWork();
> if (mWork.getPathToAliases().get(taskTmpDir) == null) {
> taskTmpDir = taskTmpDir.intern();
> Path taskTmpDirPath =
> StringInternUtils.internUriStringsInPath(new Path(taskTmpDir));
> mWork.removePathToAlias(taskTmpDirPath);
> mWork.addPathToAlias(taskTmpDirPath, taskTmpDir);
> mWork.addPathToPartitionInfo(taskTmpDirPath, new
> PartitionDesc(tt_desc, null));
> mWork.getAliasToWork().put(taskTmpDir, topOperators.get(pos));
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17314) LazySimpleSerializeWrite.writeString() contains if with an empty body
[
https://issues.apache.org/jira/browse/HIVE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130065#comment-16130065
]
Oleg Danilov commented on HIVE-17314:
-
[~kgyrtkirk], uploaded.
> LazySimpleSerializeWrite.writeString() contains if with an empty body
> -
>
> Key: HIVE-17314
> URL: https://issues.apache.org/jira/browse/HIVE-17314
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17314.patch
>
>
> Looking at the LazySimpleSerializeWrite.java I found odd 'if':
> Lines 234-235:
> {code:java}
> if (v.equals(nullSequenceBytes)) {
> }
> {code}
> Seems like either something is missed there or this 'if' could be dropped out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
