[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396993#comment-15396993
]
Amareshwari Sriramadasu commented on HIVE-14264:
[~gszadovszky] We are running queries on hive server on version 2.1 and using
MR execution engine. The tables are partitioned and the issue mostly happens
when col is not projected but it is only in filter clause with 'IS NOT NULL'
filter. But we have seen failures in some cases when column is projected as
well.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[
https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396984#comment-15396984
]
Hive QA commented on HIVE-14259:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820303/HIVE-14259.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/668/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/668/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-668/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7938668 HIVE-14364. Update timeouts for llap comparator tests.
(Siddharth Seth, reviewed by Gunther Hagleitner)
+ git clean -f -d
Removing b/
Removing ql/src/test/queries/clientpositive/orc_schema_evolution_float.q
Removing ql/src/test/results/clientpositive/orc_schema_evolution_float.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7938668 HIVE-14364. Update timeouts for llap comparator tests.
(Siddharth Seth, reviewed by Gunther Hagleitner)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820303 - PreCommit-HIVE-MASTER-Build
> FileUtils.isSubDir may return incorrect result
> --
>
> Key: HIVE-14259
> URL: https://issues.apache.org/jira/browse/HIVE-14259
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch
>
>
> while I was working on HIVE-12244 i've looked around for utility
> methods...i've found this method; but it considers path: `/dir12` inside
> `/dir1`
> which is not true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396980#comment-15396980 ] zhihai xu commented on HIVE-14368: -- I attached a patch HIVE-14368.000.patch which add exception's stack trace to the error message of TGetOperationStatusResp. > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-14368: - Attachment: HIVE-14368.000.patch > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-14368: - Status: Patch Available (was: Open) > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396943#comment-15396943
]
Hive QA commented on HIVE-14333:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820552/HIVE-14333.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10373 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/666/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/666/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-666/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820552 - PreCommit-HIVE-MASTER-Build
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch, HIVE-14333.2.patch,
> HIVE-14333.3.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[
https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saket Saurabh updated HIVE-14366:
-
Status: Patch Available (was: Open)
> Conversion of a Non-ACID table to an ACID table produces non-unique primary
> keys
>
>
> Key: HIVE-14366
> URL: https://issues.apache.org/jira/browse/HIVE-14366
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Eugene Koifman
> Attachments: HIVE-14366.01.patch
>
>
> When a Non-ACID table is converted to an ACID table, the primary key
> consisting of (original transaction id, bucket_id, row_id) is not generated
> uniquely. Currently, the row_id is always set to 0 for most rows. This leads
> to correctness issue for such tables.
> Quickest way to reproduce is to add the following unit test to
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
> {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
> @Test
> public void testOriginalReader() throws Exception {
> FileSystem fs = FileSystem.get(hiveConf);
> FileStatus[] status;
> // 1. Insert five rows to Non-ACID table.
> runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
> values(1,2),(3,4),(5,6),(7,8),(9,10)");
> // 2. Convert NONACIDORCTBL to ACID table.
> runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
> TBLPROPERTIES ('transactional'='true')");
> // 3. Perform a major compaction.
> runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
> 'MAJOR'");
> runWorker(hiveConf);
> // 4. Perform a delete.
> runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a =
> 1");
> // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since
> (1,2) has been deleted.
> List rs = runStatementOnDriver("select a,b from " +
> Table.NONACIDORCTBL + " order by a,b");
> int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
> Assert.assertEquals(stringifyValues(resultData), rs);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[
https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saket Saurabh updated HIVE-14366:
-
Attachment: HIVE-14366.01.patch
Initial patch to fix the bug by reusing the globally unique rowNumbers found in
the orc files.
> Conversion of a Non-ACID table to an ACID table produces non-unique primary
> keys
>
>
> Key: HIVE-14366
> URL: https://issues.apache.org/jira/browse/HIVE-14366
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Eugene Koifman
> Attachments: HIVE-14366.01.patch
>
>
> When a Non-ACID table is converted to an ACID table, the primary key
> consisting of (original transaction id, bucket_id, row_id) is not generated
> uniquely. Currently, the row_id is always set to 0 for most rows. This leads
> to correctness issue for such tables.
> Quickest way to reproduce is to add the following unit test to
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
> {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
> @Test
> public void testOriginalReader() throws Exception {
> FileSystem fs = FileSystem.get(hiveConf);
> FileStatus[] status;
> // 1. Insert five rows to Non-ACID table.
> runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
> values(1,2),(3,4),(5,6),(7,8),(9,10)");
> // 2. Convert NONACIDORCTBL to ACID table.
> runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
> TBLPROPERTIES ('transactional'='true')");
> // 3. Perform a major compaction.
> runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
> 'MAJOR'");
> runWorker(hiveConf);
> // 4. Perform a delete.
> runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a =
> 1");
> // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since
> (1,2) has been deleted.
> List rs = runStatementOnDriver("select a,b from " +
> Table.NONACIDORCTBL + " order by a,b");
> int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
> Assert.assertEquals(stringifyValues(resultData), rs);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14353) Performance degradation after Projection Pruning in CBO
[ https://issues.apache.org/jira/browse/HIVE-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396922#comment-15396922 ] Nemon Lou commented on HIVE-14353: -- [~pxiong] Sorry for the misleading. Performance degradation is at run time(an application run on YARN),not compile time. HiveRelFieldTrimmer adds a projection rel node above table scan.The projection node then compiled to select operator in hive. That's why I record the time spent in select operator during run time. > Performance degradation after Projection Pruning in CBO > > > Key: HIVE-14353 > URL: https://issues.apache.org/jira/browse/HIVE-14353 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou > > TPC-DS with factor 1024. > Hive on Spark. > With and without projection prunning,time spent are quite different. > The way to disable projection prunning : disable HiveRelFieldTrimmer in code > and compile a new jar. > ||queries||CBO_no_projection_prune||CBO|| > |q27| 160|251 | > |q7 | 200|312 | > |q88| 701|1092| > |q68| 234|345 | > |q39|53|78 | > |q73| 160|228 | > |q31| 463|659 | > |q79| 242|343 | > |q46| 256|363 | > |q60| 271|382 | > |q66| 198|278 | > |q34| 155|217 | > |q19| 184|256 | > |q26| 154|214 | > |q56| 262|364 | > |q75| 942|1303| > |q71| 288|388 | > |q25| 329|442 | > |q52| 142|190 | > |q42| 142|189 | > |q3 | 139|185 | > |q98| 153|203 | > |q89| 187|248 | > |q58| 264|340 | > |q43| 127|162 | > |q32| 174|221 | > |q96| 156|197 | > |q70| 320|404 | > |q29| 499|629 | > |q18| 266|329 | > |q21| 76 |92 | > |q90| 139|165 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Attachment: HIVE-14367.1.patch > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Status: Patch Available (was: Open) > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Attachment: (was: HIVE-14367.patch) > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14353) Performance degradation after Projection Pruning in CBO
[ https://issues.apache.org/jira/browse/HIVE-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396905#comment-15396905 ] Pengcheng Xiong commented on HIVE-14353: [~nemon], thanks for your data. I am bit confused for the CBO_time_in_SelectOP that you mentioned... What i would like to see is how much time did we spend on compiling the query (I mean the time spent on the application of HiveRelFieldTrimmer). For example, for q27, the time difference is 251-160=91. If the time spent on the application of HiveRelFieldTrimmer is around 90s, then that means we should improve the compilation. Otherwise if the time spent on the application of HiveRelFieldTrimmer is around 1-5s, then that means the difference comes from somewhere else rather than compilation.. Thanks. > Performance degradation after Projection Pruning in CBO > > > Key: HIVE-14353 > URL: https://issues.apache.org/jira/browse/HIVE-14353 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou > > TPC-DS with factor 1024. > Hive on Spark. > With and without projection prunning,time spent are quite different. > The way to disable projection prunning : disable HiveRelFieldTrimmer in code > and compile a new jar. > ||queries||CBO_no_projection_prune||CBO|| > |q27| 160|251 | > |q7 | 200|312 | > |q88| 701|1092| > |q68| 234|345 | > |q39|53|78 | > |q73| 160|228 | > |q31| 463|659 | > |q79| 242|343 | > |q46| 256|363 | > |q60| 271|382 | > |q66| 198|278 | > |q34| 155|217 | > |q19| 184|256 | > |q26| 154|214 | > |q56| 262|364 | > |q75| 942|1303| > |q71| 288|388 | > |q25| 329|442 | > |q52| 142|190 | > |q42| 142|189 | > |q3 | 139|185 | > |q98| 153|203 | > |q89| 187|248 | > |q58| 264|340 | > |q43| 127|162 | > |q32| 174|221 | > |q96| 156|197 | > |q70| 320|404 | > |q29| 499|629 | > |q18| 266|329 | > |q21| 76 |92 | > |q90| 139|165 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14353) Performance degradation after Projection Pruning in CBO
[ https://issues.apache.org/jira/browse/HIVE-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396893#comment-15396893 ] Nemon Lou commented on HIVE-14353: -- ||queries||CBO_total_time||CBO_time_in_SelectOP|| |q27| 266.494|251 | |q7 | 328.259|98.8 | |q68| 369.159|105 | |q46| 392.777|91.75| I just run a few of them because of time limit. The time spent in selectOP is calculated by adding up total times spent for selectOP in one executor ,and then divide number of cores.(4 in my case). Also,I have run q46 without projection pruning.And total time is 266.226,time spent in selectOP is 0.125 seconds. > Performance degradation after Projection Pruning in CBO > > > Key: HIVE-14353 > URL: https://issues.apache.org/jira/browse/HIVE-14353 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou > > TPC-DS with factor 1024. > Hive on Spark. > With and without projection prunning,time spent are quite different. > The way to disable projection prunning : disable HiveRelFieldTrimmer in code > and compile a new jar. > ||queries||CBO_no_projection_prune||CBO|| > |q27| 160|251 | > |q7 | 200|312 | > |q88| 701|1092| > |q68| 234|345 | > |q39|53|78 | > |q73| 160|228 | > |q31| 463|659 | > |q79| 242|343 | > |q46| 256|363 | > |q60| 271|382 | > |q66| 198|278 | > |q34| 155|217 | > |q19| 184|256 | > |q26| 154|214 | > |q56| 262|364 | > |q75| 942|1303| > |q71| 288|388 | > |q25| 329|442 | > |q52| 142|190 | > |q42| 142|189 | > |q3 | 139|185 | > |q98| 153|203 | > |q89| 187|248 | > |q58| 264|340 | > |q43| 127|162 | > |q32| 174|221 | > |q96| 156|197 | > |q70| 320|404 | > |q29| 499|629 | > |q18| 266|329 | > |q21| 76 |92 | > |q90| 139|165 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396884#comment-15396884 ] Amareshwari Sriramadasu commented on HIVE-14335: [~szehon], Can we commit this to branch-2.1 as well? Thanks > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.2.0 > > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396871#comment-15396871
]
Xuefu Zhang commented on HIVE-14303:
Ok. Thanks for the explanation. +1
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice.
> -
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.1.0
>
> Attachments: HIVE-14303.0.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements
> Closeable interface and ExecReducer.close can be called multiple time. We saw
> the following NPE which hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first reducer.close.
> The following reducer log can give more proof:
> {code}
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written -
> 53466
> 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing
> operators - failing tree
> 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at
[jira] [Assigned] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[
https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-14366:
-
Assignee: Eugene Koifman
> Conversion of a Non-ACID table to an ACID table produces non-unique primary
> keys
>
>
> Key: HIVE-14366
> URL: https://issues.apache.org/jira/browse/HIVE-14366
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Eugene Koifman
>
> When a Non-ACID table is converted to an ACID table, the primary key
> consisting of (original transaction id, bucket_id, row_id) is not generated
> uniquely. Currently, the row_id is always set to 0 for most rows. This leads
> to correctness issue for such tables.
> Quickest way to reproduce is to add the following unit test to
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
> {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
> @Test
> public void testOriginalReader() throws Exception {
> FileSystem fs = FileSystem.get(hiveConf);
> FileStatus[] status;
> // 1. Insert five rows to Non-ACID table.
> runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
> values(1,2),(3,4),(5,6),(7,8),(9,10)");
> // 2. Convert NONACIDORCTBL to ACID table.
> runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
> TBLPROPERTIES ('transactional'='true')");
> // 3. Perform a major compaction.
> runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
> 'MAJOR'");
> runWorker(hiveConf);
> // 4. Perform a delete.
> runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a =
> 1");
> // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since
> (1,2) has been deleted.
> List rs = runStatementOnDriver("select a,b from " +
> Table.NONACIDORCTBL + " order by a,b");
> int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
> Assert.assertEquals(stringifyValues(resultData), rs);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396866#comment-15396866
]
Prasanth Jayachandran commented on HIVE-14355:
--
[~sershe] can you plz take a look at the changes?
To summarize
- LlapInputFormat disables llap io if it finds unsafe schema evolution
- OrcEncodedDataConsumer has to known the file schema which it gets from
OrcEncodedDataReader and reader schema from job conf. Both are required to
create schema evolution object.
- This check can happen only at runtime because we don't know the file schema
at compile time
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
>
[jira] [Updated] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14355:
-
Status: Patch Available (was: Open)
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
> ... 19 more
> {code}
> {code:title=When vectorization is disabled}
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.OperatorUtils: Setting output
> collector: RS[4] --> Reducer 2
> 2016-07-27T01:52:43,329 ERROR [TezTaskRunner
>
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396864#comment-15396864
]
Prasanth Jayachandran commented on HIVE-14355:
--
This patch also depends on HIVE-14310 changes.
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
> ... 19 more
> {code}
> {code:title=When vectorization is disabled}
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.OperatorUtils: Setting output
> collector: RS[4] --> Reducer 2
>
[jira] [Updated] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14355:
-
Attachment: HIVE-14355.1.patch
HIVE-14355-java-only.patch
Attaching java only patch and another patch that adds all schema evolution
tests to llap.
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
> ... 19 more
> {code}
> {code:title=When vectorization is disabled}
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)]
[jira] [Commented] (HIVE-14331) Task should set exception for failed map reduce job.
[ https://issues.apache.org/jira/browse/HIVE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396854#comment-15396854 ] zhihai xu commented on HIVE-14331: -- thanks [~jxiang] for reviewing and committing the patch! > Task should set exception for failed map reduce job. > > > Key: HIVE-14331 > URL: https://issues.apache.org/jira/browse/HIVE-14331 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.1.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14331.000.patch > > > Task should set exception for failed map reduce job. So the exception can be > seen in HookContext. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396851#comment-15396851
]
zhihai xu commented on HIVE-14303:
--
Thanks for the review [~xuefuz]! Currently {{IOUtils.cleanup}} already checks
if reducer is null, but the reducer is not null, because the exception happened
inside {{reducer.close()}} at line 453 and it will exit before the reducer is
set to null at line 454. Also {{ExecReducer}} didn't follow {{Closeable}}
interface
rule:https://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html#close()
{{code}}
public static void cleanup(Log log, java.io.Closeable... closeables) {
for (java.io.Closeable c : closeables) {
if (c != null) {// check if reducer is null here
try {
c.close();
} catch(IOException e) {
if (log != null && log.isDebugEnabled()) {
log.debug("Exception in closing " + c, e);
}
}
}
}
}
{{code}}
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice.
> -
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.1.0
>
> Attachments: HIVE-14303.0.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements
> Closeable interface and ExecReducer.close can be called multiple time. We saw
> the following NPE which hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first reducer.close.
> The following reducer log can give more proof:
> {code}
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing...
> 2016-07-14 22:24:51,016 INFO
[jira] [Comment Edited] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396851#comment-15396851
]
zhihai xu edited comment on HIVE-14303 at 7/28/16 3:25 AM:
---
Thanks for the review [~xuefuz]! Currently {{IOUtils.cleanup}} already checks
if reducer is null, but the reducer is not null, because the exception happened
inside {{reducer.close()}} at line 453 and it will exit before the reducer is
set to null at line 454. Also {{ExecReducer}} didn't follow {{Closeable}}
interface
rule:https://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html#close()
{code}
public static void cleanup(Log log, java.io.Closeable... closeables) {
for (java.io.Closeable c : closeables) {
if (c != null) {// check if reducer is null here
try {
c.close();
} catch(IOException e) {
if (log != null && log.isDebugEnabled()) {
log.debug("Exception in closing " + c, e);
}
}
}
}
}
{code}
was (Author: zxu):
Thanks for the review [~xuefuz]! Currently {{IOUtils.cleanup}} already checks
if reducer is null, but the reducer is not null, because the exception happened
inside {{reducer.close()}} at line 453 and it will exit before the reducer is
set to null at line 454. Also {{ExecReducer}} didn't follow {{Closeable}}
interface
rule:https://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html#close()
{{code}}
public static void cleanup(Log log, java.io.Closeable... closeables) {
for (java.io.Closeable c : closeables) {
if (c != null) {// check if reducer is null here
try {
c.close();
} catch(IOException e) {
if (log != null && log.isDebugEnabled()) {
log.debug("Exception in closing " + c, e);
}
}
}
}
}
{{code}}
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice.
> -
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.1.0
>
> Attachments: HIVE-14303.0.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements
> Closeable interface and ExecReducer.close can be called multiple time. We saw
> the following NPE which hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first
[jira] [Comment Edited] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396851#comment-15396851
]
zhihai xu edited comment on HIVE-14303 at 7/28/16 3:26 AM:
---
Thanks for the review [~xuefuz]! Currently {{IOUtils.cleanup}} already checks
if reducer is null, but the reducer is not null, because the exception happened
inside {{reducer.close()}} at line 453 and it will exit before the reducer is
set to null at line 454. Also {{ExecReducer}} didn't follow {{Closeable}}
interface rule:
https://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html#close()
{code}
public static void cleanup(Log log, java.io.Closeable... closeables) {
for (java.io.Closeable c : closeables) {
if (c != null) {// check if reducer is null here
try {
c.close();
} catch(IOException e) {
if (log != null && log.isDebugEnabled()) {
log.debug("Exception in closing " + c, e);
}
}
}
}
}
{code}
was (Author: zxu):
Thanks for the review [~xuefuz]! Currently {{IOUtils.cleanup}} already checks
if reducer is null, but the reducer is not null, because the exception happened
inside {{reducer.close()}} at line 453 and it will exit before the reducer is
set to null at line 454. Also {{ExecReducer}} didn't follow {{Closeable}}
interface
rule:https://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html#close()
{code}
public static void cleanup(Log log, java.io.Closeable... closeables) {
for (java.io.Closeable c : closeables) {
if (c != null) {// check if reducer is null here
try {
c.close();
} catch(IOException e) {
if (log != null && log.isDebugEnabled()) {
log.debug("Exception in closing " + c, e);
}
}
}
}
}
{code}
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice.
> -
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.1.0
>
> Attachments: HIVE-14303.0.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to
> avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements
> Closeable interface and ExecReducer.close can be called multiple time. We saw
> the following NPE which hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first
[jira] [Commented] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396836#comment-15396836
]
Hive QA commented on HIVE-14346:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820283/HIVE-14346.0.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10371 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_view
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/665/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/665/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-665/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820283 - PreCommit-HIVE-MASTER-Build
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Attachment: HIVE-14367.patch > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[
https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saket Saurabh updated HIVE-14366:
-
Description:
When a Non-ACID table is converted to an ACID table, the primary key consisting
of (original transaction id, bucket_id, row_id) is not generated uniquely.
Currently, the row_id is always set to 0 for most rows. This leads to
correctness issue for such tables.
Quickest way to reproduce is to add the following unit test to
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
{code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
@Test
public void testOriginalReader() throws Exception {
FileSystem fs = FileSystem.get(hiveConf);
FileStatus[] status;
// 1. Insert five rows to Non-ACID table.
runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
values(1,2),(3,4),(5,6),(7,8),(9,10)");
// 2. Convert NONACIDORCTBL to ACID table.
runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
TBLPROPERTIES ('transactional'='true')");
// 3. Perform a major compaction.
runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
'MAJOR'");
runWorker(hiveConf);
// 4. Perform a delete.
runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");
// 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since
(1,2) has been deleted.
List rs = runStatementOnDriver("select a,b from " +
Table.NONACIDORCTBL + " order by a,b");
int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
Assert.assertEquals(stringifyValues(resultData), rs);
}
{code}
was:
When a Non-ACID table is converted to an ACID table, the primary key consisting
of (original transaction id, bucket_id, row_id) is not generated uniquely.
Currently, the row_id is always set to 0 for most rows. This leads to
correctness issue for such tables.
Quickest way to reproduce is to add the following unit test to
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
{code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
@Test
public void testOriginalReader() throws Exception {
FileSystem fs = FileSystem.get(hiveConf);
FileStatus[] status;
// 1. Insert five rows to Non-ACID table.
runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
values(1,2),(3,4),(5,6),(7,8),(9,10)");
// 2. Convert NONACIDORCTBL to ACID table.
runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
TBLPROPERTIES ('transactional'='true')");
// 3. Perform a major compaction.
runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
'MAJOR'");
runWorker(hiveConf);
// 3. Perform a delete.
runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");
// Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since
(1,2) has been deleted.
List rs = runStatementOnDriver("select a,b from " +
Table.NONACIDORCTBL + " order by a,b");
int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
Assert.assertEquals(stringifyValues(resultData), rs);
}
{code}
> Conversion of a Non-ACID table to an ACID table produces non-unique primary
> keys
>
>
> Key: HIVE-14366
> URL: https://issues.apache.org/jira/browse/HIVE-14366
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saket Saurabh
>
> When a Non-ACID table is converted to an ACID table, the primary key
> consisting of (original transaction id, bucket_id, row_id) is not generated
> uniquely. Currently, the row_id is always set to 0 for most rows. This leads
> to correctness issue for such tables.
> Quickest way to reproduce is to add the following unit test to
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
> {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
> @Test
> public void testOriginalReader() throws Exception {
> FileSystem fs = FileSystem.get(hiveConf);
> FileStatus[] status;
> // 1. Insert five rows to Non-ACID table.
> runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
> values(1,2),(3,4),(5,6),(7,8),(9,10)");
> // 2. Convert NONACIDORCTBL to ACID table.
> runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET
> TBLPROPERTIES ('transactional'='true')");
> // 3. Perform a major compaction.
> runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact
> 'MAJOR'");
> runWorker(hiveConf);
> // 4. Perform a delete.
> runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a =
> 1");
> // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only
[jira] [Updated] (HIVE-14352) Beeline can't run sub-second queries in HTTP mode
[
https://issues.apache.org/jira/browse/HIVE-14352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14352:
---
Summary: Beeline can't run sub-second queries in HTTP mode (was: Beeline
can't run sub-second queries)
> Beeline can't run sub-second queries in HTTP mode
> -
>
> Key: HIVE-14352
> URL: https://issues.apache.org/jira/browse/HIVE-14352
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Gopal V
>
> Even a 12ms query execution takes 1000+ ms in Beeline.
> {code}
> private static final int DEFAULT_QUERY_PROGRESS_INTERVAL = 1000;
> ...
> while (hiveStatement.hasMoreLogs()) {
> Thread.sleep(DEFAULT_QUERY_PROGRESS_INTERVAL);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396742#comment-15396742
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820561/HIVE-14035.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10387 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/664/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/664/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820561 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14363) bucketmap inner join query fails due to NullPointerException in some cases
[
https://issues.apache.org/jira/browse/HIVE-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14363:
-
Attachment: HIVE-14363.1.patch
> bucketmap inner join query fails due to NullPointerException in some cases
> --
>
> Key: HIVE-14363
> URL: https://issues.apache.org/jira/browse/HIVE-14363
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jagruti Varia
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14363.1.patch
>
>
> Bucketmap inner join query between bucketed tables throws following exception
> when one table contains all the empty buckets while other has all the
> non-empty buckets.
> {noformat}
> Vertex failed, vertexName=Map 2, vertexId=vertex_1466710232033_0432_4_01,
> diagnostics=[Task failed, taskId=task_1466710232033_0432_4_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0432_4_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:330)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getKeyValueReader(MapRecordProcessor.java:372)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.initializeMapRecordSources(MapRecordProcessor.java:344)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:292)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1466710232033_0432_4_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
>
[jira] [Resolved] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-14364. --- Resolution: Fixed Fix Version/s: 2.2.0 Committed to master. > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14363) bucketmap inner join query fails due to NullPointerException in some cases
[
https://issues.apache.org/jira/browse/HIVE-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14363:
-
Status: Patch Available (was: Open)
> bucketmap inner join query fails due to NullPointerException in some cases
> --
>
> Key: HIVE-14363
> URL: https://issues.apache.org/jira/browse/HIVE-14363
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jagruti Varia
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14363.1.patch
>
>
> Bucketmap inner join query between bucketed tables throws following exception
> when one table contains all the empty buckets while other has all the
> non-empty buckets.
> {noformat}
> Vertex failed, vertexName=Map 2, vertexId=vertex_1466710232033_0432_4_01,
> diagnostics=[Task failed, taskId=task_1466710232033_0432_4_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0432_4_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:330)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getKeyValueReader(MapRecordProcessor.java:372)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.initializeMapRecordSources(MapRecordProcessor.java:344)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:292)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1466710232033_0432_4_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
>
[jira] [Updated] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-13813:
Attachment: HIVE-13183.1.patch
Oops forgot about this. Re-attaching patch to test again.
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.1.patch, HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14332) Reduce logging from VectorMapOperator
[ https://issues.apache.org/jira/browse/HIVE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14332: Resolution: Fixed Status: Resolved (was: Patch Available) > Reduce logging from VectorMapOperator > - > > Key: HIVE-14332 > URL: https://issues.apache.org/jira/browse/HIVE-14332 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-14332.01.patch > > > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator: VectorMapOperator > path: > hdfs://cn108-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2451710, > read type VECTORIZED_INPUT_FILE_FORMAT, vector deserialize type NONE, > aliases store_sales > Lines like this repeat all over the log. This gets really big with a large > number of partitions. 6MB of logs per node for a 30 task query running for 20 > seconds on a 3 node cluster. > Instead of logging this line - can we have a consolidated log / logging only > if something abnormal happens ... or a shorter log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396687#comment-15396687
]
Matt McCline commented on HIVE-14336:
-
Committed to master.
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Fix Version/s: 2.2.0
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14338: --- Affects Version/s: 2.1.0 > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14332) Reduce logging from VectorMapOperator
[ https://issues.apache.org/jira/browse/HIVE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396686#comment-15396686 ] Matt McCline commented on HIVE-14332: - Committed to master. > Reduce logging from VectorMapOperator > - > > Key: HIVE-14332 > URL: https://issues.apache.org/jira/browse/HIVE-14332 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-14332.01.patch > > > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator: VectorMapOperator > path: > hdfs://cn108-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2451710, > read type VECTORIZED_INPUT_FILE_FORMAT, vector deserialize type NONE, > aliases store_sales > Lines like this repeat all over the log. This gets really big with a large > number of partitions. 6MB of logs per node for a 30 task query running for 20 > seconds on a 3 node cluster. > Instead of logging this line - can we have a consolidated log / logging only > if something abnormal happens ... or a shorter log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14338: --- Fix Version/s: 2.1.1 2.2.0 > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14338: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14332) Reduce logging from VectorMapOperator
[ https://issues.apache.org/jira/browse/HIVE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14332: Fix Version/s: 2.2.0 > Reduce logging from VectorMapOperator > - > > Key: HIVE-14332 > URL: https://issues.apache.org/jira/browse/HIVE-14332 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-14332.01.patch > > > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator: VectorMapOperator > path: > hdfs://cn108-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2451710, > read type VECTORIZED_INPUT_FILE_FORMAT, vector deserialize type NONE, > aliases store_sales > Lines like this repeat all over the log. This gets really big with a large > number of partitions. 6MB of logs per node for a 30 task query running for 20 > seconds on a 3 node cluster. > Instead of logging this line - can we have a consolidated log / logging only > if something abnormal happens ... or a shorter log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14332) Reduce logging from VectorMapOperator
[ https://issues.apache.org/jira/browse/HIVE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396681#comment-15396681 ] Matt McCline commented on HIVE-14332: - Test failures are not related. > Reduce logging from VectorMapOperator > - > > Key: HIVE-14332 > URL: https://issues.apache.org/jira/browse/HIVE-14332 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14332.01.patch > > > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator: VectorMapOperator > path: > hdfs://cn108-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2451710, > read type VECTORIZED_INPUT_FILE_FORMAT, vector deserialize type NONE, > aliases store_sales > Lines like this repeat all over the log. This gets really big with a large > number of partitions. 6MB of logs per node for a 30 task query running for 20 > seconds on a 3 node cluster. > Instead of logging this line - can we have a consolidated log / logging only > if something abnormal happens ... or a shorter log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396663#comment-15396663 ] Siddharth Seth commented on HIVE-14364: --- bq. Why cannot classloader stuff be moved into pre-test by touching the same classes? That way the test timeout doesn't need to increase. -0 The entire test will hang if there's a disk issue (or take a really long time). It's better to fail fast instead of running forever because of bad hardware. Touching classes as a part of a test because of slow classloading is going to be extremely confusing for anyone looking at this code later. Not maintainable at all. It should not be restricted to this test only. > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396663#comment-15396663 ] Siddharth Seth edited comment on HIVE-14364 at 7/27/16 11:39 PM: - bq. Why cannot classloader stuff be moved into pre-test by touching the same classes? That way the test timeout doesn't need to increase. -0 The entire test will hang if there's a disk issue (or take a really long time). It's better to fail fast instead of running forever because of bad hardware. Touching classes as a part of a test because of slow classloading is going to be extremely confusing for anyone looking at this code later. Not maintainable at all. It should not be restricted to this test only. Could you please revoke your -0. was (Author: sseth): bq. Why cannot classloader stuff be moved into pre-test by touching the same classes? That way the test timeout doesn't need to increase. -0 The entire test will hang if there's a disk issue (or take a really long time). It's better to fail fast instead of running forever because of bad hardware. Touching classes as a part of a test because of slow classloading is going to be extremely confusing for anyone looking at this code later. Not maintainable at all. It should not be restricted to this test only. > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14350) Aborted txns cause false positive "Not enough history available..." msgs
[ https://issues.apache.org/jira/browse/HIVE-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14350: -- Attachment: HIVE-14350.6.patch > Aborted txns cause false positive "Not enough history available..." msgs > > > Key: HIVE-14350 > URL: https://issues.apache.org/jira/browse/HIVE-14350 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-14350.2.patch, HIVE-14350.3.patch, > HIVE-14350.5.patch, HIVE-14350.6.patch > > > this is a followup to HIVE-13369. Only open txns should prevent use of a > base file. But ValidTxnList does not make a distinction between open and > aborted txns. The presence of aborted txns causes false positives which can > happen too often since the flow is > 1. Worker generates a new base file, > 2. then asynchronously Cleaner removes now-compacted aborted txns. (strictly > speaking it's Initiator that does the actual clean up) > So we may have base_5 and base_10 and txnid 7 aborted. Then current impl > will disallow use of base_10 though there is no need for that. Worse, if > txnid_4 is aborted and hasn't been purged yet, base_5 will be rejected as > well and then an error will be raised since there is no suitable base file > left. > ErrorMsg.ACID_NOT_ENOUGH_HISTORY is msg produced -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13595) HiveServer2: Evaluate if ThriftJDBCBinarySerde should implement VectorizedSerde
[ https://issues.apache.org/jira/browse/HIVE-13595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao reassigned HIVE-13595: -- Assignee: Ziyang Zhao > HiveServer2: Evaluate if ThriftJDBCBinarySerde should implement > VectorizedSerde > --- > > Key: HIVE-13595 > URL: https://issues.apache.org/jira/browse/HIVE-13595 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > > As part of HIVE-12049, ThriftJDBCBinarySerde was introduced which buffers > rows and writes thrift converted columnar row batches as part of the final > task output. Hive has VectorizedSerde which is used during vectorized > operations. We should explore if ThriftJDBCBinarySerde should implement that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396634#comment-15396634 ] Sergey Shelukhin commented on HIVE-14364: - Why cannot classloader stuff be moved into pre-test by touching the same classes? That way the test timeout doesn't need to increase. -0 > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14365) Simplify logic for check introduced in HIVE-10022
[ https://issues.apache.org/jira/browse/HIVE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-14365: Affects Version/s: 2.1.1 2.2.0 > Simplify logic for check introduced in HIVE-10022 > - > > Key: HIVE-14365 > URL: https://issues.apache.org/jira/browse/HIVE-14365 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 2.1.1 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14365.patch > > > We introduced a parent-check/glob-check/file-check in SQLAuthorizationUtils > in HIVE-10022, but the logic for that is more convoluted than it needs to be. > Taking a cue off RANGER-1126 , we should simplify this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14365) Simplify logic for check introduced in HIVE-10022
[ https://issues.apache.org/jira/browse/HIVE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan reassigned HIVE-14365: --- Assignee: Sushanth Sowmyan > Simplify logic for check introduced in HIVE-10022 > - > > Key: HIVE-14365 > URL: https://issues.apache.org/jira/browse/HIVE-14365 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14365.patch > > > We introduced a parent-check/glob-check/file-check in SQLAuthorizationUtils > in HIVE-10022, but the logic for that is more convoluted than it needs to be. > Taking a cue off RANGER-1126 , we should simplify this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14365) Simplify logic for check introduced in HIVE-10022
[ https://issues.apache.org/jira/browse/HIVE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-14365: Attachment: HIVE-14365.patch Patch attached. > Simplify logic for check introduced in HIVE-10022 > - > > Key: HIVE-14365 > URL: https://issues.apache.org/jira/browse/HIVE-14365 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan > Attachments: HIVE-14365.patch > > > We introduced a parent-check/glob-check/file-check in SQLAuthorizationUtils > in HIVE-10022, but the logic for that is more convoluted than it needs to be. > Taking a cue off RANGER-1126 , we should simplify this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14365) Simplify logic for check introduced in HIVE-10022
[ https://issues.apache.org/jira/browse/HIVE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-14365: Status: Patch Available (was: Open) > Simplify logic for check introduced in HIVE-10022 > - > > Key: HIVE-14365 > URL: https://issues.apache.org/jira/browse/HIVE-14365 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14365.patch > > > We introduced a parent-check/glob-check/file-check in SQLAuthorizationUtils > in HIVE-10022, but the logic for that is more convoluted than it needs to be. > Taking a cue off RANGER-1126 , we should simplify this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396615#comment-15396615 ] Gunther Hagleitner commented on HIVE-14364: --- +1 > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[
https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396607#comment-15396607
]
Pengcheng Xiong commented on HIVE-14338:
{code}
org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms
{code}
sounds related.
> Delete/Alter table calls failing with HiveAccessControlException
>
>
> Key: HIVE-14338
> URL: https://issues.apache.org/jira/browse/HIVE-14338
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14338.01.patch
>
>
> Many Hcatalog/Webhcat tests are failing with below error, when tests try to
> alter/delete/describe tables. Error is thrown when the same user or a
> different user (same group) who created the table is trying to run the
> delete/alter table call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14350) Aborted txns cause false positive "Not enough history available..." msgs
[ https://issues.apache.org/jira/browse/HIVE-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14350: -- Attachment: HIVE-14350.5.patch > Aborted txns cause false positive "Not enough history available..." msgs > > > Key: HIVE-14350 > URL: https://issues.apache.org/jira/browse/HIVE-14350 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-14350.2.patch, HIVE-14350.3.patch, > HIVE-14350.5.patch > > > this is a followup to HIVE-13369. Only open txns should prevent use of a > base file. But ValidTxnList does not make a distinction between open and > aborted txns. The presence of aborted txns causes false positives which can > happen too often since the flow is > 1. Worker generates a new base file, > 2. then asynchronously Cleaner removes now-compacted aborted txns. (strictly > speaking it's Initiator that does the actual clean up) > So we may have base_5 and base_10 and txnid 7 aborted. Then current impl > will disallow use of base_10 though there is no need for that. Worse, if > txnid_4 is aborted and hasn't been purged yet, base_5 will be rejected as > well and then an error will be raised since there is no suitable base file > left. > ErrorMsg.ACID_NOT_ENOUGH_HISTORY is msg produced -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14310) ORC schema evolution should not completely disable PPD
[ https://issues.apache.org/jira/browse/HIVE-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396582#comment-15396582 ] Owen O'Malley commented on HIVE-14310: -- +1 > ORC schema evolution should not completely disable PPD > -- > > Key: HIVE-14310 > URL: https://issues.apache.org/jira/browse/HIVE-14310 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14310.1.patch, HIVE-14310.2.patch, > HIVE-14310.4.patch, HIVE-14310.WIP.patch > > > Follow up for HIVE-14214 which completely shuts off PPD when there is any > schema evolution. Some evolutions are safer for PPD like > byte -> short -> int -> long > -float -> double- (This is unsafe, see comments in SchemaEvolution.java in > the patch) > varchar <-> string (string to char, varchar to char and vice versa is also > unsafe conversion as Orc stores internal index with padded spaces for char) > For all other conversions we can disable PPD for that specific column that > has evolved by returning TruthValue.YES_NO -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14310) ORC schema evolution should not completely disable PPD
[ https://issues.apache.org/jira/browse/HIVE-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14310: - Attachment: HIVE-14310.4.patch Addressed [~owen.omalley]'s review comments. > ORC schema evolution should not completely disable PPD > -- > > Key: HIVE-14310 > URL: https://issues.apache.org/jira/browse/HIVE-14310 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14310.1.patch, HIVE-14310.2.patch, > HIVE-14310.4.patch, HIVE-14310.WIP.patch > > > Follow up for HIVE-14214 which completely shuts off PPD when there is any > schema evolution. Some evolutions are safer for PPD like > byte -> short -> int -> long > -float -> double- (This is unsafe, see comments in SchemaEvolution.java in > the patch) > varchar <-> string (string to char, varchar to char and vice versa is also > unsafe conversion as Orc stores internal index with padded spaces for char) > For all other conversions we can disable PPD for that specific column that > has evolved by returning TruthValue.YES_NO -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[
https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396544#comment-15396544
]
Hive QA commented on HIVE-14338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820207/HIVE-14338.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10370 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/663/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/663/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820207 - PreCommit-HIVE-MASTER-Build
> Delete/Alter table calls failing with HiveAccessControlException
>
>
> Key: HIVE-14338
> URL: https://issues.apache.org/jira/browse/HIVE-14338
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14338.01.patch
>
>
> Many Hcatalog/Webhcat tests are failing with below error, when tests try to
> alter/delete/describe tables. Error is thrown when the same user or a
> different user (same group) who created the table is trying to run the
> delete/alter table call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14310) ORC schema evolution should not completely disable PPD
[
https://issues.apache.org/jira/browse/HIVE-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396511#comment-15396511
]
Owen O'Malley commented on HIVE-14310:
--
Comments:
* You should not throw in the file schema has more columns than the reader
schema. We want to support schema evolution that includes deleting columns.
* You should comment that included, readerFileTypes, and ppdSafeConversion are
all indexed by the reader column id.
* The comment about why float -> double isn't support is a little broken. How
about:
{code}
// NOTE: Float to double evolution is not safe as floats are stored as doubles
in ORC's
// internal index, but when doing predicate evaluation for queries like "select
* from
// orc_float where f = 74.72" the constant on the filter is converted from
string -> double
// so the precisions will be different and the comparison will fail.
// Soon, we should convert all sargs that compare equality between floats or
// doubles to range predicates.
{code}
Other than that, it looks good.
> ORC schema evolution should not completely disable PPD
> --
>
> Key: HIVE-14310
> URL: https://issues.apache.org/jira/browse/HIVE-14310
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14310.1.patch, HIVE-14310.2.patch,
> HIVE-14310.WIP.patch
>
>
> Follow up for HIVE-14214 which completely shuts off PPD when there is any
> schema evolution. Some evolutions are safer for PPD like
> byte -> short -> int -> long
> -float -> double- (This is unsafe, see comments in SchemaEvolution.java in
> the patch)
> varchar <-> string (string to char, varchar to char and vice versa is also
> unsafe conversion as Orc stores internal index with padded spaces for char)
> For all other conversions we can disable PPD for that specific column that
> has evolved by returning TruthValue.YES_NO
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14310) ORC schema evolution should not completely disable PPD
[
https://issues.apache.org/jira/browse/HIVE-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396511#comment-15396511
]
Owen O'Malley edited comment on HIVE-14310 at 7/27/16 10:15 PM:
Comments:
* You should not throw if the file schema has more columns than the reader
schema. We want to support schema evolution that includes deleting columns.
* You should comment that included, readerFileTypes, and ppdSafeConversion are
all indexed by the reader column id.
* The comment about why float -> double isn't support is a little broken. How
about:
{code}
// NOTE: Float to double evolution is not safe as floats are stored as doubles
in ORC's
// internal index, but when doing predicate evaluation for queries like "select
* from
// orc_float where f = 74.72" the constant on the filter is converted from
string -> double
// so the precisions will be different and the comparison will fail.
// Soon, we should convert all sargs that compare equality between floats or
// doubles to range predicates.
{code}
Other than that, it looks good.
was (Author: owen.omalley):
Comments:
* You should not throw in the file schema has more columns than the reader
schema. We want to support schema evolution that includes deleting columns.
* You should comment that included, readerFileTypes, and ppdSafeConversion are
all indexed by the reader column id.
* The comment about why float -> double isn't support is a little broken. How
about:
{code}
// NOTE: Float to double evolution is not safe as floats are stored as doubles
in ORC's
// internal index, but when doing predicate evaluation for queries like "select
* from
// orc_float where f = 74.72" the constant on the filter is converted from
string -> double
// so the precisions will be different and the comparison will fail.
// Soon, we should convert all sargs that compare equality between floats or
// doubles to range predicates.
{code}
Other than that, it looks good.
> ORC schema evolution should not completely disable PPD
> --
>
> Key: HIVE-14310
> URL: https://issues.apache.org/jira/browse/HIVE-14310
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14310.1.patch, HIVE-14310.2.patch,
> HIVE-14310.WIP.patch
>
>
> Follow up for HIVE-14214 which completely shuts off PPD when there is any
> schema evolution. Some evolutions are safer for PPD like
> byte -> short -> int -> long
> -float -> double- (This is unsafe, see comments in SchemaEvolution.java in
> the patch)
> varchar <-> string (string to char, varchar to char and vice versa is also
> unsafe conversion as Orc stores internal index with padded spaces for char)
> For all other conversions we can disable PPD for that specific column that
> has evolved by returning TruthValue.YES_NO
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13560: -- Attachment: HIVE-13560.10.patch For retest. > Adding Omid as connection manager for HBase Metastore > - > > Key: HIVE-13560 > URL: https://issues.apache.org/jira/browse/HIVE-13560 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13560.1.patch, HIVE-13560.10.patch, > HIVE-13560.2.patch, HIVE-13560.3.patch, HIVE-13560.4.patch, > HIVE-13560.5.patch, HIVE-13560.6.patch, HIVE-13560.7.patch, > HIVE-13560.8.patch, HIVE-13560.9.patch > > > Adding Omid as a transaction manager to HBase Metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14293) PerfLogger.openScopes should be transient
[
https://issues.apache.org/jira/browse/HIVE-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-14293:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
The last precommit run is clear. All test failures are seen in other runs.
Patch pushed to both branch-2.1 and master.
> PerfLogger.openScopes should be transient
> -
>
> Key: HIVE-14293
> URL: https://issues.apache.org/jira/browse/HIVE-14293
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14293.1.patch, HIVE-14293.2.patch,
> HIVE-14293.3.patch, HIVE-14293.4.patch
>
>
> See the following exception when running Hive e2e tests:
> {code}
> 0: jdbc:hive2://nat-r6-ojss-hsihs2-1.openstac> SELECT s.name, s2.age, s.gpa,
> v.registration, v2.contributions FROM student s INNER JOIN voter v ON (s.name
> = v.name) INNER JOIN student s2 ON (s2.age = v.age and s.name = s2.name)
> INNER JOIN voter v2 ON (v2.name = s2.name and v2.age = s2.age) WHERE v2.age =
> s.age ORDER BY s.name, s2.age, s.gpa, v.registration, v2.contributions;
> INFO : Compiling
> command(queryId=hive_20160717224915_3a52719f-539f-4f82-a9cd-0c0af4e09ef8):
> SELECT s.name, s2.age, s.gpa, v.registration, v2.contributions FROM student s
> INNER JOIN voter v ON (s.name = v.name) INNER JOIN student s2 ON (s2.age =
> v.age and s.name = s2.name) INNER JOIN voter v2 ON (v2.name = s2.name and
> v2.age = s2.age) WHERE v2.age = s.age ORDER BY s.name, s2.age, s.gpa,
> v.registration, v2.contributions
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:s.name,
> type:string, comment:null), FieldSchema(name:s2.age, type:int, comment:null),
> FieldSchema(name:s.gpa, type:double, comment:null),
> FieldSchema(name:v.registration, type:string, comment:null),
> FieldSchema(name:v2.contributions, type:float, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160717224915_3a52719f-539f-4f82-a9cd-0c0af4e09ef8);
> Time taken: 1.165 seconds
> INFO : Executing
> command(queryId=hive_20160717224915_3a52719f-539f-4f82-a9cd-0c0af4e09ef8):
> SELECT s.name, s2.age, s.gpa, v.registration, v2.contributions FROM student s
> INNER JOIN voter v ON (s.name = v.name) INNER JOIN student s2 ON (s2.age =
> v.age and s.name = s2.name) INNER JOIN voter v2 ON (v2.name = s2.name and
> v2.age = s2.age) WHERE v2.age = s.age ORDER BY s.name, s2.age, s.gpa,
> v.registration, v2.contributions
> INFO : Query ID = hive_20160717224915_3a52719f-539f-4f82-a9cd-0c0af4e09ef8
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Session is already open
> INFO : Dag name: SELECT s.name, s2.age, sv2.contributions(Stage-1)
> ERROR : Failed to execute tez graph.
> java.lang.RuntimeException: Error caching map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> Serialization trace:
> classes (sun.misc.Launcher$AppClassLoader)
> classloader (java.security.ProtectionDomain)
> context (java.security.AccessControlContext)
> acc (org.apache.hadoop.hive.ql.exec.UDFClassLoader)
> classLoader (org.apache.hadoop.hive.conf.HiveConf)
> conf (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> openScopes (org.apache.hadoop.hive.ql.log.PerfLogger)
> perfLogger (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:582)
> ~[hive-exec-2.1.0.2.5.0.0-1009.jar:2.1.0.2.5.0.0-1009]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:516)
> ~[hive-exec-2.1.0.2.5.0.0-1009.jar:2.1.0.2.5.0.0-1009]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:601)
> ~[hive-exec-2.1.0.2.5.0.0-1009.jar:2.1.0.2.5.0.0-1009]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1147)
> ~[hive-exec-2.1.0.2.5.0.0-1009.jar:2.1.0.2.5.0.0-1009]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:390)
>
[jira] [Updated] (HIVE-14364) Update timeouts for llap comparator tests
[ https://issues.apache.org/jira/browse/HIVE-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14364: -- Attachment: HIVE-14364.01.patch Trivial patch. [~sershe] - could you please take a look. > Update timeouts for llap comparator tests > - > > Key: HIVE-14364 > URL: https://issues.apache.org/jira/browse/HIVE-14364 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14364.01.patch > > > The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. > NO_PRECOMMIT_TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: Open) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Attachment: HIVE-14035.12.patch Refactor the way delete event writers are created for compaction case in favor of a better abstraction. Discovered that this design would be much simpler now, while working on improving compaction for ACID. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Open (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14310) ORC schema evolution should not completely disable PPD
[ https://issues.apache.org/jira/browse/HIVE-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396456#comment-15396456 ] Prasanth Jayachandran commented on HIVE-14310: -- rbt post and updating patch with base diff worked. Thanks for the tip! > ORC schema evolution should not completely disable PPD > -- > > Key: HIVE-14310 > URL: https://issues.apache.org/jira/browse/HIVE-14310 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14310.1.patch, HIVE-14310.2.patch, > HIVE-14310.WIP.patch > > > Follow up for HIVE-14214 which completely shuts off PPD when there is any > schema evolution. Some evolutions are safer for PPD like > byte -> short -> int -> long > -float -> double- (This is unsafe, see comments in SchemaEvolution.java in > the patch) > varchar <-> string (string to char, varchar to char and vice versa is also > unsafe conversion as Orc stores internal index with padded spaces for char) > For all other conversions we can disable PPD for that specific column that > has evolved by returning TruthValue.YES_NO -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14333:
-
Attachment: HIVE-14333.3.patch
Fixes some schema evolution test diffs
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch, HIVE-14333.2.patch,
> HIVE-14333.3.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14270) Write temporary data to HDFS when doing inserts on tables located on S3
[ https://issues.apache.org/jira/browse/HIVE-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396375#comment-15396375 ] Sergio Peña commented on HIVE-14270: Another thing I just thought about having a configuration variable is that Hive won't be able to ensure the quality of new values set by the user as there won't be tests that could verify that. We will provide S3 tests for now, but if something fails with azure for instance, then users will complain about it. I think it would be better to define a list of supported blob store schemes for now that are verified by tests and committers. What do you think? Another thing, I am thinking on add a bunch of S3 tests like Hadoop does in another task. This testing is taking me a little more time, and S3 is already supported in Hive without such S3 integration test. Also, this patch will unblock the other subtasks for the S3 umbrella jira. Does this sound reasonable? > Write temporary data to HDFS when doing inserts on tables located on S3 > --- > > Key: HIVE-14270 > URL: https://issues.apache.org/jira/browse/HIVE-14270 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14270.1.patch > > > Currently, when doing INSERT statements on tables located at S3, Hive writes > and reads temporary (or intermediate) files to S3 as well. > If HDFS is still the default filesystem on Hive, then we can keep such > temporary files on HDFS to keep things run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396367#comment-15396367
]
Naveen Gangam commented on HIVE-14296:
--
No worries. Thank you
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState:
[jira] [Commented] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396366#comment-15396366
]
Sergey Shelukhin commented on HIVE-14336:
-
+1
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396359#comment-15396359
]
Szehon Ho commented on HIVE-14296:
--
Sorry missed this email :), thanks Aihua for pushing it.
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
>
[jira] [Updated] (HIVE-14361) Empty method in TestClientCommandHookFactory
[ https://issues.apache.org/jira/browse/HIVE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14361: -- Status: Patch Available (was: Open) Removed > Empty method in TestClientCommandHookFactory > > > Key: HIVE-14361 > URL: https://issues.apache.org/jira/browse/HIVE-14361 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Trivial > Attachments: HIVE-14361.patch > > > Remove the empty method left in TestClientCommandHookFactory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14361) Empty method in TestClientCommandHookFactory
[ https://issues.apache.org/jira/browse/HIVE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14361: -- Attachment: HIVE-14361.patch > Empty method in TestClientCommandHookFactory > > > Key: HIVE-14361 > URL: https://issues.apache.org/jira/browse/HIVE-14361 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Trivial > Attachments: HIVE-14361.patch > > > Remove the empty method left in TestClientCommandHookFactory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14329) fix flapping qtests - because of output string ordering
[
https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14329:
Attachment: HIVE-14329.1.patch
> fix flapping qtests - because of output string ordering
> ---
>
> Key: HIVE-14329
> URL: https://issues.apache.org/jira/browse/HIVE-14329
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14329.1.patch
>
>
> it's a bit annoying to see some tests come and go in testresults; for example:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/631/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_stats_list_bucket/history/
> These tests fail occasionally because of the ordering is different in the map.
> The usual cause of these failures is a simple hashmap in
> {{MetaDataFormatUtils}}:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java#L411
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14063) beeline to auto connect to the HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396329#comment-15396329
]
Naveen Gangam commented on HIVE-14063:
--
[~vihangk1] This will be useful as user move from CLI to beeline and looking
for similar user experience. Thanks for putting this together.
I have a couple of comments
1) I think it will be more useful to make the default properties file be
specific to a user and should be someplace under the ${user.home} aka user's
home directory (quick suggestion ~/.beeline/whateverfilename.properties) . This
prevents users having to overwrite each other's files and allows them to run
their jobs simultaneously across different HS2 instances. It is very realistic
to have users from different projects using a shared cluster be assigned to
different HS2 instance.
2) Having the password in this file is a big concern of mine too. However, it
is always a good practice on the user's behalf to take away read permissions to
any files that might contain passwords. In lieu of #1, it allows users to
control file permissions to the file thats owned by them (in their home
directories). So if we choose to support passwords in the future, there will be
no changes to the file location.
3) Minor point, since there has been a couple of comments on what the file name
should be, though I would offer my thoughts. Since the file really contains
connection properties for HS2, (and because beeline.properties is already
taken), perhaps something like default-hiveserver.properties or
hiveserver-connection.properties or default-hs2-connection.properties or
something alike?
> beeline to auto connect to the HiveServer2
> --
>
> Key: HIVE-14063
> URL: https://issues.apache.org/jira/browse/HIVE-14063
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: beeline.conf.template
>
>
> Currently one has to give an jdbc:hive2 url in order for Beeline to connect a
> hiveserver2 instance. It would be great if Beeline can get the info somehow
> (from a properties file at a well-known location?) and connect automatically
> if user doesn't specify such a url. If the properties file is not present,
> then beeline would expect user to provide the url and credentials using
> !connect or ./beeline -u .. commands
> While Beeline is flexible (being a mere JDBC client), most environments would
> have just a single HS2. Having users to manually connect into this via either
> "beeline ~/.propsfile" or -u or !connect statements is lowering the
> experience part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14329) fix flapping qtests - because of output string ordering
[
https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14329:
Status: Patch Available (was: Open)
preliminary test run
> fix flapping qtests - because of output string ordering
> ---
>
> Key: HIVE-14329
> URL: https://issues.apache.org/jira/browse/HIVE-14329
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14329.1.patch
>
>
> it's a bit annoying to see some tests come and go in testresults; for example:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/631/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_stats_list_bucket/history/
> These tests fail occasionally because of the ordering is different in the map.
> The usual cause of these failures is a simple hashmap in
> {{MetaDataFormatUtils}}:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java#L411
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396299#comment-15396299
]
Aihua Xu commented on HIVE-14251:
-
[~ctang.ma], [~mohitsabharwal] and [~ashutoshc] Can you take another look at
the patch? Tried to follow what Postgres and Oracle do and we will now force to
use cast if the types are not in the same type group.
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396294#comment-15396294
]
Aihua Xu commented on HIVE-14251:
-
Those tests are not related.
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396289#comment-15396289
]
Hive QA commented on HIVE-14251:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820180/HIVE-14251.4.patch
{color:green}SUCCESS:{color} +1 due to 15 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10368 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/662/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/662/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820180 - PreCommit-HIVE-MASTER-Build
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396270#comment-15396270
]
Naveen Gangam commented on HIVE-14296:
--
Thank you [~aihuaxu] [~mohitsabharwal] and [~szehon]
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
[jira] [Commented] (HIVE-14329) fix flapping qtests - because of output string ordering
[
https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396265#comment-15396265
]
Zoltan Haindrich commented on HIVE-14329:
-
removed: inconsitency: {{StatsSetupConst#setColumnStatsState}} would put
lowercase colnames if record is new; original colnames during update/addition
choosen path: case sensitive
> fix flapping qtests - because of output string ordering
> ---
>
> Key: HIVE-14329
> URL: https://issues.apache.org/jira/browse/HIVE-14329
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
>
> it's a bit annoying to see some tests come and go in testresults; for example:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/631/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_stats_list_bucket/history/
> These tests fail occasionally because of the ordering is different in the map.
> The usual cause of these failures is a simple hashmap in
> {{MetaDataFormatUtils}}:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java#L411
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14296:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Naveen for the contribution and Szehon and Mohit for
reviewing.
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
>
[jira] [Commented] (HIVE-14329) fix flapping qtests - because of output string ordering
[
https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396251#comment-15396251
]
Zoltan Haindrich commented on HIVE-14329:
-
removed: upgrade-while-read from: {{StatsSetupConst#areBasicStatsUptoDate}}
> fix flapping qtests - because of output string ordering
> ---
>
> Key: HIVE-14329
> URL: https://issues.apache.org/jira/browse/HIVE-14329
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
>
> it's a bit annoying to see some tests come and go in testresults; for example:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/631/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_stats_list_bucket/history/
> These tests fail occasionally because of the ordering is different in the map.
> The usual cause of these failures is a simple hashmap in
> {{MetaDataFormatUtils}}:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java#L411
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14329) fix flapping qtests - because of output string ordering
[
https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396221#comment-15396221
]
Zoltan Haindrich commented on HIVE-14329:
-
contains: fix for {{StatsSetupConst#setBasicStatsState}}: failed to upgrade
basic stats status
> fix flapping qtests - because of output string ordering
> ---
>
> Key: HIVE-14329
> URL: https://issues.apache.org/jira/browse/HIVE-14329
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
>
> it's a bit annoying to see some tests come and go in testresults; for example:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/631/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_stats_list_bucket/history/
> These tests fail occasionally because of the ordering is different in the map.
> The usual cause of these failures is a simple hashmap in
> {{MetaDataFormatUtils}}:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java#L411
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396216#comment-15396216
]
Matt McCline commented on HIVE-14336:
-
No related failures.
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[
https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14343:
Attachment: HIVE-14343.0.patch
The current query command is saved in context during compilation, so use that.
> HiveDriverRunHookContext's command is null in HS2 mode
> --
>
> Key: HIVE-14343
> URL: https://issues.apache.org/jira/browse/HIVE-14343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
> Attachments: HIVE-14343.0.patch
>
>
> Looking at the {{Driver#runInternal(String command, boolean
> alreadyCompiled)}}:
> {code}
> HiveDriverRunHookContext hookContext = new
> HiveDriverRunHookContextImpl(conf, command);
> // Get all the driver run hooks and pre-execute them.
> List driverRunHooks;
> {code}
> The context is initialized with the {{command}} passed in to the method.
> However, this command is always null if {{alreadyCompiled}} is true, which is
> the case for HS2 mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14348) Add tests for alter table exchange partition
[ https://issues.apache.org/jira/browse/HIVE-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396148#comment-15396148 ] Thejas M Nair commented on HIVE-14348: -- +1 > Add tests for alter table exchange partition > > > Key: HIVE-14348 > URL: https://issues.apache.org/jira/browse/HIVE-14348 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14348.1.patch, HIVE-14348.2.patch, > HIVE-14348.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[
https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14343:
Assignee: Chao Sun
Status: Patch Available (was: Open)
> HiveDriverRunHookContext's command is null in HS2 mode
> --
>
> Key: HIVE-14343
> URL: https://issues.apache.org/jira/browse/HIVE-14343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14343.0.patch
>
>
> Looking at the {{Driver#runInternal(String command, boolean
> alreadyCompiled)}}:
> {code}
> HiveDriverRunHookContext hookContext = new
> HiveDriverRunHookContextImpl(conf, command);
> // Get all the driver run hooks and pre-execute them.
> List driverRunHooks;
> {code}
> The context is initialized with the {{command}} passed in to the method.
> However, this command is always null if {{alreadyCompiled}} is true, which is
> the case for HS2 mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14251:
Status: Patch Available (was: Open)
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14251:
Status: Open (was: Patch Available)
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14341) Altered skewed location is not respected for list bucketing
[
https://issues.apache.org/jira/browse/HIVE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396138#comment-15396138
]
Aihua Xu commented on HIVE-14341:
-
Attached patch-1: removed the code that we will reload skew mapping info from
the directory rather than from the database. Seems incorrect from what I can
tell since it would gather incorrect info after the following command e.g., it
will get a map {{new1 => /user/hive/warehouse/hdfs_skewed/new1}.
alter table list_bucket_single set skewed location
(''1"="/user/hive/warehouse/hdfs_skewed/new1");
> Altered skewed location is not respected for list bucketing
> ---
>
> Key: HIVE-14341
> URL: https://issues.apache.org/jira/browse/HIVE-14341
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14341.1.patch
>
>
> CREATE TABLE list_bucket_single (key STRING, value STRING)
> SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;
> alter table list_bucket_single set skewed location
> (''1"="/user/hive/warehouse/hdfs_skewed/new1");
> While when you insert a row to key 1, the location falls back to the default
> one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14341) Altered skewed location is not respected for list bucketing
[
https://issues.apache.org/jira/browse/HIVE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14341:
Status: Patch Available (was: Open)
> Altered skewed location is not respected for list bucketing
> ---
>
> Key: HIVE-14341
> URL: https://issues.apache.org/jira/browse/HIVE-14341
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14341.1.patch
>
>
> CREATE TABLE list_bucket_single (key STRING, value STRING)
> SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;
> alter table list_bucket_single set skewed location
> (''1"="/user/hive/warehouse/hdfs_skewed/new1");
> While when you insert a row to key 1, the location falls back to the default
> one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14348) Add tests for alter table exchange partition
[ https://issues.apache.org/jira/browse/HIVE-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14348: Attachment: HIVE-14348.3.patch > Add tests for alter table exchange partition > > > Key: HIVE-14348 > URL: https://issues.apache.org/jira/browse/HIVE-14348 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14348.1.patch, HIVE-14348.2.patch, > HIVE-14348.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14340: Attachment: HIVE-14340.0.patch > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14340: Status: Patch Available (was: Open) > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14340: Attachment: (was: HIVE-14340.0.patch) > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[
https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14340:
Attachment: HIVE-14340.0.patch
Attaching initial patch. This adds a new configuration:
{code}
hive.query.hooks
{code}
which triggers before a query's compilation and after its execution.
> Add a new hook triggers before query compilation and after query execution
> --
>
> Key: HIVE-14340
> URL: https://issues.apache.org/jira/browse/HIVE-14340
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14340.0.patch
>
>
> In some cases we may need to have a hook that activates before a query
> compilation and after its execution. For instance, dynamically generate a UDF
> specifically for the running query and clean up the resource after the query
> is done. The current hooks only covers pre & post semantic analysis, pre &
> post query execution, which doesn't fit the requirement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14341) Altered skewed location is not respected for list bucketing
[
https://issues.apache.org/jira/browse/HIVE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14341:
Attachment: HIVE-14341.1.patch
> Altered skewed location is not respected for list bucketing
> ---
>
> Key: HIVE-14341
> URL: https://issues.apache.org/jira/browse/HIVE-14341
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14341.1.patch
>
>
> CREATE TABLE list_bucket_single (key STRING, value STRING)
> SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;
> alter table list_bucket_single set skewed location
> (''1"="/user/hive/warehouse/hdfs_skewed/new1");
> While when you insert a row to key 1, the location falls back to the default
> one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
