[jira] [Commented] (HIVE-8544) Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with extra double quotes
[
https://issues.apache.org/jira/browse/HIVE-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179241#comment-14179241
]
Vikram Dixit K commented on HIVE-8544:
--
+1 for 0.14
Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with
extra double quotes
--
Key: HIVE-8544
URL: https://issues.apache.org/jira/browse/HIVE-8544
Project: Hive
Issue Type: Bug
Components: CLI, JDBC
Affects Versions: 0.14.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Noticed additional double quotes in select * complex datatypes output
while running it through HiveServer2/Beeline
Actual
{noformat}
[18,19] {col1:alice brown,col2:18}{alice
brown:18}
[18,19] {col1:alice garcia,col2:18} {alice
garcia:18}
[18,19] {col1:alice garcia,col2:18} {alice
garcia:18}
[18,19] {col1:alice thompson,col2:18} {alice
thompson:18}
[18,19] {col1:alice xylophone,col2:18}{alice
xylophone:18}
[18,19] {col1:bob hernandez,col2:18} {bob
hernandez:18}
[18,19] {col1:bob ichabod,col2:18}{bob
ichabod:18}
{noformat}
Expected
{noformat}
[18,19] {col1:alice brown,col2:18}{alice brown:18}
[18,19] {col1:alice garcia,col2:18} {alice garcia:18}
[18,19] {col1:alice garcia,col2:18} {alice garcia:18}
[18,19] {col1:alice thompson,col2:18} {alice thompson:18}
[18,19] {col1:alice xylophone,col2:18}{alice xylophone:18}
[18,19] {col1:bob hernandez,col2:18} {bob hernandez:18}
[18,19] {col1:bob ichabod,col2:18}{bob ichabod:18}
{noformat}
Run these queries through HiveServer2 to reproduce the issue:
{noformat}
create table insert_9 (a arrayint, b structcol1:string,col2:int, c
mapstring, int);
insert overwrite table insert_9 select array(age, age+1), struct(name, age),
map(name, age) from studenttab10k;
select * from insert_9;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8390) CBO produces annoying exception message and wraps exceptions too much
[
https://issues.apache.org/jira/browse/HIVE-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179243#comment-14179243
]
Vikram Dixit K commented on HIVE-8390:
--
+1 for 0.14
CBO produces annoying exception message and wraps exceptions too much
-
Key: HIVE-8390
URL: https://issues.apache.org/jira/browse/HIVE-8390
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor
Fix For: 0.15.0
Attachments: HIVE-8390.01.patch, HIVE-8390.patch
{noformat}
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer$OptiqBasedPlanner.getOptimizedAST(SemanticAnalyzer.java:12249)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer$OptiqBasedPlanner.access$200(SemanticAnalyzer.java:12223)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9904)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:344)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:832)
at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask(TestCliDriver.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.lang.RuntimeException:
java.lang.reflect.UndeclaredThrowableException
at
net.hydromatic.optiq.tools.Frameworks.withPrepare(Frameworks.java:153)
at
net.hydromatic.optiq.tools.Frameworks.withPlanner(Frameworks.java:109)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer$OptiqBasedPlanner.getOptimizedAST(SemanticAnalyzer.java:12243)
... 37 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy28.getDistinctRowCount(Unknown Source)
at
org.eigenbase.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:326)
at
org.eigenbase.rel.rules.LoptOptimizeJoinRule.computeJoinCardinality(LoptOptimizeJoinRule.java:581)
at
org.eigenbase.rel.rules.LoptOptimizeJoinRule.getBestNextFactor(LoptOptimizeJoinRule.java:774)
at
[jira] [Commented] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly
[
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179244#comment-14179244
]
Vikram Dixit K commented on HIVE-6934:
--
+1 for 0.14
PartitionPruner doesn't handle top level constant expression correctly
--
Key: HIVE-6934
URL: https://issues.apache.org/jira/browse/HIVE-6934
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-6934.4.patch, HIVE-6934.5.patch, HIVE-6934.6.patch,
HIVE-6934.7.patch, HIVE-6934.8.patch, HIVE-6934.9.patch, HIVE-6934.91.patch,
HIVE-6934.92.patch
You hit this error indirectly, because how we handle invalid constant
comparisons. Consider:
{code}
create table x(key int, value string) partitioned by (dt int, ts string);
-- both these queries hit this issue
select * from x where key = 'abc';
select * from x where dt = 'abc';
-- the issue is the comparison get converted to the constant false
-- and the PartitionPruner doesn't handle top level constant exprs corrcetly
{code}
Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for
HIVE-5376
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8551) NPE in FunctionRegistry (affects CBO in negative tests)
[ https://issues.apache.org/jira/browse/HIVE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179393#comment-14179393 ] Vikram Dixit K commented on HIVE-8551: -- +1 for 0.14 NPE in FunctionRegistry (affects CBO in negative tests) --- Key: HIVE-8551 URL: https://issues.apache.org/jira/browse/HIVE-8551 Project: Hive Issue Type: Bug Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8551.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8186) Self join may fail if one side have virtual column(s) and other doesn't
[ https://issues.apache.org/jira/browse/HIVE-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179394#comment-14179394 ] Vikram Dixit K commented on HIVE-8186: -- +1 for 0.14 Self join may fail if one side have virtual column(s) and other doesn't --- Key: HIVE-8186 URL: https://issues.apache.org/jira/browse/HIVE-8186 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Navis Fix For: 0.15.0 Attachments: HIVE-8186.1.patch.txt, HIVE-8186.2.patch.txt, HIVE-8186.3.patch.txt, HIVE-8186.4.patch.txt, HIVE-8186.5.patch.txt, HIVE-8186.6.patch.txt See comments. This also fails on trunk, although not on original join_vc query -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.9.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, HIVE-8409.8.patch, HIVE-8409.9.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7914) Simplify join predicates for CBO to avoid cross products
[
https://issues.apache.org/jira/browse/HIVE-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177174#comment-14177174
]
Vikram Dixit K commented on HIVE-7914:
--
+1 for 0.14
Simplify join predicates for CBO to avoid cross products
Key: HIVE-7914
URL: https://issues.apache.org/jira/browse/HIVE-7914
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 0.13.1
Reporter: Mostafa Mokhtar
Assignee: Laljo John Pullokkaran
Fix For: 0.15.0
Attachments: HIVE-7914.patch
Simplify join predicates for disjunctive predicates to avoid cross products.
For TPC-DS query 13 we generate a cross products.
The join predicate on (store_sales x customer_demographics) , (store_sales x
household_demographics) and (store_sales x customer_address) can be pull up
to avoid the cross products
{code}
select avg(ss_quantity)
,avg(ss_ext_sales_price)
,avg(ss_ext_wholesale_cost)
,sum(ss_ext_wholesale_cost)
from store_sales
,store
,customer_demographics
,household_demographics
,customer_address
,date_dim
where store.s_store_sk = store_sales.ss_store_sk
and store_sales.ss_sold_date_sk = date_dim.d_date_sk and date_dim.d_year =
2001
and((store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
and customer_demographics.cd_marital_status = 'M'
and customer_demographics.cd_education_status = '4 yr Degree'
and store_sales.ss_sales_price between 100.00 and 150.00
and household_demographics.hd_dep_count = 3
)or
(store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
and customer_demographics.cd_marital_status = 'D'
and customer_demographics.cd_education_status = 'Primary'
and store_sales.ss_sales_price between 50.00 and 100.00
and household_demographics.hd_dep_count = 1
) or
(store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = ss_cdemo_sk
and customer_demographics.cd_marital_status = 'U'
and customer_demographics.cd_education_status = 'Advanced Degree'
and store_sales.ss_sales_price between 150.00 and 200.00
and household_demographics.hd_dep_count = 1
))
and((store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('KY', 'GA', 'NM')
and store_sales.ss_net_profit between 100 and 200
) or
(store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('MT', 'OR', 'IN')
and store_sales.ss_net_profit between 150 and 300
) or
(store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('WI', 'MO', 'WV')
and store_sales.ss_net_profit between 50 and 250
))
;
{code}
This is the plan currently generated without any predicate simplification
{code}
Warning: Map Join MAPJOIN[59][bigTable=?] in task 'Map 8' is a cross product
Warning: Map Join MAPJOIN[58][bigTable=?] in task 'Map 8' is a cross product
Warning: Shuffle Join JOIN[29][tables = [$hdt$_5, $hdt$_6]] in Stage 'Reducer
2' is a cross product
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
Edges:
Map 7 - Map 8 (BROADCAST_EDGE)
Map 8 - Map 5 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
Reducer 2 - Map 1 (SIMPLE_EDGE), Map 4 (BROADCAST_EDGE), Map 7
(SIMPLE_EDGE)
Reducer 3 - Reducer 2 (SIMPLE_EDGE)
DagName: mmokhtar_20140828155050_7059c24b-501b-4683-86c0-4f3c023f0b0e:1
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: customer_address
Statistics: Num rows: 4000 Data size: 40595195284 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: ca_address_sk (type: int), ca_state (type:
string), ca_country (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 4000 Data size: 40595195284
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
Statistics: Num rows: 4000 Data size: 40595195284
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type:
string), _col2 (type: string)
Execution mode:
[jira] [Commented] (HIVE-8387) add retry logic to ZooKeeperStorage in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177485#comment-14177485 ] Vikram Dixit K commented on HIVE-8387: -- +1 for 0.14 add retry logic to ZooKeeperStorage in WebHCat -- Key: HIVE-8387 URL: https://issues.apache.org/jira/browse/HIVE-8387 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.14.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Priority: Critical Fix For: 0.15.0 Attachments: HIVE-8387.2.patch, HIVE-8387.patch ZK interactions may run into transient errors that should be retried. Currently there is no retry logic in WebHCat for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8492) Enhance Constant Folding to propagate constants for simple expressions
[
https://issues.apache.org/jira/browse/HIVE-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177489#comment-14177489
]
Vikram Dixit K commented on HIVE-8492:
--
+1 for 0.14
Enhance Constant Folding to propagate constants for simple expressions
---
Key: HIVE-8492
URL: https://issues.apache.org/jira/browse/HIVE-8492
Project: Hive
Issue Type: Improvement
Components: Logical Optimizer
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Fix For: 0.15.0
Attachments: HIVE-8492.1.patch, HIVE-8492.patch
currently it propagates only for projects, can be enhanced to propagate for
simple expressions like casts, e.g,
{code}
SELECT src1.key, src1.key + 1, src2.value FROM src src1 join src src2 ON
src1.key = src2.key AND cast(src1.key as double) = 86;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8350) Constant folding should happen before group-by optimization
[ https://issues.apache.org/jira/browse/HIVE-8350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177493#comment-14177493 ] Vikram Dixit K commented on HIVE-8350: -- +1 for 0.14 Constant folding should happen before group-by optimization --- Key: HIVE-8350 URL: https://issues.apache.org/jira/browse/HIVE-8350 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8350.1.patch, HIVE-8350.patch Constant folding should happen as early as possible, so later optimizations can take advantages of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8358) Constant folding should happen before PCR
[ https://issues.apache.org/jira/browse/HIVE-8358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177490#comment-14177490 ] Vikram Dixit K commented on HIVE-8358: -- +1 for 0.14 Constant folding should happen before PCR - Key: HIVE-8358 URL: https://issues.apache.org/jira/browse/HIVE-8358 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8358.1.patch, HIVE-8358.2.patch, HIVE-8358.3.patch, HIVE-8358.4.patch, HIVE-8358.patch So, that partition pruning and transitive predicate propagation may take advantage of constant folding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8497) StatsNoJobTask doesn't close RecordReader, FSDataInputStream of which keeps open to prevent stale data clean
[
https://issues.apache.org/jira/browse/HIVE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177512#comment-14177512
]
Vikram Dixit K commented on HIVE-8497:
--
+1 for 0.14
StatsNoJobTask doesn't close RecordReader, FSDataInputStream of which keeps
open to prevent stale data clean
Key: HIVE-8497
URL: https://issues.apache.org/jira/browse/HIVE-8497
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Environment: Windows
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
Priority: Critical
Attachments: HIVE-8497.1.patch, HIVE-8497.2.patch
run the test
{noformat}
mvn -Phadoop-2 test -Dtest=TestCliDriver -Dqfile=alter_merge_stats_orc.q
{noformat}
to reproduce it. Simply, this query does three data loads which generates
three base orc files.
ANALYZE TABLE...COMPUTE STATISTICS NOSCAN will execute StatsNoJobTask to get
stats, where file handle is held so as not able to clean base file. As a
result, after running ALTER TABLE..CONCATENATE, follow-up queries go to stale
base file and merged file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8501) Fix CBO to use indexes when GenericUDFBridge is applied
[ https://issues.apache.org/jira/browse/HIVE-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177709#comment-14177709 ] Vikram Dixit K commented on HIVE-8501: -- +1 for 0.14 Fix CBO to use indexes when GenericUDFBridge is applied Key: HIVE-8501 URL: https://issues.apache.org/jira/browse/HIVE-8501 Project: Hive Issue Type: Improvement Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Minor Fix For: 0.15.0 Attachments: HIVE-8501.1.patch, HIVE-8501.2.patch, HIVE-8501.3.patch, HIVE-8501.4.patch previous https://issues.apache.org/jira/browse/HIVE-8389 assumes that we have predicate: ((UDFToDouble(key) UDFToDouble(80)) and (UDFToDouble(key) UDFToDouble(100))) for example. This does not work for the case when we have predicate: ((UDFToDouble(key) 80.0) and (UDFToDouble(key) 100.0)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Status: Patch Available (was: Open) Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8479.1.patch, HIVE-8479.2.patch, HIVE-8479.3.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Status: Open (was: Patch Available) Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8479.1.patch, HIVE-8479.2.patch, HIVE-8479.3.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Attachment: HIVE-8479.3.patch Uploading again to trigger run of unit tests. Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8479.1.patch, HIVE-8479.2.patch, HIVE-8479.3.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Component/s: (was: Tez) Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8479.1.patch, HIVE-8479.2.patch, HIVE-8479.3.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.7.patch Address review comments. SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.8.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, HIVE-8409.8.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8349) DISTRIBUTE BY should work with tez auto-parallelism enabled
[ https://issues.apache.org/jira/browse/HIVE-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175368#comment-14175368 ] Vikram Dixit K commented on HIVE-8349: -- +1 for 0.14 DISTRIBUTE BY should work with tez auto-parallelism enabled --- Key: HIVE-8349 URL: https://issues.apache.org/jira/browse/HIVE-8349 Project: Hive Issue Type: Bug Components: Physical Optimizer, Tez Affects Versions: 0.14.0 Reporter: Gopal V Assignee: Gopal V Fix For: 0.14.0 Attachments: HIVE-8349.1.patch, HIVE-8349.2.patch, HIVE-8349.3.patch, HIVE-8349.4.patch Current implementation of DISTRIBUTE BY does not work when tez auto-parallelism is turned on, because of hashCode distribution issues. In case of distribute by, the key is actually zero bytes, with only partitioning enabled via hashCode - this adversely affects the uniform hashing implementation. In an ideal scenario, the edge should go from the ordered kv input to the unordered partitioned edge, to speed up the processing massively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8489) Add sanity check to dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175638#comment-14175638 ] Vikram Dixit K commented on HIVE-8489: -- +1 LGTM. +1 for 0.14 as well. Add sanity check to dynamic partition pruning - Key: HIVE-8489 URL: https://issues.apache.org/jira/browse/HIVE-8489 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8489.1.patch For debugging/safety it would help to make sure we received the right number of events from tez tasks. This is always supposed to be the case, but would be really hard to debug if something goes wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8452) Cleanup handling of resource configuration for tez
[ https://issues.apache.org/jira/browse/HIVE-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175640#comment-14175640 ] Vikram Dixit K commented on HIVE-8452: -- +1 LGTM. +1 for 0.14 as well. Cleanup handling of resource configuration for tez -- Key: HIVE-8452 URL: https://issues.apache.org/jira/browse/HIVE-8452 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8452.1.patch Currently there's a way to configure container size for tez as well as java opts to start containers. However, if you only set container size and not java opts, we fall back to MR settings for java opts which will likely fail (killed by yarn). vcores is always taken from the MR settings. Need to have a way to overwrite that too. Tez 0.5 and onwards will automatically generate the right java opts string when the container/cpus are specified. This is preferable to falling back to MR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8478) Vectorized Reduce-Side Group By doesn't handle Decimal type correctly
[ https://issues.apache.org/jira/browse/HIVE-8478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175655#comment-14175655 ] Vikram Dixit K commented on HIVE-8478: -- +1 for 0.14 as well. Vectorized Reduce-Side Group By doesn't handle Decimal type correctly - Key: HIVE-8478 URL: https://issues.apache.org/jira/browse/HIVE-8478 Project: Hive Issue Type: Bug Components: Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8478.01.patch, HIVE-8478.02.patch, HIVE-8478.03.patch, HIVE-8478.04.patch Note that DecimalColumnVector is different than LongColumnVector because it keeps (an instance) reference to a Decimal128 class whereas the latter stores a long primitive value. So, trouble if you set the reference instead of updating the object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8428) PCR doesnt remove filters involving casts
[ https://issues.apache.org/jira/browse/HIVE-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174122#comment-14174122 ] Vikram Dixit K commented on HIVE-8428: -- +1 for 0.14 PCR doesnt remove filters involving casts - Key: HIVE-8428 URL: https://issues.apache.org/jira/browse/HIVE-8428 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8428.1.patch, HIVE-8428.2.patch, HIVE-8428.3.patch, HIVE-8428.patch e.g., select key,value from srcpart where hr = cast(11 as double); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8476) JavaDoc updates to HiveEndPoint.newConnection() for secure streaming with Kerberos
[ https://issues.apache.org/jira/browse/HIVE-8476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174595#comment-14174595 ] Vikram Dixit K commented on HIVE-8476: -- +1 for 0.14 JavaDoc updates to HiveEndPoint.newConnection() for secure streaming with Kerberos -- Key: HIVE-8476 URL: https://issues.apache.org/jira/browse/HIVE-8476 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.13.1 Reporter: Roshan Naik Assignee: Roshan Naik Fix For: 0.14.0 Attachments: HIVE-8476.patch Add additional notes on using kerberos authenticated streaming connection in HiveEndPoint.newConnection() method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8464) Vectorized reducer nested group by query returns wrong results
[ https://issues.apache.org/jira/browse/HIVE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174596#comment-14174596 ] Vikram Dixit K commented on HIVE-8464: -- +1 for 0.14 Vectorized reducer nested group by query returns wrong results -- Key: HIVE-8464 URL: https://issues.apache.org/jira/browse/HIVE-8464 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8464.01.patch Query has 2 Group By operators in reduce-stage and VectorGroupByOperator can only handle one Group By operator being fed key batches by reduce-shuffle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8460) ORC SARG literal creation for double from float may lead to wrong evaluation of SARG
[ https://issues.apache.org/jira/browse/HIVE-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174597#comment-14174597 ] Vikram Dixit K commented on HIVE-8460: -- +1 for 0.14 ORC SARG literal creation for double from float may lead to wrong evaluation of SARG Key: HIVE-8460 URL: https://issues.apache.org/jira/browse/HIVE-8460 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.13.1 Reporter: Ashutosh Chauhan Assignee: Prasanth J Attachments: HIVE-8460.1.patch, HIVE-8460.2.patch e.g., expression like d = 0.22 where 0.22 is of float type, gets converted to double d = 0.21986 in SARG creation. This will cause erroneous evaluation of SARG expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8462) CBO duplicates columns
[
https://issues.apache.org/jira/browse/HIVE-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174627#comment-14174627
]
Vikram Dixit K commented on HIVE-8462:
--
+1 for 0.14
CBO duplicates columns
--
Key: HIVE-8462
URL: https://issues.apache.org/jira/browse/HIVE-8462
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-8462.01.patch, HIVE-8462.patch
{noformat}
select *, rank() over(partition by key order by value) as rr from src1
{noformat}
Original plan appears to be incorrect:
{noformat}
HiveProjectRel(key=[$0], value=[$1], (tok_function rank (tok_windowspec
(tok_partitioningspec (tok_distributeby (tok_table_or_col key)) (tok_orderby
(tok_tabsortcolnameasc (tok_table_or_col value))=[$5], rr=[$5])
HiveProjectRel(key=[$0], value=[$1], block__offset__inside__file=[$2],
input__file__name=[$3], row__id=[$4], (tok_function rank (tok_windowspec
(tok_partitioningspec (tok_distributeby (tok_table_or_col key)) (tok_orderby
(tok_tabsortcolnameasc (tok_table_or_col value))=[rank() OVER (PARTITION
BY $0 ORDER BY $1 ROWS BETWEEN 2147483647 FOLLOWING AND 2147483647
PRECEDING)])
HiveTableScanRel(table=[[default.src1]])
{noformat}
and final AST has {noformat}
TOK_SELEXPR
.
TOK_TABLE_OR_COL
$hdt$_0
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
TOK_SELEXPR
.
TOK_TABLE_OR_COL
$hdt$_0
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
rr
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7733) Ambiguous column reference error on query
[
https://issues.apache.org/jira/browse/HIVE-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173031#comment-14173031
]
Vikram Dixit K commented on HIVE-7733:
--
+1 for 0.14
Ambiguous column reference error on query
-
Key: HIVE-7733
URL: https://issues.apache.org/jira/browse/HIVE-7733
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Jason Dere
Assignee: Navis
Fix For: 0.15.0
Attachments: HIVE-7733.1.patch.txt, HIVE-7733.2.patch.txt,
HIVE-7733.3.patch.txt, HIVE-7733.4.patch.txt, HIVE-7733.5.patch.txt,
HIVE-7733.6.patch.txt, HIVE-7733.7.patch.txt
{noformat}
CREATE TABLE agg1
(
col0 INT,
col1 STRING,
col2 DOUBLE
);
explain SELECT single_use_subq11.a1 AS a1,
single_use_subq11.a2 AS a2
FROM (SELECT Sum(agg1.col2) AS a1
FROM agg1
GROUP BY agg1.col0) single_use_subq12
JOIN (SELECT alias.a2 AS a0,
alias.a1 AS a1,
alias.a1 AS a2
FROM (SELECT agg1.col1 AS a0,
'42' AS a1,
agg1.col0 AS a2
FROM agg1
UNION ALL
SELECT agg1.col1 AS a0,
'41' AS a1,
agg1.col0 AS a2
FROM agg1) alias
GROUP BY alias.a2,
alias.a1) single_use_subq11
ON ( single_use_subq11.a0 = single_use_subq11.a0 );
{noformat}
Gets the following error:
FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
Looks like this query had been working in 0.12 but starting failing with this
error in 0.13
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6715) Hive JDBC should include username into open session request for non-sasl connection
[ https://issues.apache.org/jira/browse/HIVE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173033#comment-14173033 ] Vikram Dixit K commented on HIVE-6715: -- +1 for 0.14 Hive JDBC should include username into open session request for non-sasl connection --- Key: HIVE-6715 URL: https://issues.apache.org/jira/browse/HIVE-6715 Project: Hive Issue Type: Bug Components: JDBC Reporter: Srinath Assignee: Prasad Mujumdar Priority: Critical Fix For: 0.14.0 Attachments: HIVE-6715.1.patch, HIVE-6715.2.patch, HIVE-6715.3.patch The only parameter from sessVars that's being set in HiveConnection.openSession() is HS2_PROXY_USER. HIVE_AUTH_USER must also be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Attachment: HIVE-8479.1.patch Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8479.1.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Status: Patch Available (was: Open) Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8479.1.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
Vikram Dixit K created HIVE-8479: Summary: Tez sessions cannot change queues once assigned to one within a CLI session Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8479.1.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8479) Tez sessions cannot change queues once assigned to one within a CLI session
[ https://issues.apache.org/jira/browse/HIVE-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8479: - Attachment: HIVE-8479.2.patch Tez sessions cannot change queues once assigned to one within a CLI session --- Key: HIVE-8479 URL: https://issues.apache.org/jira/browse/HIVE-8479 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8479.1.patch, HIVE-8479.2.patch Observed during execution. Once associated with a queue, the CLI session cannot change queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8415) Vectorized comparison of timestamp and integer needs to treat integer as seconds since epoch
[ https://issues.apache.org/jira/browse/HIVE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171547#comment-14171547 ] Vikram Dixit K commented on HIVE-8415: -- +1 for 0.14 Vectorized comparison of timestamp and integer needs to treat integer as seconds since epoch Key: HIVE-8415 URL: https://issues.apache.org/jira/browse/HIVE-8415 Project: Hive Issue Type: Bug Components: Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8415.01.patch, HIVE-8415.02.patch, HIVE-8415.03.patch, HIVE-8415.04.patch, HIVE-8415.05.patch, HIVE-8415.06.patch Current vectorization code treats integers as nanoseconds which doesn't match non-vectorized Hive behavior that interprets integers and doubles as seconds since epoch. Currently, the patch includes Jason's changes from patch #2 in HIVE-8391. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8389) Fix CBO when indexes are used
[ https://issues.apache.org/jira/browse/HIVE-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171907#comment-14171907 ] Vikram Dixit K commented on HIVE-8389: -- +1 for 0.14 Fix CBO when indexes are used - Key: HIVE-8389 URL: https://issues.apache.org/jira/browse/HIVE-8389 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Fix For: 0.15.0 Attachments: HIVE-8389.1.patch, HIVE-8389.2.patch, HIVE-8389.3.patch Disable CBO when indexes are used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct, having in incorrect context
[ https://issues.apache.org/jira/browse/HIVE-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169538#comment-14169538 ] Vikram Dixit K commented on HIVE-8413: -- +1 for 0.14 [CBO] Handle ill-formed queries which have distinct, having in incorrect context - Key: HIVE-8413 URL: https://issues.apache.org/jira/browse/HIVE-8413 Project: Hive Issue Type: Bug Components: CBO Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8413.1.patch, HIVE-8413.2.patch, HIVE-8413.patch e.g., select hash (distinct key) from src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8421) [CBO] Use OptiqSemanticException in error conditions
[ https://issues.apache.org/jira/browse/HIVE-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169546#comment-14169546 ] Vikram Dixit K commented on HIVE-8421: -- +1 for 0.14 [CBO] Use OptiqSemanticException in error conditions Key: HIVE-8421 URL: https://issues.apache.org/jira/browse/HIVE-8421 Project: Hive Issue Type: Bug Components: CBO Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8421.patch TestNegativeCliDriver -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
[ https://issues.apache.org/jira/browse/HIVE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8443: - Attachment: HIVE-8443.1.patch Disable tez_smb_1 for mapreduce and prevent from test hang -- Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
Vikram Dixit K created HIVE-8443: Summary: Disable tez_smb_1 for mapreduce and prevent from test hang Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
[ https://issues.apache.org/jira/browse/HIVE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169621#comment-14169621 ] Vikram Dixit K commented on HIVE-8443: -- [~jdere] Can you take a look? Disable tez_smb_1 for mapreduce and prevent from test hang -- Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
[ https://issues.apache.org/jira/browse/HIVE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8443: - Status: Patch Available (was: Open) Disable tez_smb_1 for mapreduce and prevent from test hang -- Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch, HIVE-8443.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
[ https://issues.apache.org/jira/browse/HIVE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169733#comment-14169733 ] Vikram Dixit K commented on HIVE-8443: -- Updated. Disable tez_smb_1 for mapreduce and prevent from test hang -- Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch, HIVE-8443.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8443) Disable tez_smb_1 for mapreduce and prevent from test hang
[ https://issues.apache.org/jira/browse/HIVE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8443: - Attachment: HIVE-8443.2.patch Disable tez_smb_1 for mapreduce and prevent from test hang -- Key: HIVE-8443 URL: https://issues.apache.org/jira/browse/HIVE-8443 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Attachments: HIVE-8443.1.patch, HIVE-8443.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8401) OrcFileMergeOperator only close last orc file it opened, which resulted in stale data in table directory
[
https://issues.apache.org/jira/browse/HIVE-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170010#comment-14170010
]
Vikram Dixit K commented on HIVE-8401:
--
+1 for 0.14
OrcFileMergeOperator only close last orc file it opened, which resulted in
stale data in table directory
Key: HIVE-8401
URL: https://issues.apache.org/jira/browse/HIVE-8401
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Environment: Windows Server
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
Priority: Critical
Attachments: HIVE-8401.1.patch, alter_merge_2_orc.q.out
run the test
{noformat}
mvn -Phadoop-2 test -Dtest=TestCliDriver -Dqfile=alter_merge_2_orc.q
{noformat}
to reproduce it. Simply, this query does three data loads which generates
three orc files, ALTER TABLE CONCATENATE tries to merge orc pieces into a
single one which is final file to queried.
Output
\hive\itests\qtest\target\qfile-results\clientpositive\alter_merge_2_orc.q.out
shows # records as 600 that is wrong as opposed to 610 expected.
Because OrcFileMergeOperator only closes last orc file, the 1st and 2nd orc
files still remain in table directory due to failure of deleting unclosed
file for old data clean when MoveTask tries to copy merged orc file from
scratch dir to table dir. Eventually the query goes to old data(1st and 2nd
orc files).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8427) Hive Streaming : secure streaming hangs leading to time outs.
[ https://issues.apache.org/jira/browse/HIVE-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170046#comment-14170046 ] Vikram Dixit K commented on HIVE-8427: -- +1 for 0.14. Hive Streaming : secure streaming hangs leading to time outs. -- Key: HIVE-8427 URL: https://issues.apache.org/jira/browse/HIVE-8427 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Reporter: Roshan Naik Assignee: Roshan Naik Labels: ACID, Streaming Fix For: 0.14.0 Attachments: HIVE-8427.patch Need to enable Thrift Sasl setting for secure mode communcation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8381) Update hive version on trunk to 0.15
[
https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-8381:
-
Attachment: HIVE-8381.4.patch
Itests were missed. There should be an easier way to do this :P
{noformat}
mvn versions:set -DnewVersion=0.15.0-SNAPSHOT -DgenerateBackupPoms=false
{noformat}
did not change it for the itests directory for some reason.
Update hive version on trunk to 0.15
Key: HIVE-8381
URL: https://issues.apache.org/jira/browse/HIVE-8381
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.15.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Attachments: HIVE-8381.1.patch, HIVE-8381.2.patch, HIVE-8381.3.patch,
HIVE-8381.4.patch
In view of the branching for 0.14 done, we need to upgrade the version of
hive on trunk to 0.15.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan
[ https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170300#comment-14170300 ] Vikram Dixit K commented on HIVE-8225: -- +1 for 0.14 CBO trunk merge: union11 test fails due to incorrect plan - Key: HIVE-8225 URL: https://issues.apache.org/jira/browse/HIVE-8225 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Pengcheng Xiong Priority: Critical Fix For: 0.15.0 Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, HIVE-8225.8.patch, HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch The result changes to as if the union didn't have count() inside. The issue can be fixed by using srcunion.value outside the subquery in count (replace count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node from union-ed queries is not present in AST at all, which might cause this result. -Interestingly, adding group by to each query in a union produces completely weird result (count(1) is 309 for each key, whereas it should be 1 and the logical incorrect value if internal count is lost is 500)- Nm, that groups by table column called key, which is weird but is what Hive does -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.3.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.2.patch, HIVE-8409.3.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8453) Fix possible unnecessary deserialization in hive-tez
Vikram Dixit K created HIVE-8453: Summary: Fix possible unnecessary deserialization in hive-tez Key: HIVE-8453 URL: https://issues.apache.org/jira/browse/HIVE-8453 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K HIVE-8409 requires deserialization of the entire row to merge on the sorted columns which is expensive. We could potentially deserialize only the sorted column from the serialized data and save on this cost. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8292) Reading from partitioned bucketed tables has high overhead in MapOperator.cleanUpInputFileChangedOp
[
https://issues.apache.org/jira/browse/HIVE-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167363#comment-14167363
]
Vikram Dixit K commented on HIVE-8292:
--
LGTM +1. +1 for 0.14 as well.
Reading from partitioned bucketed tables has high overhead in
MapOperator.cleanUpInputFileChangedOp
---
Key: HIVE-8292
URL: https://issues.apache.org/jira/browse/HIVE-8292
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Environment: cn105
Reporter: Mostafa Mokhtar
Assignee: Gopal V
Fix For: 0.14.0
Attachments: 2014_09_29_14_46_04.jfr, HIVE-8292.1.patch,
HIVE-8292.2.patch
Reading from bucketed partitioned tables has significantly higher overhead
compared to non-bucketed non-partitioned files.
50% of the profile is spent in MapOperator.cleanUpInputFileChangedOp
5% the CPU in
{code}
Path onepath = normalizePath(onefile);
{code}
And
45% the CPU in
{code}
onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri());
{code}
From the profiler
{code}
Stack Trace Sample CountPercentage(%)
hive.ql.exec.tez.MapRecordSource.processRow(Object) 5,327 62.348
hive.ql.exec.vector.VectorMapOperator.process(Writable)5,326 62.336
hive.ql.exec.Operator.cleanUpInputFileChanged() 4,851 56.777
hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() 4,849 56.753
java.net.URI.relativize(URI) 3,903 45.681
java.net.URI.relativize(URI, URI) 3,903
45.681
java.net.URI.normalize(String) 2,169
25.386
java.net.URI.equal(String, String)
526 6.156
java.net.URI.equalIgnoringCase(String,
String) 1 0.012
java.lang.String.substring(int)
1 0.012
hive.ql.exec.MapOperator.normalizePath(String)506 5.922
org.apache.commons.logging.impl.Log4JLogger.info(Object) 32
0.375
java.net.URI.equals(Object) 12 0.14
java.util.HashMap$KeySet.iterator() 5
0.059
java.util.HashMap.get(Object)4
0.047
java.util.LinkedHashMap.get(Object) 3
0.035
hive.ql.exec.Operator.cleanUpInputFileChanged() 1 0.012
hive.ql.exec.Operator.forward(Object, ObjectInspector) 473 5.536
hive.ql.exec.mr.ExecMapperContext.inputFileChanged()1 0.012
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8420) TestHadoop20SAuthBridge broken with hadoop-1
[
https://issues.apache.org/jira/browse/HIVE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167376#comment-14167376
]
Vikram Dixit K commented on HIVE-8420:
--
Yes. We should get this into 0.14 as well. +1 for 0.14
TestHadoop20SAuthBridge broken with hadoop-1
Key: HIVE-8420
URL: https://issues.apache.org/jira/browse/HIVE-8420
Project: Hive
Issue Type: Test
Components: Tests
Affects Versions: 0.14.0
Reporter: Szehon Ho
Assignee: Szehon Ho
Priority: Blocker
Attachments: HIVE-8420.patch
Looks like TestHadoop20SAuthBridge had new dependencies on hadoop-2 methods,
with upgrade to hadoop2.5.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:[50,44]
cannot find symbol
[ERROR] symbol: class DefaultImpersonationProvider
[ERROR] location: package org.apache.hadoop.security.authorize
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:[133,21]
cannot find symbol
[ERROR] symbol: variable DefaultImpersonationProvider
[ERROR] location: class org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:[296,7]
cannot find symbol
[ERROR] symbol: variable DefaultImpersonationProvider
[ERROR] location: class org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8328) MapJoin implementation in Tez should not reload hashtables
[
https://issues.apache.org/jira/browse/HIVE-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167418#comment-14167418
]
Vikram Dixit K commented on HIVE-8328:
--
+1 LGTM. +1 for 0.14 as well.
MapJoin implementation in Tez should not reload hashtables
---
Key: HIVE-8328
URL: https://issues.apache.org/jira/browse/HIVE-8328
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Blocker
Labels: Regression
Fix For: 0.14.0
Attachments: HIVE-8328.1.patch, HIVE-8328.WIP.patch
{code}
private void loadHashTable() throws HiveException {
if ((this.getExecContext() != null)
((this.getExecContext().getLocalWork() == null) ||
(!this.getExecContext()
.getLocalWork().getInputFileChangeSensitive( {
if (hashTblInitedOnce) {
return;
} else {
hashTblInitedOnce = true;
}
}
{code}
This is not exiting in Tez, even if hashTblInitedOnce=true;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8381) Update hive version on trunk to 0.15
[ https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8381: - Attachment: HIVE-8381.3.patch Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8381.1.patch, HIVE-8381.2.patch, HIVE-8381.3.patch In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8432) Remove tez_smb_1 from tests. (Hopefully reduces test run times)
Vikram Dixit K created HIVE-8432: Summary: Remove tez_smb_1 from tests. (Hopefully reduces test run times) Key: HIVE-8432 URL: https://issues.apache.org/jira/browse/HIVE-8432 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8432) Remove tez_smb_1 from tests. (Hopefully reduces test run times)
[ https://issues.apache.org/jira/browse/HIVE-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8432: - Attachment: HIVE-8432.1.patch Will commit this. Hopefully reduces the wait times on queues. Remove tez_smb_1 from tests. (Hopefully reduces test run times) --- Key: HIVE-8432 URL: https://issues.apache.org/jira/browse/HIVE-8432 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8432.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8432) Remove tez_smb_1 from tests. (Hopefully reduces test run times)
[ https://issues.apache.org/jira/browse/HIVE-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8432: - Status: Patch Available (was: Open) Remove tez_smb_1 from tests. (Hopefully reduces test run times) --- Key: HIVE-8432 URL: https://issues.apache.org/jira/browse/HIVE-8432 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8432.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8432) Remove tez_smb_1 from tests. (Hopefully reduces test run times)
[ https://issues.apache.org/jira/browse/HIVE-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8432: - Resolution: Fixed Status: Resolved (was: Patch Available) Remove tez_smb_1 from tests. (Hopefully reduces test run times) --- Key: HIVE-8432 URL: https://issues.apache.org/jira/browse/HIVE-8432 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8432.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164885#comment-14164885
]
Vikram Dixit K commented on HIVE-7932:
--
+1 for 0.14
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Fix For: 0.15.0
Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8280) CBO : When filter is applied on dimension table PK/FK code path is not in effect.
[
https://issues.apache.org/jira/browse/HIVE-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164892#comment-14164892
]
Vikram Dixit K commented on HIVE-8280:
--
+1 for 0.14
CBO : When filter is applied on dimension table PK/FK code path is not in
effect.
-
Key: HIVE-8280
URL: https://issues.apache.org/jira/browse/HIVE-8280
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 0.14.0
Reporter: Mostafa Mokhtar
Assignee: Harish Butani
Fix For: 0.14.0
Attachments: HIVE-8280.1.patch, HIVE-8280.2.patch, HIVE-8280.3.patch
When a filter is applied on PK side joins don't qualify as PK/FK join.
In getUniqueKeys when a filter is applied on the table the child is no
longer a table scan.
{code}
public SetBitSet getUniqueKeys(ProjectRelBase rel, boolean ignoreNulls) {
RelNode child = rel.getChild();
if (!(child instanceof HiveTableScanRel)) {
FunctionRelNode, Metadata fn = RelMdUniqueKeys.SOURCE.apply(
rel.getClass(), BuiltInMetadata.UniqueKeys.class);
return ((BuiltInMetadata.UniqueKeys) fn.apply(rel))
.getUniqueKeys(ignoreNulls);
}
{code}
Repro
{code}
with ss as
(select
ss_customer_sk, ss_item_sk, ss_ticket_number
from
store_sales,
store
where
s_store_sk = ss_store_sk
and s_market_id = 4),
sr as
(select sr_customer_sk,sr_item_sk ,sr_ticket_number from store_returns, store
where s_store_sk = sr_store_sk and s_market_id=4)
select
count(*)
from
ss,
sr
where
ss_customer_sk = sr_customer_sk
and ss_item_sk = sr_item_sk
and ss_ticket_number = sr_ticket_number;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.2.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.2.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8380) NanoTime class serializes and deserializes Timestamp incorrectly
[ https://issues.apache.org/jira/browse/HIVE-8380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165499#comment-14165499 ] Vikram Dixit K commented on HIVE-8380: -- +1 for 0.14. NanoTime class serializes and deserializes Timestamp incorrectly Key: HIVE-8380 URL: https://issues.apache.org/jira/browse/HIVE-8380 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Blocker Fix For: 0.14.0 Attachments: HIVE-8380.patch, HIVE-8380.patch Same as PARQUET-114 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8407) [CBO] Handle filters with non-boolean return type
[ https://issues.apache.org/jira/browse/HIVE-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165905#comment-14165905 ] Vikram Dixit K commented on HIVE-8407: -- +1 for 0.14 [CBO] Handle filters with non-boolean return type - Key: HIVE-8407 URL: https://issues.apache.org/jira/browse/HIVE-8407 Project: Hive Issue Type: Bug Components: CBO Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8407.patch e.g. select * from src where 'foo'; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8315) CBO : Negate condition underestimates selectivity which results in an in-efficient plan
[
https://issues.apache.org/jira/browse/HIVE-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163650#comment-14163650
]
Vikram Dixit K commented on HIVE-8315:
--
+1 for 0.14
CBO : Negate condition underestimates selectivity which results in an
in-efficient plan
---
Key: HIVE-8315
URL: https://issues.apache.org/jira/browse/HIVE-8315
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 0.14.0
Reporter: Mostafa Mokhtar
Assignee: Harish Butani
Fix For: 0.15.0
Attachments: HIVE-8315.1.patch
For TPC-DS Q64 the predicate cd1.cd_marital_status cd2.cd_marital_status
under estimate the join selectivity by a huge margin and results in
in-efficient join order.
This is a subset of the logical plan showing that item was joined very last
{code}
HiveJoinRel(condition=[=($0, $37)],
joinType=[inner]): rowcount = 1.0, cumulative cost = {6.386017602518958E8
rows, 0.0 cpu, 0.0 io}, id = 3790
HiveJoinRel(condition=[=($0, $33)],
joinType=[inner]): rowcount = 1.0, cumulative cost = {6.386017582518958E8
rows, 0.0 cpu, 0.0 io}, id = 3067
HiveFilterRel(condition=[($30, $32)]):
rowcount = 1.8252236387887635, cumulative cost = {6.386017554266721E8 rows,
0.0 cpu, 0.0 io}, id = 1153
HiveProjectRel(ss_item_sk=[$2],
ss_customer_sk=[$3], ss_cdemo_sk=[$4], ss_hdemo_sk=[$5], ss_addr_sk=[$6],
ss_store_sk=[$7], ss_promo_sk=[$8], ss_ticket_number=[$9],
ss_wholesale_cost=[$10], ss_list_price=[$11], ss_coupon_amt=[$12],
ss_sold_date_sk=[$13], sr_item_sk=[$0], sr_ticket_number=[$1],
c_customer_sk=[$23], c_current_cdemo_sk=[$24], c_current_hdemo_sk=[$25],
c_current_addr_sk=[$26], c_first_shipto_date_sk=[$27],
c_first_sales_date_sk=[$28], d_date_sk=[$14], d_year=[$15], d_date_sk0=[$29],
d_year0=[$30], d_date_sk1=[$31], d_year1=[$32], s_store_sk=[$18],
s_store_name=[$19], s_zip=[$20], cd_demo_sk=[$16], cd_marital_status=[$17],
cd_demo_sk0=[$21], cd_marital_status0=[$22]): rowcount =
3.6246005783468924E7, cumulative cost = {6.386017554266721E8 rows, 0.0 cpu,
0.0 io}, id = 2312
HiveJoinRel(condition=[AND(=($2, $0),
=($9, $1))], joinType=[inner]): rowcount = 3.6246005783468924E7, cumulative
cost = {6.386017554266721E8 rows, 0.0 cpu, 0.0 io}, id = 2310
HiveProjectRel(sr_item_sk=[$1],
sr_ticket_number=[$8]): rowcount = 5.5578005E7, cumulative cost = {0.0 rows,
0.0 cpu, 0.0 io}, id = 912
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200_orig.store_returns]]):
rowcount = 5.5578005E7, cumulative cost = {0}, id = 62
HiveJoinRel(condition=[=($1, $21)],
joinType=[inner]): rowcount = 1.2950939439433252E7, cumulative cost =
{5.700728109872389E8 rows, 0.0 cpu, 0.0 io}, id = 2308
HiveJoinRel(condition=[=($5,
$16)], joinType=[inner]): rowcount = 5491530.921341597, cumulative cost =
{5.629812800658973E8 rows, 0.0 cpu, 0.0 io}, id = 2301
HiveJoinRel(condition=[=($2,
$14)], joinType=[inner]): rowcount = 5491530.921341597, cumulative cost =
{5.574895371445558E8 rows, 0.0 cpu, 0.0 io}, id = 2299
HiveJoinRel(condition=[=($11,
$12)], joinType=[inner]): rowcount = 5491530.921341597, cumulative cost =
{5.500772062232143E8 rows, 0.0 cpu, 0.0 io}, id = 1898
HiveProjectRel(ss_item_sk=[$1], ss_customer_sk=[$2], ss_cdemo_sk=[$3],
ss_hdemo_sk=[$4], ss_addr_sk=[$5], ss_store_sk=[$6], ss_promo_sk=[$7],
ss_ticket_number=[$8], ss_wholesale_cost=[$10], ss_list_price=[$11],
ss_coupon_amt=[$18], ss_sold_date_sk=[$22]): rowcount = 5.50076554E8,
cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 909
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200_orig.store_sales]]):
rowcount = 5.50076554E8, cumulative cost = {0}, id = 55{code}
Query
{code}
select cs1.product_name ,cs1.store_name ,cs1.store_zip ,cs1.b_street_number
,cs1.b_streen_name ,cs1.b_city
,cs1.b_zip ,cs1.c_street_number ,cs1.c_street_name ,cs1.c_city
,cs1.c_zip ,cs1.syear ,cs1.cnt
,cs1.s1 ,cs1.s2 ,cs1.s3
,cs2.s1 ,cs2.s2 ,cs2.s3 ,cs2.syear ,cs2.cnt
from
(select i_product_name as product_name ,i_item_sk as item_sk ,s_store_name as
store_name
,s_zip as store_zip ,ad1.ca_street_number as b_street_number
,ad1.ca_street_name as b_streen_name
[jira] [Commented] (HIVE-8344) Hive on Tez sets mapreduce.framework.name to yarn-tez
[ https://issues.apache.org/jira/browse/HIVE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163931#comment-14163931 ] Vikram Dixit K commented on HIVE-8344: -- +1 LGTM. +1 for 0.14 as well. Hive on Tez sets mapreduce.framework.name to yarn-tez - Key: HIVE-8344 URL: https://issues.apache.org/jira/browse/HIVE-8344 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8344.1.patch, HIVE-8344.2.patch, HIVE-8344.3.patch This was done to run MR jobs when in Tez mode (emulate MR on Tez). However, we don't switch back when the user specifies MR as exec engine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7917) Hive max reducers count has regressed from a prime number to 999 (re-apply HIVE-7158)
[ https://issues.apache.org/jira/browse/HIVE-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164114#comment-14164114 ] Vikram Dixit K commented on HIVE-7917: -- +1 for 0.14 Hive max reducers count has regressed from a prime number to 999 (re-apply HIVE-7158) -- Key: HIVE-7917 URL: https://issues.apache.org/jira/browse/HIVE-7917 Project: Hive Issue Type: Bug Components: Configuration Reporter: Gopal V Assignee: Gopal V Attachments: HIVE-7917.1.patch, HIVE-7917.2.patch HIVE-6037 overwrote configuration changes made by HIVE-7158 for Configuration parameters related to reducer parallelism. The changes need to be reapplied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8363) AccumuloStorageHandler compile failure hadoop-1
[
https://issues.apache.org/jira/browse/HIVE-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164215#comment-14164215
]
Vikram Dixit K commented on HIVE-8363:
--
+1 for 0.14
AccumuloStorageHandler compile failure hadoop-1
---
Key: HIVE-8363
URL: https://issues.apache.org/jira/browse/HIVE-8363
Project: Hive
Issue Type: Bug
Components: StorageHandler
Reporter: Szehon Ho
Assignee: Josh Elser
Priority: Blocker
Fix For: 0.14.0
Attachments: HIVE-8363.1.patch
There's an error about AccumuloStorageHandler compiling on hadoop-1. It
seems the signature of split() is not the same. Looks like we can should use
another utils to fix this.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-accumulo-handler: Compilation failure
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMapper.java:[57,52]
no suitable method found for split(java.lang.String,char)
[ERROR] method
org.apache.hadoop.util.StringUtils.split(java.lang.String,char,char) is not
applicable
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8409) SMB joins fail intermittently on tez
Vikram Dixit K created HIVE-8409: Summary: SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.1.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Patch Available (was: Open) SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Open (was: Patch Available) SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.1.patch SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: (was: HIVE-8409.1.patch) SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Patch Available (was: Open) SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez
[ https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164596#comment-14164596 ] Vikram Dixit K commented on HIVE-8364: -- Superseded by HIVE-8409. We're not waiting for all inputs in MapRecordProcessor on Tez - Key: HIVE-8364 URL: https://issues.apache.org/jira/browse/HIVE-8364 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Vikram Dixit K Fix For: 0.14.0 Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch Seems like this could be a race condition: We're blocking for some inputs to become available, but the main MR input is just assumed ready... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez
[ https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8364: - Resolution: Duplicate Status: Resolved (was: Patch Available) We're not waiting for all inputs in MapRecordProcessor on Tez - Key: HIVE-8364 URL: https://issues.apache.org/jira/browse/HIVE-8364 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Vikram Dixit K Fix For: 0.14.0 Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch Seems like this could be a race condition: We're blocking for some inputs to become available, but the main MR input is just assumed ready... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8376) Umbrella Jira for HiveServer2 dynamic service discovery
[ https://issues.apache.org/jira/browse/HIVE-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162171#comment-14162171 ] Vikram Dixit K commented on HIVE-8376: -- +1 for 0.14 Umbrella Jira for HiveServer2 dynamic service discovery --- Key: HIVE-8376 URL: https://issues.apache.org/jira/browse/HIVE-8376 Project: Hive Issue Type: New Feature Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Labels: TODOC14 Fix For: 0.14.0 Creating an ☂ Jira for documentation purpose. I'll add a detailed doc for the implementation usage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8366) CBO fails if there is a table sample in subquery
[ https://issues.apache.org/jira/browse/HIVE-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162169#comment-14162169 ] Vikram Dixit K commented on HIVE-8366: -- +1 for 0.14 as well. CBO fails if there is a table sample in subquery Key: HIVE-8366 URL: https://issues.apache.org/jira/browse/HIVE-8366 Project: Hive Issue Type: Bug Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8366.patch Bail out from cbo in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8261) CBO : Predicate pushdown is removed by Optiq
[
https://issues.apache.org/jira/browse/HIVE-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162172#comment-14162172
]
Vikram Dixit K commented on HIVE-8261:
--
+1 for 0.14
CBO : Predicate pushdown is removed by Optiq
-
Key: HIVE-8261
URL: https://issues.apache.org/jira/browse/HIVE-8261
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 0.14.0, 0.13.1
Reporter: Mostafa Mokhtar
Assignee: Harish Butani
Fix For: 0.14.0
Attachments: HIVE-8261.1.patch
Plan for TPC-DS Q64 wasn't optimal upon looking at the logical plan I
realized that predicate pushdown is not applied on date_dim d1.
Interestingly before optiq we have the predicate pushed :
{code}
HiveFilterRel(condition=[=($5, $1)])
HiveJoinRel(condition=[=($3, $6)], joinType=[inner])
HiveProjectRel(_o__col0=[$0], _o__col1=[$2], _o__col2=[$3],
_o__col3=[$1])
HiveFilterRel(condition=[=($0, 2000)])
HiveAggregateRel(group=[{0, 1}], agg#0=[count()], agg#1=[sum($2)])
HiveProjectRel($f0=[$4], $f1=[$5], $f2=[$2])
HiveJoinRel(condition=[=($1, $8)], joinType=[inner])
HiveJoinRel(condition=[=($1, $5)], joinType=[inner])
HiveJoinRel(condition=[=($0, $3)], joinType=[inner])
HiveProjectRel(ss_sold_date_sk=[$0], ss_item_sk=[$2],
ss_wholesale_cost=[$11])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.store_sales]])
HiveProjectRel(d_date_sk=[$0], d_year=[$6])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.date_dim]])
HiveFilterRel(condition=[AND(in($2, 'maroon', 'burnished',
'dim', 'steel', 'navajo', 'chocolate'), between(false, $1, 35, +(35, 10)),
between(false, $1, +(35, 1), +(35, 15)))])
HiveProjectRel(i_item_sk=[$0], i_current_price=[$5],
i_color=[$17])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.item]])
HiveProjectRel(_o__col0=[$0])
HiveAggregateRel(group=[{0}])
HiveProjectRel($f0=[$0])
HiveJoinRel(condition=[AND(=($0, $2), =($1, $3))],
joinType=[inner])
HiveProjectRel(cs_item_sk=[$15],
cs_order_number=[$17])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.catalog_sales]])
HiveProjectRel(cr_item_sk=[$2], cr_order_number=[$16])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.catalog_returns]])
HiveProjectRel(_o__col0=[$0], _o__col1=[$2], _o__col3=[$1])
HiveFilterRel(condition=[=($0, +(2000, 1))])
HiveAggregateRel(group=[{0, 1}], agg#0=[count()])
HiveProjectRel($f0=[$4], $f1=[$5], $f2=[$2])
HiveJoinRel(condition=[=($1, $8)], joinType=[inner])
HiveJoinRel(condition=[=($1, $5)], joinType=[inner])
HiveJoinRel(condition=[=($0, $3)], joinType=[inner])
HiveProjectRel(ss_sold_date_sk=[$0], ss_item_sk=[$2],
ss_wholesale_cost=[$11])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.store_sales]])
HiveProjectRel(d_date_sk=[$0], d_year=[$6])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.date_dim]])
HiveFilterRel(condition=[AND(in($2, 'maroon', 'burnished',
'dim', 'steel', 'navajo', 'chocolate'), between(false, $1, 35, +(35, 10)),
between(false, $1, +(35, 1), +(35, 15)))])
HiveProjectRel(i_item_sk=[$0], i_current_price=[$5],
i_color=[$17])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.item]])
HiveProjectRel(_o__col0=[$0])
HiveAggregateRel(group=[{0}])
HiveProjectRel($f0=[$0])
HiveJoinRel(condition=[AND(=($0, $2), =($1, $3))],
joinType=[inner])
HiveProjectRel(cs_item_sk=[$15],
cs_order_number=[$17])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.catalog_sales]])
HiveProjectRel(cr_item_sk=[$2], cr_order_number=[$16])
HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_200.catalog_returns]])
{code}
While after Optiq the filter on date_dim gets pulled up the plan
{code}
HiveFilterRel(condition=[=($5, $1)]): rowcount = 1.0, cumulative cost =
{5.50188454E8 rows, 0.0 cpu, 0.0 io}, id = 6895
HiveProjectRel(_o__col0=[$0], _o__col1=[$1], _o__col2=[$2],
_o__col3=[$3], _o__col00=[$4], _o__col10=[$5], _o__col30=[$6]): rowcount =
1.0, cumulative cost =
[jira] [Commented] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
[ https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162174#comment-14162174 ] Vikram Dixit K commented on HIVE-8258: -- +1 for 0.14 Compactor cleaners can be starved on a busy table or partition. --- Key: HIVE-8258 URL: https://issues.apache.org/jira/browse/HIVE-8258 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Alan Gates Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, HIVE-8258.patch Currently the cleaning thread in the compactor does not run on a table or partition while any locks are held on this partition. This leaves it open to starvation in the case of a busy table or partition. It only needs to wait until all locks on the table/partition at the time of the compaction have expired. Any jobs initiated after that (and thus any locks obtained) will be for the new versions of the files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8381) Update hive version on trunk to 0.15
Vikram Dixit K created HIVE-8381: Summary: Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8381) Update hive version on trunk to 0.15
[ https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8381: - Attachment: HIVE-8381.1.patch [~brocknoland] Updated version of trunk. Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8381.1.patch In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8381) Update hive version on trunk to 0.15
[ https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8381: - Status: Patch Available (was: Open) Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8381.1.patch In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8360) Add cross cluster support for webhcat E2E tests
[ https://issues.apache.org/jira/browse/HIVE-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162437#comment-14162437 ] Vikram Dixit K commented on HIVE-8360: -- +1 for 0.14. Add cross cluster support for webhcat E2E tests --- Key: HIVE-8360 URL: https://issues.apache.org/jira/browse/HIVE-8360 Project: Hive Issue Type: Test Components: Tests, WebHCat Environment: Secure cluster Reporter: Aswathy Chellammal Sreekumar Priority: Critical Fix For: 0.14.0 Attachments: AD-MIT.patch In current Webhcat E2E test setup, cross domain secure cluster runs will fail since the realm name for user principles are not included in the kinit command. This patch concatenates the realm name to the user principal there by resulting in a successful kinit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8393) Handle SIGINT on Tez
[ https://issues.apache.org/jira/browse/HIVE-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162925#comment-14162925 ] Vikram Dixit K commented on HIVE-8393: -- +1 LGTM pending tests. +1 for 0.14 as well. Handle SIGINT on Tez Key: HIVE-8393 URL: https://issues.apache.org/jira/browse/HIVE-8393 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8393.1.patch The signal handler in hive currently only interrupts MR jobs. The same logic can be used to interrupt Tez dags as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6972) jdbc HTTP configuration options should be part of sessionConf part of connection string
[ https://issues.apache.org/jira/browse/HIVE-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162937#comment-14162937 ] Vikram Dixit K commented on HIVE-6972: -- +1 for 0.14 jdbc HTTP configuration options should be part of sessionConf part of connection string --- Key: HIVE-6972 URL: https://issues.apache.org/jira/browse/HIVE-6972 Project: Hive Issue Type: Bug Components: HiveServer2, JDBC Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-6972.1.patch, HIVE-6972.2.patch, HIVE-6972.3.patch The http connection parameters are currently part of the HiveConf section of the connection string. It should ideally be part of SessionConf section, as that is where rest of the connection parameters are. HTTP transport parameters are not part of the hiveconfiguration parameters that need to be set. ie instead of jdbc:hive2://host:port/db?hive.server2.transport.mode=http;hive.server2.thrift.http.path=http_endpoint it should be - jdbc:hive2://host:port/db;transportMode=http;httpPath=http_endpoint -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8272) Query with particular decimal expression causes NPE during execution initialization
[
https://issues.apache.org/jira/browse/HIVE-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162964#comment-14162964
]
Vikram Dixit K commented on HIVE-8272:
--
+1 for 0.14
Query with particular decimal expression causes NPE during execution
initialization
---
Key: HIVE-8272
URL: https://issues.apache.org/jira/browse/HIVE-8272
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Reporter: Matt McCline
Assignee: Jason Dere
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-8272.1.patch
Query:
{code}
select
cast(sum(dc)*100 as decimal(11,3)) as c1
from somedecimaltable
order by c1
limit 100;
{code}
Fails during execution initialization due to *null* ExprNodeDesc.
Noticed while trying to simplify a Vectorization issue and realized it was a
more general issue.
{code}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
... 22 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initializeOp(ReduceSinkOperator.java:215)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:427)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:425)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.getExprString(ExprNodeGenericFuncDesc.java:154)
at
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.getExprString(ExprNodeGenericFuncDesc.java:154)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initializeOp(ReduceSinkOperator.java:148)
... 38 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8381) Update hive version on trunk to 0.15
[ https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8381: - Attachment: HIVE-8381.2.patch Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8381.1.patch, HIVE-8381.2.patch In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8381) Update hive version on trunk to 0.15
[ https://issues.apache.org/jira/browse/HIVE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163114#comment-14163114 ] Vikram Dixit K commented on HIVE-8381: -- My bad. Partial patch. Version 2 updates all poms. Update hive version on trunk to 0.15 Key: HIVE-8381 URL: https://issues.apache.org/jira/browse/HIVE-8381 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-8381.1.patch, HIVE-8381.2.patch In view of the branching for 0.14 done, we need to upgrade the version of hive on trunk to 0.15. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8336) Update pom, now that Optiq is renamed to Calcite
[
https://issues.apache.org/jira/browse/HIVE-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14160622#comment-14160622
]
Vikram Dixit K commented on HIVE-8336:
--
+1 for 0.14
Update pom, now that Optiq is renamed to Calcite
Key: HIVE-8336
URL: https://issues.apache.org/jira/browse/HIVE-8336
Project: Hive
Issue Type: Bug
Reporter: Julian Hyde
Assignee: Gunther Hagleitner
Fix For: 0.14.0
Attachments: HIVE-8336.1.patch
Apache Optiq is in the process of renaming to Apache Calcite. See INFRA-8413
and OPTIQ-430.
There is not yet a snapshot of {groupId: 'org.apache.calcite', artifactId:
'calcite-*'} deployed to nexus. When there is, I'll post a patch to pom.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8348) Fix Hive to match changes introduced by TEZ-1510
[
https://issues.apache.org/jira/browse/HIVE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161237#comment-14161237
]
Vikram Dixit K commented on HIVE-8348:
--
+1 for 0.14
Fix Hive to match changes introduced by TEZ-1510
Key: HIVE-8348
URL: https://issues.apache.org/jira/browse/HIVE-8348
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Critical
Fix For: 0.14.0
Attachments: HIVE-8348.1.patch
TEZ-1510 changes the way configuration objects are handled in tez, by
avoiding polluting the {{new Configuration();}} object within HiveServer2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8372) Potential NPE in Tez MergeFileRecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161355#comment-14161355 ] Vikram Dixit K commented on HIVE-8372: -- Pending tests pass. Potential NPE in Tez MergeFileRecordProcessor - Key: HIVE-8372 URL: https://issues.apache.org/jira/browse/HIVE-8372 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Attachments: HIVE-8372.1.patch MergeFileRecordProcessor retrieves map work from cache. This map work can be instance of merge file work. When the merge file work already exists in the cache casting the map work to merge file work is missing which will result in NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8372) Potential NPE in Tez MergeFileRecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161354#comment-14161354 ] Vikram Dixit K commented on HIVE-8372: -- +1 LGTM. +1 for 0.14 as well. Potential NPE in Tez MergeFileRecordProcessor - Key: HIVE-8372 URL: https://issues.apache.org/jira/browse/HIVE-8372 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Attachments: HIVE-8372.1.patch MergeFileRecordProcessor retrieves map work from cache. This map work can be instance of merge file work. When the merge file work already exists in the cache casting the map work to merge file work is missing which will result in NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez
[ https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8364: - Attachment: HIVE-8364.2.patch Improves on the proposed patch. We're not waiting for all inputs in MapRecordProcessor on Tez - Key: HIVE-8364 URL: https://issues.apache.org/jira/browse/HIVE-8364 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Vikram Dixit K Fix For: 0.14.0 Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch Seems like this could be a race condition: We're blocking for some inputs to become available, but the main MR input is just assumed ready... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8322) VectorReduceSinkOperator: ClassCastException: ~StandardUnionObjectInspector$StandardUnion cannot be cast to ~IntWritable
[ https://issues.apache.org/jira/browse/HIVE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158232#comment-14158232 ] Vikram Dixit K commented on HIVE-8322: -- +1 for 0.14 VectorReduceSinkOperator: ClassCastException: ~StandardUnionObjectInspector$StandardUnion cannot be cast to ~IntWritable Key: HIVE-8322 URL: https://issues.apache.org/jira/browse/HIVE-8322 Project: Hive Issue Type: Bug Components: Tez, Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8322.01.patch, HIVE-8322.02.patch, HIVE-8322.03.patch, HIVE-8322.04.patch Some queries with count(distinct(..)) fail now in VectorReduceSinkOperator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7960) Upgrade to Hadoop 2.5
[ https://issues.apache.org/jira/browse/HIVE-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158369#comment-14158369 ] Vikram Dixit K commented on HIVE-7960: -- +1 for 0.14 Upgrade to Hadoop 2.5 - Key: HIVE-7960 URL: https://issues.apache.org/jira/browse/HIVE-7960 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Gunther Hagleitner Attachments: HIVE-7960.1.patch Tracking JIRA for upgrading to 2.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8332) Reading an ACID table with vectorization on results in NPE
[
https://issues.apache.org/jira/browse/HIVE-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156960#comment-14156960
]
Vikram Dixit K commented on HIVE-8332:
--
+1 for 0.14.
Reading an ACID table with vectorization on results in NPE
--
Key: HIVE-8332
URL: https://issues.apache.org/jira/browse/HIVE-8332
Project: Hive
Issue Type: Bug
Components: Transactions, Vectorization
Affects Versions: 0.14.0
Reporter: Alan Gates
Assignee: Alan Gates
Priority: Critical
Fix For: 0.14.0
Attachments: HIVE-8332.patch
On a transactional table, insert some data, then with vectorization turned on
do a select. The result is:
{code}
Caused by: java.lang.NullPointerException at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$1.getObjectInspector(OrcInputFormat.java:1137)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowReader.init(VectorizedOrcAcidRowReader.java:61)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1041)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:246)
... 25 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7695) hive stats issue when insert query is appending data into table
[
https://issues.apache.org/jira/browse/HIVE-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157154#comment-14157154
]
Vikram Dixit K commented on HIVE-7695:
--
+1 for 0.14
hive stats issue when insert query is appending data into table
---
Key: HIVE-7695
URL: https://issues.apache.org/jira/browse/HIVE-7695
Project: Hive
Issue Type: Bug
Components: Logical Optimizer, Statistics
Affects Versions: 0.13.0, 0.13.1
Reporter: Thejas M Nair
Assignee: Ashutosh Chauhan
Fix For: 0.15.0
Attachments: HIVE-7695-test.patch, HIVE-7695.1.patch,
HIVE-7695.2.patch, HIVE-7695.3.patch, HIVE-7695.4.patch, tests.patch
When 'insert' is used to append (not overrite), the stats don't seem to get
updated.
{code}
set hive.compute.query.using.stats=true;
set hive.stats.autogather=true;
//insert 4 rows
insert into table t2 select * from t1;
select count(*) from t2;
4
//insert 4 again (appending instead of overrite)
insert into table t2 select * from t1;
select count(*) from t2;
4
select * from t2;
1
2
3
4
1
2
3
4
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8318) Null Scan optimizer throws exception when no partitions are selected
[
https://issues.apache.org/jira/browse/HIVE-8318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157157#comment-14157157
]
Vikram Dixit K commented on HIVE-8318:
--
+1 for 0.14
Null Scan optimizer throws exception when no partitions are selected
Key: HIVE-8318
URL: https://issues.apache.org/jira/browse/HIVE-8318
Project: Hive
Issue Type: Bug
Components: Physical Optimizer
Affects Versions: 0.14.0
Reporter: Mostafa Mokhtar
Assignee: Ashutosh Chauhan
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-8318.patch
Run query that involves a join and constant folding that is always false
{code}
select s_store_id as store_id,
sum(ss_ext_sales_price) as sales
from store_sales,
date_dim,
store
where ss_sold_date_sk = d_date_sk
and d_date between cast('1998-08-04' as date)
and (cast('1998-09-04' as date))
and ss_store_sk = s_store_sk and 1 = 2
group by s_store_id
{code}
If vectorization is enable we hit this exception
{code}
, TaskAttempt 3 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:172)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:167)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:93)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:272)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:164)
... 13 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:85)
... 16 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.NullWritable
cannot be cast to org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.processOp(VectorFilterOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:801)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:801)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
... 17 more
{code}
If vectorization is disabled we hit this exception
{code}
Caused by: java.lang.RuntimeException: cannot find field ss_store_sk from
[org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@52c64dde]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldRef(UnionStructObjectInspector.java:112)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:940)
at
[jira] [Commented] (HIVE-8240) VectorColumnAssignFactory throws Incompatible Bytes vector column and primitive category VARCHAR
[ https://issues.apache.org/jira/browse/HIVE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157331#comment-14157331 ] Vikram Dixit K commented on HIVE-8240: -- +1 for 0.14 VectorColumnAssignFactory throws Incompatible Bytes vector column and primitive category VARCHAR -- Key: HIVE-8240 URL: https://issues.apache.org/jira/browse/HIVE-8240 Project: Hive Issue Type: Bug Components: Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8240.01.patch, HIVE-8240.02.patch, HIVE-8240.04.patch, HIVE-8240.05.patch, HIVE-8240.06.patch, HIVE-8240.07.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
