[jira] [Updated] (HIVE-7653) Hive AvroSerDe does not support circular references in Schema
[
https://issues.apache.org/jira/browse/HIVE-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7653:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Yea the test failures dont look related. Committed to trunk, thanks for the
contribution.
Hive AvroSerDe does not support circular references in Schema
-
Key: HIVE-7653
URL: https://issues.apache.org/jira/browse/HIVE-7653
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Sachin Goyal
Assignee: Sachin Goyal
Attachments: HIVE-7653.1.patch, HIVE-7653.2.patch, HIVE-7653.3.patch,
HIVE-7653.4.patch, HIVE-7653.5.patch, HIVE-7653.6.patch, HIVE-7653.6.patch
Avro allows nullable circular references but Hive AvroSerDe does not.
Example of circular references (passing in Avro but failing in AvroSerDe):
{code}
class AvroCycleParent {
AvroCycleChild child;
public AvroCycleChild getChild () {return child;}
public void setChild (AvroCycleChild child) {this.child = child;}
}
class AvroCycleChild {
AvroCycleParent parent;
public AvroCycleParent getParent () {return parent;}
public void setParent (AvroCycleParent parent) {this.parent = parent;}
}
{code}
Due to this discrepancy, Hive is unable to read Avro records having
circular-references. For some third-party code with such references, it
becomes very hard to directly serialize it with Avro and use in Hive.
I have a patch for this with a unit-test and I will submit it shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327096#comment-14327096 ] Jason Dere commented on HIVE-3454: -- If this config setting is initialized once in a static block, then for hiveserver2 all subsequent sessions would be stuck with the initial setting regardless of the config settings of the session, right? During the MR jobs, would we then see the sec/msec behavior flip to use the session's config settings since the static variable is being initialized for the first time in MR task? Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9613) Left join query plan outputs wrong column when using subquery
[
https://issues.apache.org/jira/browse/HIVE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Xin resolved HIVE-9613.
--
Resolution: Fixed
Fix Version/s: 1.2.0
seems an issue in ColumnPruner and fixed in HIVE-9327
Left join query plan outputs wrong column when using subquery
--
Key: HIVE-9613
URL: https://issues.apache.org/jira/browse/HIVE-9613
Project: Hive
Issue Type: Bug
Components: Parser, Query Planning
Affects Versions: 0.14.0, 1.0.0
Environment: apache hadoop 2.5.1
Reporter: Li Xin
Fix For: 1.2.0
Attachments: test.sql
I have a query that outputs a column with wrong contents when using
subquery,and the contents of that column is equal to another column,not its
own.
I have three tables,as follows:
table 1: _hivetemp.category_city_rank_:
||category||city||rank||
|jinrongfuwu|shanghai|1|
|ktvjiuba|shanghai|2|
table 2:_hivetemp.category_match_:
||src_category_en||src_category_cn||dst_category_en||dst_category_cn||
|danbaobaoxiantouzi|投资担保|担保/贷款|jinrongfuwu|
|zpwentiyingshi|娱乐/休闲|KTV/酒吧|ktvjiuba|
table 3:_hivetemp.city_match_:
||src_city_name_en||dst_city_name_en||city_name_cn||
|sh|shanghai|上海|
And the query is :
{code}
select
a.category,
a.city,
a.rank,
b.src_category_en,
c.src_city_name_en
from
hivetemp.category_city_rank a
left outer join
(select
src_category_en,
dst_category_en
from
hivetemp.category_match) b
on a.category = b.dst_category_en
left outer join
(select
src_city_name_en,
dst_city_name_en
from
hivetemp.city_match) c
on a.city = c.dst_city_name_en
{code}
which shoud output the results as follows,and i test it in hive 0.13:
||category||city||rank||src_category_en||src_city_name_en||
|jinrongfuwu|shanghai|1|danbaobaoxiantouzi|sh|
|ktvjiuba|shanghai|2|zpwentiyingshi|sh|
but int hive0.14,the results in the column *src_category_en* is wrong,and is
just the *city* contents:
||category||city||rank||src_category_en||src_city_name_en||
|jinrongfuwu|shanghai|1|shanghai|sh|
|ktvjiuba|shanghai|2|shanghai|sh|
Using explain to examine the execution plan,i can see the first subquery just
outputs one column of *dst_category_en*,and *src_category_en* is just missing.
{quote}
b:category_match
TableScan
alias: category_match
Statistics: Num rows: 131 Data size: 13149 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: dst_category_en (type: string)
outputColumnNames: _col1
Statistics: Num rows: 131 Data size: 13149 Basic stats:
COMPLETE Column stats: NONE
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9613) Left join query plan outputs wrong column when using subquery
[
https://issues.apache.org/jira/browse/HIVE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327144#comment-14327144
]
Li Xin commented on HIVE-9613:
--
Hi,[~csun], I just tested the upstream trunk and the issue is gone.
Thank you for your time and next time i will do some test in the upstream trunk
first.
I will close this one as fixed~
Left join query plan outputs wrong column when using subquery
--
Key: HIVE-9613
URL: https://issues.apache.org/jira/browse/HIVE-9613
Project: Hive
Issue Type: Bug
Components: Parser, Query Planning
Affects Versions: 0.14.0, 1.0.0
Environment: apache hadoop 2.5.1
Reporter: Li Xin
Fix For: 1.2.0
Attachments: test.sql
I have a query that outputs a column with wrong contents when using
subquery,and the contents of that column is equal to another column,not its
own.
I have three tables,as follows:
table 1: _hivetemp.category_city_rank_:
||category||city||rank||
|jinrongfuwu|shanghai|1|
|ktvjiuba|shanghai|2|
table 2:_hivetemp.category_match_:
||src_category_en||src_category_cn||dst_category_en||dst_category_cn||
|danbaobaoxiantouzi|投资担保|担保/贷款|jinrongfuwu|
|zpwentiyingshi|娱乐/休闲|KTV/酒吧|ktvjiuba|
table 3:_hivetemp.city_match_:
||src_city_name_en||dst_city_name_en||city_name_cn||
|sh|shanghai|上海|
And the query is :
{code}
select
a.category,
a.city,
a.rank,
b.src_category_en,
c.src_city_name_en
from
hivetemp.category_city_rank a
left outer join
(select
src_category_en,
dst_category_en
from
hivetemp.category_match) b
on a.category = b.dst_category_en
left outer join
(select
src_city_name_en,
dst_city_name_en
from
hivetemp.city_match) c
on a.city = c.dst_city_name_en
{code}
which shoud output the results as follows,and i test it in hive 0.13:
||category||city||rank||src_category_en||src_city_name_en||
|jinrongfuwu|shanghai|1|danbaobaoxiantouzi|sh|
|ktvjiuba|shanghai|2|zpwentiyingshi|sh|
but int hive0.14,the results in the column *src_category_en* is wrong,and is
just the *city* contents:
||category||city||rank||src_category_en||src_city_name_en||
|jinrongfuwu|shanghai|1|shanghai|sh|
|ktvjiuba|shanghai|2|shanghai|sh|
Using explain to examine the execution plan,i can see the first subquery just
outputs one column of *dst_category_en*,and *src_category_en* is just missing.
{quote}
b:category_match
TableScan
alias: category_match
Statistics: Num rows: 131 Data size: 13149 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: dst_category_en (type: string)
outputColumnNames: _col1
Statistics: Num rows: 131 Data size: 13149 Basic stats:
COMPLETE Column stats: NONE
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9641) Fill out remaining partition functions in HBaseStore
[
https://issues.apache.org/jira/browse/HIVE-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328003#comment-14328003
]
Hive QA commented on HIVE-9641:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12699721/HIVE-9641.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2823/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2823/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2823/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2823/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/thirdparty
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-jmh/target itests/hive-unit/target itests/custom-serde/target
itests/util/target itests/qtest-spark/target hcatalog/target
hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target service/target contrib/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1660985.
At revision 1660985.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12699721 - PreCommit-HIVE-TRUNK-Build
Fill out remaining partition functions in HBaseStore
Key: HIVE-9641
URL: https://issues.apache.org/jira/browse/HIVE-9641
Project: Hive
Issue Type: Sub-task
Components: Metastore
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-9641.patch
A number of the listPartition and getPartition methods are not implemented.
The rest need to be implemented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9641) Fill out remaining partition functions in HBaseStore
[ https://issues.apache.org/jira/browse/HIVE-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9641: - Status: Patch Available (was: Open) Fill out remaining partition functions in HBaseStore Key: HIVE-9641 URL: https://issues.apache.org/jira/browse/HIVE-9641 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-9641.patch A number of the listPartition and getPartition methods are not implemented. The rest need to be implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Fix version for hbase-metastore branch
This is what we have been doing for cbo work. e.g. https://issues.apache.org/jira/browse/HIVE-9581 On Thu, Feb 19, 2015 at 11:17 AM, Thejas Nair thejas.n...@gmail.com wrote: I agree, using a label for fix version makes sense in this case. I believe that is what had been done for hive-on-spark and hive-on-tez. On Thu, Feb 19, 2015 at 10:56 AM, Alan Gates alanfga...@gmail.com wrote: I've been marking JIRAs on this branch as fixed in 1.2, since that's the next version. But that seems wrong as I doubt this code will be in by 1.2. What's the usual practice here? It seems it would make sense to make a label for this branch and mark them as fixed with that label and then when we actually release this in a version we can update all the JIRAs with that label. Alan.
Re: Fix version for hbase-metastore branch
Looks like hive-on-tez and hive-on-spark didn't update the fix version after merge to trunk. But I think updating the fix version after merge makes sense. On Thu, Feb 19, 2015 at 11:17 AM, Thejas Nair thejas.n...@gmail.com wrote: I agree, using a label for fix version makes sense in this case. I believe that is what had been done for hive-on-spark and hive-on-tez. On Thu, Feb 19, 2015 at 10:56 AM, Alan Gates alanfga...@gmail.com wrote: I've been marking JIRAs on this branch as fixed in 1.2, since that's the next version. But that seems wrong as I doubt this code will be in by 1.2. What's the usual practice here? It seems it would make sense to make a label for this branch and mark them as fixed with that label and then when we actually release this in a version we can update all the JIRAs with that label. Alan.
[jira] [Commented] (HIVE-7653) Hive AvroSerDe does not support circular references in Schema
[
https://issues.apache.org/jira/browse/HIVE-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328050#comment-14328050
]
Sachin Goyal commented on HIVE-7653:
Thank you [~szehon]!
Appreciate your time on reviewing this.
Hive AvroSerDe does not support circular references in Schema
-
Key: HIVE-7653
URL: https://issues.apache.org/jira/browse/HIVE-7653
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Sachin Goyal
Assignee: Sachin Goyal
Attachments: HIVE-7653.1.patch, HIVE-7653.2.patch, HIVE-7653.3.patch,
HIVE-7653.4.patch, HIVE-7653.5.patch, HIVE-7653.6.patch, HIVE-7653.6.patch
Avro allows nullable circular references but Hive AvroSerDe does not.
Example of circular references (passing in Avro but failing in AvroSerDe):
{code}
class AvroCycleParent {
AvroCycleChild child;
public AvroCycleChild getChild () {return child;}
public void setChild (AvroCycleChild child) {this.child = child;}
}
class AvroCycleChild {
AvroCycleParent parent;
public AvroCycleParent getParent () {return parent;}
public void setParent (AvroCycleParent parent) {this.parent = parent;}
}
{code}
Due to this discrepancy, Hive is unable to read Avro records having
circular-references. For some third-party code with such references, it
becomes very hard to directly serialize it with Avro and use in Hive.
I have a patch for this with a unit-test and I will submit it shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9641) Fill out remaining partition functions in HBaseStore
[ https://issues.apache.org/jira/browse/HIVE-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9641: - Attachment: HIVE-9641.patch Fill out remaining partition functions in HBaseStore Key: HIVE-9641 URL: https://issues.apache.org/jira/browse/HIVE-9641 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-9641.patch A number of the listPartition and getPartition methods are not implemented. The rest need to be implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Srinivas updated HIVE-9718:
-
Attachment: HIVE-9718-0.14.patch
HIVE-9718-1.0.patch
HIVE-9718.patch
Patches for trunk, 0.14 release, 1.0 release
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: HIVE-9718-0.14.patch, HIVE-9718-1.0.patch,
HIVE-9718.patch, nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name` string,
`n_regionkey` int,
`n_comment` string)
ROW FORMAT DELIMITED
Re: Fix version for hbase-metastore branch
I agree, using a label for fix version makes sense in this case. I believe that is what had been done for hive-on-spark and hive-on-tez. On Thu, Feb 19, 2015 at 10:56 AM, Alan Gates alanfga...@gmail.com wrote: I've been marking JIRAs on this branch as fixed in 1.2, since that's the next version. But that seems wrong as I doubt this code will be in by 1.2. What's the usual practice here? It seems it would make sense to make a label for this branch and mark them as fixed with that label and then when we actually release this in a version we can update all the JIRAs with that label. Alan.
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328026#comment-14328026 ] david serafini commented on HIVE-7100: -- Looks ok to me. Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: david serafini Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7100.1.patch, HIVE-7100.10.patch, HIVE-7100.11.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, HIVE-7100.8.patch, HIVE-7100.9.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6977) Delete Hiveserver1
[ https://issues.apache.org/jira/browse/HIVE-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327240#comment-14327240 ] Lefty Leverenz commented on HIVE-6977: -- Was the commit to version 1.0.0 reverted, or should this issue have fix version 1.0.0 as well as 1.1.0? Delete Hiveserver1 -- Key: HIVE-6977 URL: https://issues.apache.org/jira/browse/HIVE-6977 Project: Hive Issue Type: Task Components: JDBC, Server Infrastructure Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Labels: TODOC15 Fix For: 1.1.0 Attachments: HIVE-6977.1.patch, HIVE-6977.patch See mailing list discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9724) Hive source pointing to http rather than https using maven
srinivas created HIVE-9724: -- Summary: Hive source pointing to http rather than https using maven Key: HIVE-9724 URL: https://issues.apache.org/jira/browse/HIVE-9724 Project: Hive Issue Type: Bug Reporter: srinivas Refereed from the below link https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-HowdoIimportintoEclipse ? Generating tarball resulted in below error. #mvn clean package -DskipTests -Phadoop-1 -Pdist ERROR:: Failed to execute goal on project hive-shims-common: Could not resolve dependencies for project org.apache.hive.shims:hive-shims-common:jar:1.2.0-SNAPSHOT: Could not transfer artifact jline:jline:jar:2.12 from/to central ( http://repo.maven.apache.org/maven2): Access denied to: http://repo.maven.apache.org/maven2/jline/jline/2.12/jline-2.12.jar, ReasonPhrase:Forbidden. - [Help 1] [ERROR] Please advice -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9723) CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch]
Jesus Camacho Rodriguez created HIVE-9723: - Summary: CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch] Key: HIVE-9723 URL: https://issues.apache.org/jira/browse/HIVE-9723 Project: Hive Issue Type: Sub-task Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9581 started by Jesus Camacho Rodriguez. - CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9722) CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch]
Jesus Camacho Rodriguez created HIVE-9722: - Summary: CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch] Key: HIVE-9722 URL: https://issues.apache.org/jira/browse/HIVE-9722 Project: Hive Issue Type: Sub-task Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327190#comment-14327190 ] Lefty Leverenz commented on HIVE-7100: -- Doc note: This is documented in the wiki for DROP TABLE and ALTER TABLE DROP PARTITION. * [Drop Table | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropTable] * [Drop Partitions | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropPartitions] I included DROP PARTITION based on deletePartitionData() in the patch -- please review my changes and correct anything that's not right. If it's okay, the TODOC14 label can be removed. Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: david serafini Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7100.1.patch, HIVE-7100.10.patch, HIVE-7100.11.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, HIVE-7100.8.patch, HIVE-7100.9.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9388) HiveServer2 fails to reconnect to MetaStore after MetaStore restart
[
https://issues.apache.org/jira/browse/HIVE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327225#comment-14327225
]
Hive QA commented on HIVE-9388:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12699629/HIVE-9388.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7561 tests executed
*Failed tests:*
{noformat}
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2822/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2822/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2822/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12699629 - PreCommit-HIVE-TRUNK-Build
HiveServer2 fails to reconnect to MetaStore after MetaStore restart
---
Key: HIVE-9388
URL: https://issues.apache.org/jira/browse/HIVE-9388
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.14.0, 0.13.1
Reporter: Piotr Ackermann
Attachments: HIVE-9388.2.patch, HIVE-9388.patch
How to reproduce:
# Use Hue to connect to HiveServer2
# Restart Metastore
# Try to execute any query in Hue
HiveServer2 report error:
{quote}
ERROR hive.log: Got exception:
org.apache.thrift.transport.TTransportException null
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1681)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62)
at
org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
[jira] [Updated] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9581: -- Summary: CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] (was: CBO (Calcite Return Path): Translate Join to Hive Op) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Apache Hive 1.1.0 Release Candidate 3
+1 1. Verified signature for bin and src 2. Built src with hadoop2 3. Ran few queries from beeline with src 4. Ran few queries from beeline with bin 5. Verified no SNAPSHOT deps Thanks Szehon On Wed, Feb 18, 2015 at 10:03 PM, Xuefu Zhang xzh...@cloudera.com wrote: +1 1. downloaded the src tarball and built w/ -Phadoop-1/2 2. verified no binary (jars) in the src tarball On Wed, Feb 18, 2015 at 8:56 PM, Brock Noland br...@cloudera.com wrote: +1 verified sigs, hashes, created tables, ran MR on YARN jobs On Wed, Feb 18, 2015 at 8:54 PM, Brock Noland br...@cloudera.com wrote: Apache Hive 1.1.0 Release Candidate 3 is available here: http://people.apache.org/~brock/apache-hive-1.1.0-rc3/ Maven artifacts are available here: https://repository.apache.org/content/repositories/orgapachehive-1026/ Source tag for RC3 is at: http://svn.apache.org/repos/asf/hive/tags/release-1.1.0-rc3/ My key is located here: https://people.apache.org/keys/group/hive.asc Voting will conclude in 72 hours
[jira] [Updated] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9581: -- Affects Version/s: cbo-branch CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9723) CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9723 started by Jesus Camacho Rodriguez. - CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch] -- Key: HIVE-9723 URL: https://issues.apache.org/jira/browse/HIVE-9723 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9722) CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9722: -- Affects Version/s: cbo-branch CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch] --- Key: HIVE-9722 URL: https://issues.apache.org/jira/browse/HIVE-9722 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9722) CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9722 started by Jesus Camacho Rodriguez. - CBO (Calcite Return Path): Translate Sort/Limit to Hive Op [CBO branch] --- Key: HIVE-9722 URL: https://issues.apache.org/jira/browse/HIVE-9722 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9723) CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9723: -- Affects Version/s: cbo-branch CBO (Calcite Return Path): Translate Union to Hive Op [CBO branch] -- Key: HIVE-9723 URL: https://issues.apache.org/jira/browse/HIVE-9723 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
JIRA change log for 1.0.0 only has 9 bugs
The release 1.0.0 JIRA change log https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12329278styleName=TextprojectId=12310843 linked to the downloads http://hive.apache.org/downloads.html page only has 9 bugs but the downloaded release notes file has more than 50. Spot checking shows that Brock removed Fix Version 1.0.0 from the missing bugs when he changed 0.15.0 to 1.1.0. For example, HIVE-6977 https://issues.apache.org/jira/browse/HIVE-6977 (Delete Hiveserver1) only says it's fixed in 1.1.0 although the patch is also in 1.0.0. Brock, can you revert those changes easily or do we need to go through the list manually? -- Lefty
[jira] [Updated] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9581: -- Attachment: (was: HIVE-9581.cbo.patch) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-9581.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327372#comment-14327372
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327377#comment-14327377
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327371#comment-14327371
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327374#comment-14327374
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Updated] (HIVE-9388) HiveServer2 fails to reconnect to MetaStore after MetaStore restart
[
https://issues.apache.org/jira/browse/HIVE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mariusz Strzelecki updated HIVE-9388:
-
Affects Version/s: 1.0.0
HiveServer2 fails to reconnect to MetaStore after MetaStore restart
---
Key: HIVE-9388
URL: https://issues.apache.org/jira/browse/HIVE-9388
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.14.0, 0.13.1, 1.0.0
Reporter: Piotr Ackermann
Attachments: HIVE-9388.2.patch, HIVE-9388.patch
How to reproduce:
# Use Hue to connect to HiveServer2
# Restart Metastore
# Try to execute any query in Hue
HiveServer2 report error:
{quote}
ERROR hive.log: Got exception:
org.apache.thrift.transport.TTransportException null
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1681)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62)
at
org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy19.getSchemas(Unknown Source)
at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:277)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:436)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at
[jira] [Updated] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9581: -- Attachment: HIVE-9581.cbo.patch [~ashutoshc] [~jpullokkaran] CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-9581.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9388) HiveServer2 fails to reconnect to MetaStore after MetaStore restart
[
https://issues.apache.org/jira/browse/HIVE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327383#comment-14327383
]
Mariusz Strzelecki commented on HIVE-9388:
--
Unit test failure seems to be unrelated.
Bug can be also reproduced without hue:
1. Start hive metastore
2. Start hive cli
3. Restart hive metastore
4. Run show databases; in cli - an exception is thrown and client is not
reconnected
Tested on 1.0.0 release.
HiveServer2 fails to reconnect to MetaStore after MetaStore restart
---
Key: HIVE-9388
URL: https://issues.apache.org/jira/browse/HIVE-9388
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.14.0, 0.13.1
Reporter: Piotr Ackermann
Attachments: HIVE-9388.2.patch, HIVE-9388.patch
How to reproduce:
# Use Hue to connect to HiveServer2
# Restart Metastore
# Try to execute any query in Hue
HiveServer2 report error:
{quote}
ERROR hive.log: Got exception:
org.apache.thrift.transport.TTransportException null
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1681)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62)
at
org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy19.getSchemas(Unknown Source)
at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:277)
at
[jira] [Updated] (HIVE-6977) Delete Hiveserver1
[ https://issues.apache.org/jira/browse/HIVE-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6977: Fix Version/s: 1.0.0 Delete Hiveserver1 -- Key: HIVE-6977 URL: https://issues.apache.org/jira/browse/HIVE-6977 Project: Hive Issue Type: Task Components: JDBC, Server Infrastructure Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Labels: TODOC15 Fix For: 1.0.0, 1.1.0 Attachments: HIVE-6977.1.patch, HIVE-6977.patch See mailing list discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327375#comment-14327375
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327376#comment-14327376
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327373#comment-14327373
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327370#comment-14327370
]
Pavan Srinivas commented on HIVE-9718:
--
Thanks for the comment. Yes, this optimization helps in preparing a better
plan. Having said that, this bug breaks backward compatibility(this query works
on Hive-0.13.1 and Hive-0.11). Ideal way is to fix this and make it compatible.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name`
[jira] [Commented] (HIVE-6977) Delete Hiveserver1
[ https://issues.apache.org/jira/browse/HIVE-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327439#comment-14327439 ] Thejas M Nair commented on HIVE-6977: - This is also part of 1.0.0 release. Delete Hiveserver1 -- Key: HIVE-6977 URL: https://issues.apache.org/jira/browse/HIVE-6977 Project: Hive Issue Type: Task Components: JDBC, Server Infrastructure Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Labels: TODOC15 Fix For: 1.0.0, 1.1.0 Attachments: HIVE-6977.1.patch, HIVE-6977.patch See mailing list discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9581: -- Attachment: HIVE-9581.cbo.patch CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 1.2.0 Attachments: HIVE-9581.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327384#comment-14327384
]
Pavan Srinivas commented on HIVE-9718:
--
Apologies for the spam. Some constancy issue with Jira, made me to add the same
comment again and again.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name` string,
`n_regionkey` int,
`n_comment` string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS INPUTFORMAT
[jira] [Commented] (HIVE-8807) Obsolete default values in webhcat-default.xml
[ https://issues.apache.org/jira/browse/HIVE-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327441#comment-14327441 ] Thejas M Nair commented on HIVE-8807: - We can add instructions to update this as part of release instructions - https://cwiki.apache.org/confluence/display/Hive/HowToRelease Obsolete default values in webhcat-default.xml -- Key: HIVE-8807 URL: https://issues.apache.org/jira/browse/HIVE-8807 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.12.0, 0.13.0, 0.14.0 Reporter: Lefty Leverenz Assignee: Eugene Koifman Fix For: 1.0.0 Attachments: HIVE8807.patch The defaults for templeton.pig.path templeton.hive.path are 0.11 in webhcat-default.xml but they ought to match current release numbers. The Pig version is 0.12.0 for Hive 0.14 RC0 (as shown in pom.xml). no precommit tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9537) string expressions on a fixed length character do not preserve trailing spaces
[
https://issues.apache.org/jira/browse/HIVE-9537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327408#comment-14327408
]
N Campbell commented on HIVE-9537:
--
I wouldn't suggest that use MySQL and Postgres alone is ideal to cross check
SQL semantics.
Should the Apache implementation for CHAR(n) stay this way going forward I
recommend you get the documentation improved.
Having seen companies migrate applications across vendors where this sort of
thing burns them. Many may not know what ISO-SQL 20xx states (let alone care)
and will presume that their source vendor(s) have followed the specification
etc.
Given the server side string operations etc, they may as well stay with VARCHAR
and know that trailing spaces are not preserved vs the 'appears' to have spaces
when projected but not in other cases.
See discussion:
http://www.postgresql.org/docs/9.1/static/datatype-character.html
See discussion of MySQL JDBC:
https://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html
re padCharsWithSpace
If you apply a fold (upper/lower) the type of the result is supposed to be
from the character value expression
Length should be the number of characters in the string value expression -
where number of characters is based upon the semantics of the character set of
the string value expression
If both operands are fixed length character strings, concatenation result is a
fixed length character string with a length equal to the sum of the lengths of
the operands where the length cannot exceed the maximum allowed for a fixed
length character string.
Cheers.
string expressions on a fixed length character do not preserve trailing spaces
--
Key: HIVE-9537
URL: https://issues.apache.org/jira/browse/HIVE-9537
Project: Hive
Issue Type: Bug
Components: SQL
Reporter: N Campbell
Assignee: Aihua Xu
When a string expression such as upper or lower is applied to a fixed length
column the trailing spaces of the fixed length character are not preserved.
{code:sql}
CREATE TABLE if not exists TCHAR (
RNUM int,
CCHAR char(32)
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
{code}
{{cchar}} as a {{char(32)}}.
{code:sql}
select cchar, concat(cchar, cchar), concat(lower(cchar), cchar),
concat(upper(cchar), cchar)
from tchar;
{code}
0|\N
1|
2|
3|BB
4|EE
5|FF
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9388) HiveServer2 fails to reconnect to MetaStore after MetaStore restart
[
https://issues.apache.org/jira/browse/HIVE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mariusz Strzelecki updated HIVE-9388:
-
Attachment: HIVE-9388.2.patch
Patch prepared for current trunk.
HiveServer2 fails to reconnect to MetaStore after MetaStore restart
---
Key: HIVE-9388
URL: https://issues.apache.org/jira/browse/HIVE-9388
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.14.0, 0.13.1
Reporter: Piotr Ackermann
Attachments: HIVE-9388.2.patch, HIVE-9388.patch
How to reproduce:
# Use Hue to connect to HiveServer2
# Restart Metastore
# Try to execute any query in Hue
HiveServer2 report error:
{quote}
ERROR hive.log: Got exception:
org.apache.thrift.transport.TTransportException null
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1681)
at com.sun.proxy.$Proxy10.getDatabases(Unknown Source)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62)
at
org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy19.getSchemas(Unknown Source)
at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:277)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:436)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at
[jira] [Created] (HIVE-9725) Need to add indices, privileges, and functions to HBaseImport [hbase-metastore branch]
Alan Gates created HIVE-9725: Summary: Need to add indices, privileges, and functions to HBaseImport [hbase-metastore branch] Key: HIVE-9725 URL: https://issues.apache.org/jira/browse/HIVE-9725 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates HBaseImport doesn't include these objects because they weren't supported in the metastore yet when it was created. These need to be added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Fix version for hbase-metastore branch
I've been marking JIRAs on this branch as fixed in 1.2, since that's the next version. But that seems wrong as I doubt this code will be in by 1.2. What's the usual practice here? It seems it would make sense to make a label for this branch and mark them as fixed with that label and then when we actually release this in a version we can update all the JIRAs with that label. Alan.
[jira] [Resolved] (HIVE-9581) CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch]
[ https://issues.apache.org/jira/browse/HIVE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-9581. Resolution: Fixed Fix Version/s: (was: 1.2.0) cbo-branch Committed to branch. Thanks, Jesus! CBO (Calcite Return Path): Translate Join to Hive Op [CBO branch] - Key: HIVE-9581 URL: https://issues.apache.org/jira/browse/HIVE-9581 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-9581.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9606) Need a tool to export metadata from RDBMS based metastore into HBase
[ https://issues.apache.org/jira/browse/HIVE-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9606: - Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) Patch 2 checked in, with added comments suggested by Thejas. Need a tool to export metadata from RDBMS based metastore into HBase Key: HIVE-9606 URL: https://issues.apache.org/jira/browse/HIVE-9606 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Fix For: 1.2.0 Attachments: HIVE-9606.2.patch, HIVE-9606.patch For testing (and eventually for end user use) we need a tool that can take data from an existing RDBMS based metastore and create the corresponding objects in an HBase based metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9726) Upgrade to spark 1.3
Brock Noland created HIVE-9726: -- Summary: Upgrade to spark 1.3 Key: HIVE-9726 URL: https://issues.apache.org/jira/browse/HIVE-9726 Project: Hive Issue Type: Sub-task Reporter: Brock Noland -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9726) Upgrade to spark 1.3 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9726: --- Affects Version/s: spark-branch Summary: Upgrade to spark 1.3 [Spark Branch] (was: Upgrade to spark 1.3) Upgrade to spark 1.3 [Spark Branch] --- Key: HIVE-9726 URL: https://issues.apache.org/jira/browse/HIVE-9726 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9671) Support Impersonation [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9671: --- Attachment: HIVE-9671.1-spark.patch Support Impersonation [Spark Branch] Key: HIVE-9671 URL: https://issues.apache.org/jira/browse/HIVE-9671 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Attachments: HIVE-9671.1-spark.patch SPARK-5493 in 1.3 implemented proxy user authentication. We need to implement using this option in spark client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9671) Support Impersonation [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9671: --- Assignee: Brock Noland Status: Patch Available (was: Open) Support Impersonation [Spark Branch] Key: HIVE-9671 URL: https://issues.apache.org/jira/browse/HIVE-9671 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-9671.1-spark.patch SPARK-5493 in 1.3 implemented proxy user authentication. We need to implement using this option in spark client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9727) GroupingID translation from Calcite
Jesus Camacho Rodriguez created HIVE-9727: - Summary: GroupingID translation from Calcite Key: HIVE-9727 URL: https://issues.apache.org/jira/browse/HIVE-9727 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez The translation from Calcite back to Hive might produce wrong results while interacting with other Calcite optimization rules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9727) GroupingID translation from Calcite
[ https://issues.apache.org/jira/browse/HIVE-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9727: -- Attachment: HIVE-9727.patch [~jpullokkaran] GroupingID translation from Calcite --- Key: HIVE-9727 URL: https://issues.apache.org/jira/browse/HIVE-9727 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-9727.patch The translation from Calcite back to Hive might produce wrong results while interacting with other Calcite optimization rules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9727) GroupingID translation from Calcite
[ https://issues.apache.org/jira/browse/HIVE-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9727 started by Jesus Camacho Rodriguez. - GroupingID translation from Calcite --- Key: HIVE-9727 URL: https://issues.apache.org/jira/browse/HIVE-9727 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez The translation from Calcite back to Hive might produce wrong results while interacting with other Calcite optimization rules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9728) LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support)
Sergey Shelukhin created HIVE-9728: -- Summary: LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support) Key: HIVE-9728 URL: https://issues.apache.org/jira/browse/HIVE-9728 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9727) GroupingID translation from Calcite
[ https://issues.apache.org/jira/browse/HIVE-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9727: -- Status: Patch Available (was: In Progress) GroupingID translation from Calcite --- Key: HIVE-9727 URL: https://issues.apache.org/jira/browse/HIVE-9727 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-9727.patch The translation from Calcite back to Hive might produce wrong results while interacting with other Calcite optimization rules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31209: HIVE-9727
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31209/ --- Review request for hive and John Pullokkaran. Bugs: HIVE-9727 https://issues.apache.org/jira/browse/HIVE-9727 Repository: hive-git Description --- GroupingID translation from Calcite Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveGroupingID.java 345b64af8514466c84e9899e9c019b679b761ba6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java ea5918110fa1255f105c646c08e7d307afb3f94b Diff: https://reviews.apache.org/r/31209/diff/ Testing --- Existing tests (groupby*.q) Thanks, Jesús Camacho Rodríguez
[jira] [Created] (HIVE-9729) LLAP: design and implement proper metadata cache
Sergey Shelukhin created HIVE-9729: -- Summary: LLAP: design and implement proper metadata cache Key: HIVE-9729 URL: https://issues.apache.org/jira/browse/HIVE-9729 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Simple approach: add external priorities to data cache, read metadata parts of orc file into it. Advantage: simple; consistent management (no need to coordinate sizes and eviction between data and metadata caches, etc); disadvantage - have to decode every time. Maybe add decoded metadata cache on top - fixed size, small and opportunistic? Or some other approach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9730) LLAP: make sure logging is never called when not needed
Sergey Shelukhin created HIVE-9730: -- Summary: LLAP: make sure logging is never called when not needed Key: HIVE-9730 URL: https://issues.apache.org/jira/browse/HIVE-9730 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin log4j logging has really inefficient serialization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9731) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified
Hari Sankar Sivarama Subramaniyan created HIVE-9731: --- Summary: WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified Key: HIVE-9731 URL: https://issues.apache.org/jira/browse/HIVE-9731 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Hadoop Streaming allows -inputreader parameter to specify use of StreamXmlRecorderReader (example) hadoop jar hadoop-streaming-2.5.1.jar \ -inputreader StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING \ (rest of the command) WebHCat's StreamingDelegator does not include -inputreader as a valid option when submitting jobs to http://www.myserver.com/templeton/v1/mapreduce/streaming endpoint. If -inputreader is specified and passed to templeton server (perhaps via CURL operation) , it will get truncated and not passed as parameter from TempletonControllerJob to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9731) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified
[ https://issues.apache.org/jira/browse/HIVE-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-9731: Status: Patch Available (was: Open) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified Key: HIVE-9731 URL: https://issues.apache.org/jira/browse/HIVE-9731 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-9731.1.patch Hadoop Streaming allows -inputreader parameter to specify use of StreamXmlRecorderReader (example) hadoop jar hadoop-streaming-2.5.1.jar \ -inputreader StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING \ (rest of the command) WebHCat's StreamingDelegator does not include -inputreader as a valid option when submitting jobs to http://www.myserver.com/templeton/v1/mapreduce/streaming endpoint. If -inputreader is specified and passed to templeton server (perhaps via CURL operation) , it will get truncated and not passed as parameter from TempletonControllerJob to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9731) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified
[ https://issues.apache.org/jira/browse/HIVE-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-9731: Attachment: HIVE-9731.1.patch [~ekoifman] / [~thejas] Can you please review the change. Thanks Hari WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified Key: HIVE-9731 URL: https://issues.apache.org/jira/browse/HIVE-9731 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-9731.1.patch Hadoop Streaming allows -inputreader parameter to specify use of StreamXmlRecorderReader (example) hadoop jar hadoop-streaming-2.5.1.jar \ -inputreader StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING \ (rest of the command) WebHCat's StreamingDelegator does not include -inputreader as a valid option when submitting jobs to http://www.myserver.com/templeton/v1/mapreduce/streaming endpoint. If -inputreader is specified and passed to templeton server (perhaps via CURL operation) , it will get truncated and not passed as parameter from TempletonControllerJob to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: setting up a branch for testing
Can you elaborate on how many machines are needed, minimum (is it 1+, just determined by the throughput of QA runs that we want; or is there some fixed requirement too), and what is the setup/process to make them work with HiveQA (in general so we'd know how we can provide machines)? Thanks! From: Szehon sze...@cloudera.com Sent: Friday, January 23, 2015 7:38 PM To: dev@hive.apache.org Subject: Re: setting up a branch for testing Yea but for precommit testing it would need a cluster setup that runs an instance of Ptest server. We only have spark branch setup for that other than trunk, we have one cluster running spark and another running trunk. Setup is doable (actually just need to setup master) but takes some steps, and physical machines. Thanks Szehon On Jan 23, 2015, at 6:04 PM, Sergey Shelukhin ser...@hortonworks.com wrote: Hi. Hive dev doc mentions that patches can be tested by HiveQA against the branch by supplying the branch name in the patch name. However, as far as I understand this requires some setup for each specific branch. Is it possible to set up llap branch for HiveQA testing? -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-9653) LLAP: create a reasonable q file test for ORC IO
[ https://issues.apache.org/jira/browse/HIVE-9653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328222#comment-14328222 ] Sergey Shelukhin commented on HIVE-9653: I added some test. We need to make it more reasonable :) LLAP: create a reasonable q file test for ORC IO Key: HIVE-9653 URL: https://issues.apache.org/jira/browse/HIVE-9653 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin multiple RGs, RG filters, variety of queries -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9730) LLAP: make sure logging is never called when not needed
[ https://issues.apache.org/jira/browse/HIVE-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-9730: -- Attachment: log4j-llap.png LLAP: make sure logging is never called when not needed --- Key: HIVE-9730 URL: https://issues.apache.org/jira/browse/HIVE-9730 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: log4j-llap.png log4j logging has really inefficient serialization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9730) LLAP: make sure logging is never called when not needed
[ https://issues.apache.org/jira/browse/HIVE-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-9730: -- Description: log4j logging has really inefficient serialization !log4j-llap.png! was:log4j logging has really inefficient serialization LLAP: make sure logging is never called when not needed --- Key: HIVE-9730 URL: https://issues.apache.org/jira/browse/HIVE-9730 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: log4j-llap.png log4j logging has really inefficient serialization !log4j-llap.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9086) Add language support to PURGE data while dropping partitions.
[
https://issues.apache.org/jira/browse/HIVE-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328533#comment-14328533
]
Hive QA commented on HIVE-9086:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686790/HIVE-9086.1.patch
{color:green}SUCCESS:{color} +1 7566 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2828/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2828/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2828/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686790 - PreCommit-HIVE-TRUNK-Build
Add language support to PURGE data while dropping partitions.
-
Key: HIVE-9086
URL: https://issues.apache.org/jira/browse/HIVE-9086
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.15.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-9086.1.patch
HIVE-9083 adds metastore-support to skip-trash while dropping partitions.
This patch includes language support to do the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9556) create UDF to calculate the Levenshtein distance between two strings
[
https://issues.apache.org/jira/browse/HIVE-9556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-9556:
-
Labels: (was: TODOC1.2)
create UDF to calculate the Levenshtein distance between two strings
Key: HIVE-9556
URL: https://issues.apache.org/jira/browse/HIVE-9556
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Fix For: 1.2.0
Attachments: HIVE-9556.1.patch, HIVE-9556.2.patch, HIVE-9556.3.patch
Levenshtein distance is a string metric for measuring the difference between
two sequences. Informally, the Levenshtein distance between two words is the
minimum number of single-character edits (i.e. insertions, deletions or
substitutions) required to change one word into the other. It is named after
Vladimir Levenshtein, who considered this distance in 1965.
Example:
The Levenshtein distance between kitten and sitting is 3
1. kitten → sitten (substitution of s for k)
2. sitten → sittin (substitution of i for e)
3. sittin → sitting (insertion of g at the end).
{code}
select levenshtein('kitten', 'sitting');
3
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9556) create UDF to calculate the Levenshtein distance between two strings
[
https://issues.apache.org/jira/browse/HIVE-9556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328585#comment-14328585
]
Lefty Leverenz commented on HIVE-9556:
--
Thanks for the doc, [~apivovarov]. I removed the TODOC1.2 label.
create UDF to calculate the Levenshtein distance between two strings
Key: HIVE-9556
URL: https://issues.apache.org/jira/browse/HIVE-9556
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Fix For: 1.2.0
Attachments: HIVE-9556.1.patch, HIVE-9556.2.patch, HIVE-9556.3.patch
Levenshtein distance is a string metric for measuring the difference between
two sequences. Informally, the Levenshtein distance between two words is the
minimum number of single-character edits (i.e. insertions, deletions or
substitutions) required to change one word into the other. It is named after
Vladimir Levenshtein, who considered this distance in 1965.
Example:
The Levenshtein distance between kitten and sitting is 3
1. kitten → sitten (substitution of s for k)
2. sitten → sittin (substitution of i for e)
3. sittin → sitting (insertion of g at the end).
{code}
select levenshtein('kitten', 'sitting');
3
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9277) Hybrid Hybrid Grace Hash Join
[ https://issues.apache.org/jira/browse/HIVE-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-9277: Attachment: HIVE-9277.01.patch Uploading 1st patch for testing Hybrid Hybrid Grace Hash Join - Key: HIVE-9277 URL: https://issues.apache.org/jira/browse/HIVE-9277 Project: Hive Issue Type: New Feature Components: Physical Optimizer Reporter: Wei Zheng Assignee: Wei Zheng Labels: join Attachments: HIVE-9277.01.patch, High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf We are proposing an enhanced hash join algorithm called “hybrid hybrid grace hash join”. We can benefit from this feature as illustrated below: o The query will not fail even if the estimated memory requirement is slightly wrong o Expensive garbage collection overhead can be avoided when hash table grows o Join execution using a Map join operator even though the small table doesn't fit in memory as spilling some data from the build and probe sides will still be cheaper than having to shuffle the large fact table The design was based on Hadoop’s parallel processing capability and significant amount of memory available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9277) Hybrid Hybrid Grace Hash Join
[ https://issues.apache.org/jira/browse/HIVE-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-9277: Status: Patch Available (was: Open) Hybrid Hybrid Grace Hash Join - Key: HIVE-9277 URL: https://issues.apache.org/jira/browse/HIVE-9277 Project: Hive Issue Type: New Feature Components: Physical Optimizer Reporter: Wei Zheng Assignee: Wei Zheng Labels: join Attachments: HIVE-9277.01.patch, High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf We are proposing an enhanced hash join algorithm called “hybrid hybrid grace hash join”. We can benefit from this feature as illustrated below: o The query will not fail even if the estimated memory requirement is slightly wrong o Expensive garbage collection overhead can be avoided when hash table grows o Join execution using a Map join operator even though the small table doesn't fit in memory as spilling some data from the build and probe sides will still be cheaper than having to shuffle the large fact table The design was based on Hadoop’s parallel processing capability and significant amount of memory available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9738) create SOUNDEX udf
Alexander Pivovarov created HIVE-9738: - Summary: create SOUNDEX udf Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9728) LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support)
[ https://issues.apache.org/jira/browse/HIVE-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328618#comment-14328618 ] Lefty Leverenz commented on HIVE-9728: -- Doc note: This adds *hive.llap.io.cache.direct* to HiveConf.java in the LLAP branch, so it will need to be documented when the branch gets merged to trunk. LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support) - Key: HIVE-9728 URL: https://issues.apache.org/jira/browse/HIVE-9728 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9726) Upgrade to spark 1.3 [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328536#comment-14328536
]
Hive QA commented on HIVE-9726:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12699788/HIVE-9671.1-spark.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7510 tests executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/739/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/739/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12699788 - PreCommit-HIVE-SPARK-Build
Upgrade to spark 1.3 [Spark Branch]
---
Key: HIVE-9726
URL: https://issues.apache.org/jira/browse/HIVE-9726
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-9671.1-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9277) Hybrid Hybrid Grace Hash Join
[ https://issues.apache.org/jira/browse/HIVE-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-9277: Attachment: HIVE-9277.02.patch Uploading 2nd patch for testing Hybrid Hybrid Grace Hash Join - Key: HIVE-9277 URL: https://issues.apache.org/jira/browse/HIVE-9277 Project: Hive Issue Type: New Feature Components: Physical Optimizer Reporter: Wei Zheng Assignee: Wei Zheng Labels: join Attachments: HIVE-9277.01.patch, HIVE-9277.02.patch, High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf We are proposing an enhanced hash join algorithm called “hybrid hybrid grace hash join”. We can benefit from this feature as illustrated below: o The query will not fail even if the estimated memory requirement is slightly wrong o Expensive garbage collection overhead can be avoided when hash table grows o Join execution using a Map join operator even though the small table doesn't fit in memory as spilling some data from the build and probe sides will still be cheaper than having to shuffle the large fact table The design was based on Hadoop’s parallel processing capability and significant amount of memory available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Fix version for hbase-metastore branch
Also, what should we use for a documentation label? (HIVE-9606 https://issues.apache.org/jira/browse/HIVE-9606 needs one.) TODOC labels are proliferating for all the releases and branches, but I don't think a generic TODOC label would be helpful. So what would be a good abbreviation for the hbase-metastore branch? Maybe TODOC-HBMETA? -- Lefty On Thu, Feb 19, 2015 at 7:12 PM, Alan Gates alanfga...@gmail.com wrote: Could someone with admin permissions on our JIRA add an hbase-metastore-branch label? I'll take care of changing all the fix versions for the few JIRA's we've already committed. Thanks. Alan. Ashutosh Chauhan hashut...@apache.org February 19, 2015 at 11:22 This is what we have been doing for cbo work. e.g. https://issues.apache.org/jira/browse/HIVE-9581 Thejas Nair thejas.n...@gmail.com February 19, 2015 at 11:17 I agree, using a label for fix version makes sense in this case. I believe that is what had been done for hive-on-spark and hive-on-tez. Alan Gates alanfga...@gmail.com February 19, 2015 at 10:56 I've been marking JIRAs on this branch as fixed in 1.2, since that's the next version. But that seems wrong as I doubt this code will be in by 1.2. What's the usual practice here? It seems it would make sense to make a label for this branch and mark them as fixed with that label and then when we actually release this in a version we can update all the JIRAs with that label. Alan.
Review Request 31222: HIVE-9738 create SOUNDEX udf
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31222/ --- Review request for hive, Alan Gates and Jason Dere. Bugs: HIVE-9738 https://issues.apache.org/jira/browse/HIVE-9738 Repository: hive-git Description --- HIVE-9738 create SOUNDEX udf Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java bfeb33c9edf79071bf106a3780e1c6bc21be3753 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSoundex.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSoundex.java PRE-CREATION ql/src/test/queries/clientpositive/udf_soundex.q PRE-CREATION ql/src/test/results/clientpositive/show_functions.q.out d4b06507f2356cc58ff7312ff78261406f3ceb4f ql/src/test/results/clientpositive/udf_soundex.q.out PRE-CREATION Diff: https://reviews.apache.org/r/31222/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Updated] (HIVE-9738) create SOUNDEX udf
[ https://issues.apache.org/jira/browse/HIVE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-9738: -- Attachment: HIVE-9738.1.patch patch #1 create SOUNDEX udf -- Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Attachments: HIVE-9738.1.patch Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 Implementation: http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Soundex.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9738) create SOUNDEX udf
[ https://issues.apache.org/jira/browse/HIVE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9738 started by Alexander Pivovarov. - create SOUNDEX udf -- Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Attachments: HIVE-9738.1.patch Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 Implementation: http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Soundex.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9738) create SOUNDEX udf
[ https://issues.apache.org/jira/browse/HIVE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-9738: -- Status: Patch Available (was: In Progress) create SOUNDEX udf -- Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Attachments: HIVE-9738.1.patch Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 Implementation: http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Soundex.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9728) LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support)
[ https://issues.apache.org/jira/browse/HIVE-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-9728. Resolution: Fixed LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support) - Key: HIVE-9728 URL: https://issues.apache.org/jira/browse/HIVE-9728 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9737) Issue come while creating the table in hbase using java Impla API
Mohit Sharma created HIVE-9737:
--
Summary: Issue come while creating the table in hbase using java
Impla API
Key: HIVE-9737
URL: https://issues.apache.org/jira/browse/HIVE-9737
Project: Hive
Issue Type: Bug
Components: Database/Schema
Environment: Linux OS, Using Java Impala API with thrift, Cloudera
Hbase database
Reporter: Mohit Sharma
I am trying to create hbase table using this query
CREATE TABLE foo4(rowkey STRING, a STRING, b STRING) STORED BY
'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES
('hbase.columns.mapping' = ':key,f:c1,f:c2') TBLPROPERTIES ('hbase.table.name'
= 'bar4');
with the help of java Impala api and I am sharing you code link
https://github.com/pauldeschacht/impala-java-client
When I am trying to create table in habse I am facing this issue
AnalysisException: Syntax error in line 2:
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
^
Encountered: BY
Expected: AS
CAUSED BY: Exception: Syntax error,HY000,0,false
Please help me what I do?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9086) Add language support to PURGE data while dropping partitions.
[ https://issues.apache.org/jira/browse/HIVE-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328606#comment-14328606 ] Lefty Leverenz commented on HIVE-9086: -- Doc note: Uh oh, I documented this prematurely (for HIVE-7100). But did I get it right, except for the jira attibution and release number? * [DDL -- Drop Partitions | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropPartitions] Add language support to PURGE data while dropping partitions. - Key: HIVE-9086 URL: https://issues.apache.org/jira/browse/HIVE-9086 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.15.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-9086.1.patch HIVE-9083 adds metastore-support to skip-trash while dropping partitions. This patch includes language support to do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9738) create SOUNDEX udf
[ https://issues.apache.org/jira/browse/HIVE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-9738: -- Description: Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 Implementation: http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Soundex.html was: Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 create SOUNDEX udf -- Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 Implementation: http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Soundex.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9738) create SOUNDEX udf
[ https://issues.apache.org/jira/browse/HIVE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-9738: -- Description: Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 create SOUNDEX udf -- Key: HIVE-9738 URL: https://issues.apache.org/jira/browse/HIVE-9738 Project: Hive Issue Type: Improvement Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. The American Soundex System The soundex code consist of the first letter of the name followed by three digits. These three digits are determined by dropping the letters a, e, i, o, u, h, w and y and adding three digits from the remaining letters of the name according to the table below. There are only two additional rules. (1) If two or more consecutive letters have the same code, they are coded as one letter. (2) If there are an insufficient numbers of letters to make the three digits, the remaining digits are set to zero. Soundex Table 1 b,f,p,v 2 c,g,j,k,q,s,x,z 3 d, t 4 l 5 m, n 6 r Examples: Miller M460 Peterson P362 Peters P362 Auerbach A612 Uhrbach U612 Moskowitz M232 Moskovitz M213 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Status: Open (was: Patch Available) Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch, HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Attachment: HIVE-6617.17.patch address [~jpullokkaran]'s comments, (1) add back keywords under a special category; (2) add negative/positive tests for these keywords Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch, HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Status: Patch Available (was: Open) Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch, HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9719) Up calcite version on cbo branch
[ https://issues.apache.org/jira/browse/HIVE-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-9719. Resolution: Fixed Fix Version/s: cbo-branch Committed to cbo branch. Up calcite version on cbo branch Key: HIVE-9719 URL: https://issues.apache.org/jira/browse/HIVE-9719 Project: Hive Issue Type: Task Components: CBO Affects Versions: cbo-branch Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: cbo-branch Attachments: HIVE-9719.cbo.patch CALCITE-594 is now checked in calcite master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9734) Correlating expression cannot contain unqualified column references
N Campbell created HIVE-9734: Summary: Correlating expression cannot contain unqualified column references Key: HIVE-9734 URL: https://issues.apache.org/jira/browse/HIVE-9734 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.14.0 Reporter: N Campbell Priority: Minor The following valid ISO-SQL constructs will not parse Unsupported SubQuery Expression 'c1': Correlating expression cannot contain unqualified column references The statement will parse if the predicate is changed to where tjoin1.c1 in ( ) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select c1 from tjoin1) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select tjoin1.c1 from tjoin1) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select x.c1 from tjoin1 x) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9733) Fix test failures on cbo branch
[ https://issues.apache.org/jira/browse/HIVE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9733: --- Resolution: Fixed Fix Version/s: cbo-branch Status: Resolved (was: Patch Available) Committed to branch. Fix test failures on cbo branch --- Key: HIVE-9733 URL: https://issues.apache.org/jira/browse/HIVE-9733 Project: Hive Issue Type: Bug Components: CBO, Query Planning Affects Versions: cbo-branch Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: cbo-branch Attachments: HIVE-9733.cbo.patch A constant changed their home class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31072: HIVE-9678 create timediff UDF
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31072/ --- (Updated Feb. 20, 2015, 1:04 a.m.) Review request for hive, Jason Dere and Thejas Nair. Changes --- remove timezone support and add nanos precision support Bugs: HIVE-9678 https://issues.apache.org/jira/browse/HIVE-9678 Repository: hive-git Description --- HIVE-9678 create timediff UDF Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 909f6ae6ec2aa24942286f15ffa7293378ca53fd ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimeDiff.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFTimeDiff.java PRE-CREATION ql/src/test/queries/clientpositive/udf_timediff.q PRE-CREATION ql/src/test/results/clientpositive/show_functions.q.out a58f453ddcca8a99319b8b465af0db3cc8f7302f ql/src/test/results/clientpositive/udf_timediff.q.out PRE-CREATION Diff: https://reviews.apache.org/r/31072/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Commented] (HIVE-9678) create timediff UDF
[
https://issues.apache.org/jira/browse/HIVE-9678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328409#comment-14328409
]
Alexander Pivovarov commented on HIVE-9678:
---
patch #3
create timediff UDF
---
Key: HIVE-9678
URL: https://issues.apache.org/jira/browse/HIVE-9678
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9678.1.patch, HIVE-9678.2.patch, HIVE-9678.3.patch
MySQL has very useful function timediff. We should have it in Hive
{code}
select timediff('2015-02-12 05:09:07', '2015-02-12 01:18:20');
OK
03:50:47
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-9674:
---
Attachment: (was: HIVE-9674.1.patch)
*DropPartitionEvent should handle partition-sets.
-
Key: HIVE-9674
URL: https://issues.apache.org/jira/browse/HIVE-9674
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Dropping a set of N partitions from a table currently results in N
DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
is wasteful, especially so for large N. It also makes it impossible to even
try to run authorization-checks on all partitions in a batch.
Taking the cue from HIVE-9609, we should compose an {{IterablePartition}}
in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9678) create timediff UDF
[
https://issues.apache.org/jira/browse/HIVE-9678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328480#comment-14328480
]
Hive QA commented on HIVE-9678:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12699787/HIVE-9678.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7566 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2826/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2826/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2826/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12699787 - PreCommit-HIVE-TRUNK-Build
create timediff UDF
---
Key: HIVE-9678
URL: https://issues.apache.org/jira/browse/HIVE-9678
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9678.1.patch, HIVE-9678.2.patch, HIVE-9678.3.patch
MySQL has very useful function timediff. We should have it in Hive
{code}
select timediff('2015-02-12 05:09:07', '2015-02-12 01:18:20');
OK
03:50:47
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9718) Insert into dynamic partitions with same column structure in the distibute by clause barfs
[
https://issues.apache.org/jira/browse/HIVE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327713#comment-14327713
]
Ashutosh Chauhan commented on HIVE-9718:
You need to name your patch file per
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing
for automated QA to run.
Insert into dynamic partitions with same column structure in the distibute
by clause barfs
Key: HIVE-9718
URL: https://issues.apache.org/jira/browse/HIVE-9718
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0, 1.0.0
Reporter: Pavan Srinivas
Priority: Critical
Attachments: nation.tbl, patch.txt
Sample reproducible query:
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
insert overwrite table nation_new_p partition (some)
select n_name as name1, n_name as name2, n_name as name3 from nation
distribute by name3;
{code}
Note: Make sure there is data in the source table to reproduce the issue.
During the optimizations done for Jira:
https://issues.apache.org/jira/browse/HIVE-4867, an optimization of
deduplication of columns is done. But, when one of the columns is used as
part of partitioned/distribute by, its not taken care of.
The above query produces exception as follows:
{code}
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{n_nationkey:0,n_name:ALGERIA,n_regionkey:0,n_comment: haggle.
carefully final deposits detect slyly agai}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 13 more
Caused by: java.lang.RuntimeException: cannot find field _col2 from [0:_col0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:410)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:325)
... 19 more
{code}
Tables used are:
{code}
CREATE EXTERNAL TABLE `nation`(
`n_nationkey` int,
`n_name` string,
`n_regionkey` int,
`n_comment` string)
ROW FORMAT DELIMITED
FIELDS
