[jira] [Created] (HIVE-23780) Fail dropTable if acid cleanup fails

2020-06-29 Thread Mustafa Iman (Jira)
Mustafa Iman created HIVE-23780:
---

 Summary: Fail dropTable if acid cleanup fails
 Key: HIVE-23780
 URL: https://issues.apache.org/jira/browse/HIVE-23780
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Standalone Metastore, Transactions
Reporter: Mustafa Iman
Assignee: Mustafa Iman


Acid cleanup happens after dropTable is committed. If cleanup fails for some 
reason, there are leftover entries in acid tables. This later causes dropped 
table's name to be unusable by new tables.

[~pvary] [~ngangam]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23779) BasicStatsTask Info is not getting printed in beeline console

2020-06-29 Thread Naresh P R (Jira)
Naresh P R created HIVE-23779:
-

 Summary: BasicStatsTask Info is not getting printed in beeline 
console
 Key: HIVE-23779
 URL: https://issues.apache.org/jira/browse/HIVE-23779
 Project: Hive
  Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R


After HIVE-16061, partition basic stats are not getting printed in beeline 
console.
{code:java}
INFO : Partition {dt=2020-06-29} stats: [numFiles=21, numRows=22, 
totalSize=14607, rawDataSize=0]{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23778) Unable to start Hive - java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProvider

2020-06-29 Thread sathyanarayanan (Jira)
sathyanarayanan created HIVE-23778:
--

 Summary: Unable to start Hive - java.lang.RuntimeException: 
java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProvider
 Key: HIVE-23778
 URL: https://issues.apache.org/jira/browse/HIVE-23778
 Project: Hive
  Issue Type: Test
  Components: CLI
Affects Versions: 2.3.7
 Environment: Hadoop 2.3.0-cdh5.0.0
Hive 2.3.7
VM - Oracle Linux 7

Bash profile 
-
export JAVA_HOME=/usr/local/java/jdk1.8.0_251/
export PATH=$PATH:$JAVA_HOME/bin 
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.3.0-cdh5.0.0
export PATH=$PATH:$HADOOP_HOME/bin

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.3.0-cdh5.0.0
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 

export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin 
export HADOOP_INSTALL=$HADOOP_HOME 

export HIVE_HOME=/usr/local/hive/apache-hive-2.3.7-bin
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/hive/apache-hive-2.3.7-bin/lib/
export 
HADOOP_CLASSPATH=/usr/local/hive/apache-hive-2.3.7-bin/lib/hive-common-2.3.7.jar
export 
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hive/apache-hive-2.3.7-bin/lib/*.jar
export 
HIVE_CLASSPATH=/usr/local/hive/apache-hive-2.3.7-bin/lib/:/usr/local/hadoop/hadoop-2.3.0-cdh5.0.0/share/hadoop/mapreduce1/lib/*.jar

export DERBY_HOME=/usr/local/derby/db-derby-10.4.2.0-bin

export PATH=$PATH:$DERBY_HOME/bin

export 
CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar
[root@hadhiv bin]# 

Reporter: sathyanarayanan


My Hadoop version is Hadoop 2.3.0-cdh5.0.0 and my Hive 2.3.7, while executive 
Hive, i'm getting the following error message. I could sense that there is some 
JAR file missing in the classpath. But could not get the relevant classpath.

 

Logging initialized using configuration in 
jar:file:/usr/local/hive/apache-hive-2.3.7-bin/lib/hive-common-2.3.7.jar!/hive-log4j2.properties
 Async: true

Exception in thread "main" java.lang.RuntimeException: 
java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProvider

 at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:91)

 at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3323)

 at 
org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:709)

 at 
org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:654)

 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)

 at 
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:553)

 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:750)

 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.lang.reflect.Method.invoke(Method.java:498)

 at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.NoClassDefFoundError: 
org/apache/hadoop/crypto/key/KeyProvider

 at java.lang.Class.forName0(Native Method)

 at java.lang.Class.forName(Class.java:264)

 at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:129)

 at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:124)

 at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:88)

 ... 12 more

Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.crypto.key.KeyProvider

 at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)

 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)

 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

 ... 17 more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23777) hive.semantic.analyzer.hook missing documentation

2020-06-29 Thread liuyan (Jira)
liuyan created HIVE-23777:
-

 Summary: hive.semantic.analyzer.hook  missing documentation 
 Key: HIVE-23777
 URL: https://issues.apache.org/jira/browse/HIVE-23777
 Project: Hive
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 3.1.2, 3.1.1, 3.1.0, 3.0.0
Reporter: liuyan


hive.semantic.analyzer.hook is missing documentation from the 
[https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] Page



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23776) Retire quickstats autocollection

2020-06-29 Thread Zoltan Haindrich (Jira)
Zoltan Haindrich created HIVE-23776:
---

 Summary: Retire quickstats autocollection
 Key: HIVE-23776
 URL: https://issues.apache.org/jira/browse/HIVE-23776
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this is about:
* num files
* datasize (sum of filesizes)
* num erasure coded files

right now these are scanned during every BasicStatsTask execution - which means 
some filesystem reads/etc - for small inserts these are visible in case the fs 
is a bit slower (s3 and friends)

I don't think they are really in use...we rely more on columnstats which are 
more accurate ; and because of the datasize in this case is for "offline" 
(ondisk) - while we should be insted calculate with "online" sizes...

proposal:

* remove collection and storage of this data
* collect it on the fly during "desc formatted" statements to provide them for 
informational purposes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23775) investigate windowing spec when an order by is present

2020-06-29 Thread Zoltan Haindrich (Jira)
Zoltan Haindrich created HIVE-23775:
---

 Summary: investigate windowing spec when an order by is present
 Key: HIVE-23775
 URL: https://issues.apache.org/jira/browse/HIVE-23775
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this is some weird stuff came up during review

https://github.com/apache/hive/pull/1126#discussion_r442266978

Order by spec -> range, unbounded preceding, current row
This also aligns with most RDBMSs implementation





--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Open old PRs

2020-06-29 Thread Zoltan Haindrich

Hey all!

Around friday I've made a small change: and enabled daily repo scanning of apache/hive; it picked up ~80 pending prs for testruns - during the weekeng it was able to finish 
with all of them :)

The periodic repo scanning adds an extra safety net which could help mitigate 
issues in case the PR event was lost for some reason.

cheers,
Zoltan


On 5/30/20 4:15 PM, Zoltan Haindrich wrote:

Hey All,

The new test executor will pick up any PR which doesn't yet have a test result - now that the patch is on the master; every PR which is mergeable with the master branch is 
a good candidate - so the right move would be to clean up our PR backlog.


I would like to ask everyone to look at https://github.com/apache/hive/pulls
and close some PRs which are already submitted or just leftovers from - 
primarily I would ask you to look at PRs opened by yourself...

cheers,
Zoltan


Announcing ApacheCon @Home 2020

2020-06-29 Thread Rich Bowen

Hi, Apache enthusiast!

(You’re receiving this because you’re subscribed to one or more dev or 
user mailing lists for an Apache Software Foundation project.)


The ApacheCon Planners and the Apache Software Foundation are pleased to 
announce that ApacheCon @Home will be held online, September 29th 
through October 1st, 2020. We’ll be featuring content from dozens of our 
projects, as well as content about community, how Apache works, business 
models around Apache software, the legal aspects of open source, and 
many other topics.


Full details about the event, and registration, is available at 
https://apachecon.com/acah2020


Due to the confusion around how and where this event was going to be 
held, and in order to open up to presenters from around the world who 
may previously have been unable or unwilling to travel, we’ve reopened 
the Call For Presentations until July 13th. Submit your talks today at 
https://acna2020.jamhosted.net/


We hope to see you at the event!
Rich Bowen, VP Conferences, The Apache Software Foundation


[jira] [Created] (HIVE-23773) Support multi-key probe MapJoins

2020-06-29 Thread Panagiotis Garefalakis (Jira)
Panagiotis Garefalakis created HIVE-23773:
-

 Summary: Support multi-key probe MapJoins
 Key: HIVE-23773
 URL: https://issues.apache.org/jira/browse/HIVE-23773
 Project: Hive
  Issue Type: Sub-task
Reporter: Panagiotis Garefalakis
Assignee: Panagiotis Garefalakis






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23774) Reduce log level at aggrColStatsForPartitions in MetaStoreDirectSql.java

2020-06-29 Thread Barnabas Maidics (Jira)
Barnabas Maidics created HIVE-23774:
---

 Summary: Reduce log level at aggrColStatsForPartitions in 
MetaStoreDirectSql.java
 Key: HIVE-23774
 URL: https://issues.apache.org/jira/browse/HIVE-23774
 Project: Hive
  Issue Type: Improvement
Reporter: Barnabas Maidics
Assignee: Barnabas Maidics


[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1589]

This log is not needed at INFO log level.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23772) Reallocate calcite-core to prevent NoSuchFiledError

2020-06-29 Thread Ayush Saxena (Jira)
Ayush Saxena created HIVE-23772:
---

 Summary: Reallocate calcite-core to prevent NoSuchFiledError
 Key: HIVE-23772
 URL: https://issues.apache.org/jira/browse/HIVE-23772
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Ayush Saxena
Assignee: Ayush Saxena


Exception trace due to conflict with {{calcite-core}}

{noformat}
Caused by: java.lang.NoSuchFieldError: operands
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter$RexVisitor.visitCall(ASTConverter.java:785)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter$RexVisitor.visitCall(ASTConverter.java:509)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.calcite.rex.RexCall.accept(RexCall.java:191) 
~[calcite-core-1.21.0.jar:1.21.0]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:239)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convertSource(ASTConverter.java:437)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:124)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:112)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1620)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:555)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12456)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:433)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:290)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:184) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:602) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:548) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:542) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]

{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23771) load数据到hive,limit 显示用户名中文正确,where 用户名乱码,并且不能使用用户名比对

2020-06-29 Thread wang (Jira)
wang created HIVE-23771:
---

 Summary: load数据到hive,limit 显示用户名中文正确,where 用户名乱码,并且不能使用用户名比对
 Key: HIVE-23771
 URL: https://issues.apache.org/jira/browse/HIVE-23771
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 2.1.1
Reporter: wang
 Fix For: 2.1.1
 Attachments: image-2020-06-29-15-04-23-999.png, 
image-2020-06-29-15-08-25-923.png, image-2020-06-29-15-10-10-310.png

建表语句:create table smg_t_usr_inf_23(
Usr_ID string,
RlgnSvcPltfrmUsr_TpCd string,
Rlgn_InsID string,
Usr_Nm string ,
) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' 
WITH SERDEPROPERTIES ("field.delim"="|@|") stored as textfile

导入数据:LOAD DATA LOCAL INPATH '/home/ap/USR_INF 20200622_0001.dat' INTO TABLE 
usr_inf

select * from usr_inf limit 10;显示数据: !image-2020-06-29-15-04-23-999.png!

 

select * from usr_inf where usr_nm = '胡学玲' ;无显示数据: 
!image-2020-06-29-15-08-25-923.png!

 

其他select * from usr_inf where usr_id='***';显示数据 
!image-2020-06-29-15-10-10-310.png! .

求大神解答,为什么导入的数据是中文但是where就有问题,直接insert into table aa select * from usr_inf;新表 
的usr_nm 字段也是同上 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)