What is the release plan for Hive 4.0.0

2020-01-21 Thread Chinna Rao Lalam 
Hi all.

Do we have any timelines for Hive 4.0.0 release.

Thanks,
Chinna

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 0

2020-01-21 Thread Zoltan Haindrich 

Hello,

Correct url seems to be:
https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-0/

Initializing the sysdb in case the metastore is backed by mysql/postgres or 
oracle fails with an error.

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: 
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error while trying to get column 
names: ERROR: syntax error at or near "{"

  Position: 72) (state=08S01,code=1)

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: 
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error while trying to get column 
names: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '{LIMIT 1}' at line 1) 
(state=08S01,code=1)


I don't think we should release it without this working...it's most probably just a missed backport - I'll take a look; and I guess I also forgot to put something on 
branch-3 which should be there...


I'm not sure...but I think I've somehow missed that there will be a 3.1.3 :)

cheers,
Zoltan

On 1/21/20 12:03 AM, Alan Gates wrote:

+1.  I checked the signature and hash, did a build, and checked the rat
output.

Alan.

On Wed, Jan 15, 2020 at 2:08 PM Naveen Gangam 
wrote:


Apache Hive 3.1.3 Release Candidate 0 is available here:
https://people.apache.org/~ngangam/hive-3.1.3-rc-0


Maven artifacts are available
here:
https://repository.apache.org/content/repositories/orgapachehive-1099/


The tag release-3.1.3-rc0 has been applied to the source for this
release in github, you can see it
athttps://github.com/apache/hive/tree/release-3.1.3-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Call for presentations for ApacheCon North America 2020 now open

2020-01-21 Thread Rich Bowen 

Dear Apache enthusiast,

(You’re receiving this message because you are subscribed to one or more 
project mailing lists at the Apache Software Foundation.)


The call for presentations for ApacheCon North America 2020 is now open 
at https://apachecon.com/acna2020/cfp


ApacheCon will be held at the Sheraton, New Orleans, September 28th 
through October 2nd, 2020.


As in past years, ApacheCon will feature tracks focusing on the various 
technologies within the Apache ecosystem, and so the call for 
presentations will ask you to select one of those tracks, or “General” 
if the content falls outside of one of our already-organized tracks. 
These tracks are:


Karaf
Internet of Things
Fineract
Community
Content Delivery
Solr/Lucene (Search)
Gobblin/Big Data Integration
Ignite
Observability
Cloudstack
Geospatial
Graph
Camel/Integration
Flagon
Tomcat
Cassandra
Groovy
Web/httpd
General/Other

The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).

Submit early, submit often, at https://apachecon.com/acna2020/cfp

Rich, for the ApacheCon Planners

Re: Review Request 71995: TopN Key optimizer should use array instead of priority queue

2020-01-21 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71995/#review219348
---




ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java
Line 42 (original), 44 (patched)


Add a counter for metrics of this


- Gopal V


On Jan. 14, 2020, 3:38 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71995/
> ---
> 
> (Updated Jan. 14, 2020, 3:38 p.m.)
> 
> 
> Review request for hive, Gopal V, Jesús Camacho Rodríguez, and Krisztian Kasa.
> 
> 
> Bugs: HIVE-22726
> https://issues.apache.org/jira/browse/HIVE-22726
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TopN key optimizer currently uses a priority queue for keeping track of 
> the largest/smallest rows. Its max size is the same as the user specified 
> limit. This should be replaced a more cache line friendly array with a small 
> (128) maximum size and see how much performance is gained.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java 4998766f064 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java b7c12502204 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java 
> 5faa038c18d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java 
> ce6efa49192 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java ff815434f0c 
> 
> 
> Diff: https://reviews.apache.org/r/71995/diff/1/
> 
> 
> Testing
> ---
> 
> with the following query:
> 
> 
> use tpcds_bin_partitioned_orc_100;
> set hive.optimize.topnkey=true;
> set hive.optimize.topnkey.max=5;
> 
> select  i_item_id,
> s_state, grouping(s_state) g_state,
> avg(ss_quantity) agg1,
> avg(ss_list_price) agg2,
> avg(ss_coupon_amt) agg3,
> avg(ss_sales_price) agg4
>  from store_sales, customer_demographics, date_dim, store, item
>  where ss_sold_date_sk = d_date_sk and
>ss_item_sk = i_item_sk and
>ss_store_sk = s_store_sk and
>ss_cdemo_sk = cd_demo_sk
>  group by rollup (i_item_id, s_state)
>  order by i_item_id
>  ,s_state
>  limit 5;
> 
> 
> Results:
>   enabled:   5 rows selected (715.26 seconds)
>   enabled:   5 rows selected (605.888 seconds)
>   disabled:  5 rows selected (1208.168 seconds)
>   disabled:  5 rows selected (1219.482 seconds)
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

[jira] [Created] (HIVE-22757) NullPointerException when executing SQLs

2020-01-21 Thread Deegue (Jira) 
Deegue created HIVE-22757:
-

 Summary: NullPointerException when executing SQLs
 Key: HIVE-22757
 URL: https://issues.apache.org/jira/browse/HIVE-22757
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.6
Reporter: Deegue


When executing SQL:


{code:java}
insert overwrite table ods.ods_1 partition(stat_day='20191209')
select
id
,user_id
,teacher_user_id
,partner_user_id
,order_id
,barcode
,sub_order_id
,item_id
,sales
,refund
,teacher_profit
,partner_profit
,teacher_refund_profit
,partner_refund_profit
,teacher_commission_value
,partner_commission_value
,biz_type
,pay_time
,item_profit_type
,black_mark
,is_deleted
,create_time
,modify_time
from src.src_1
where partition_date='20191209'
union all
select
t1.id
,t1.user_id
,t1.teacher_user_id
,t1.partner_user_id
,t1.order_id
,t1.barcode
,t1.sub_order_id
,t1.item_id
,t1.sales
,t1.refund
,t1.teacher_profit
,t1.partner_profit
,t1.teacher_refund_profit
,t1.partner_refund_profit
,t1.teacher_commission_value
,t1.partner_commission_value
,t1.biz_type
,t1.pay_time
,t1.item_profit_type
,t1.black_mark
,t1.is_deleted
,t1.create_time
,t1.modify_time
from
(select *
from ods.ods_1
where stat_day='20191208'
) t1
left join
( select order_id
,sub_order_id
from src.src_1
where partition_date='20191209'
) t2
on t1.order_id=t2.order_id
and t1.sub_order_id=t2.sub_order_id
where t2.order_id is null
{code}

`java.lang.NullPointerException` thrown because the array list 
`neededNestedColumnPaths` haven't been inited when `addAll` method is invoked.


{code:java}
Launching Job 5 out of 5
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1566481621886_4925755, Tracking URL = 
http://TXIDC65-bigdata-resourcemanager1:8042/proxy/application_1566481621886_4925755/
Kill Command = /usr/local/yunji/hadoop/bin/hadoop job  -kill 
job_1566481621886_4925755
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
2019-12-24 16:00:40,584 Stage-4 map = 0%,  reduce = 0%
2019-12-24 16:01:40,956 Stage-4 map = 0%,  reduce = 0%
2019-12-24 16:02:41,451 Stage-4 map = 0%,  reduce = 0%
2019-12-24 16:02:45,550 Stage-4 map = 100%,  reduce = 0%
Ended Job = job_1566481621886_4925755 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1566481621886_4925755_m_00 (and more) from job 
job_1566481621886_4925755

Task with the most failures(4):
-
Task ID:
  task_1566481621886_4925755_m_00

URL:
  
http://TXIDC65-bigdata-resourcemanager1:8088/taskdetails.jsp?jobid=job_1566481621886_4925755=task_1566481621886_4925755_m_00
-
Diagnostic Messages for this Task:
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:271)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:217)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:345)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:695)
at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:257)
... 11 more
Caused by: java.lang.NullPointerException
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
at 
org.apache.hadoop.hive.ql.io.parquet.ProjectionPusher.pushProjectionsAndFilters(ProjectionPusher.java:118)
at

[jira] [Created] (HIVE-22758) Create database with permission error when doas set to true

2020-01-21 Thread Chiran Ravani (Jira) 
Chiran Ravani created HIVE-22758:


 Summary: Create database with permission error when doas set to 
true
 Key: HIVE-22758
 URL: https://issues.apache.org/jira/browse/HIVE-22758
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.1.0, 3.0.0
Reporter: Chiran Ravani
Assignee: Chiran Ravani


With doAs set to true, running create database on external location fails due 
to permission denied to write on directory specified for hive user (User with 
HMS is running).

Steps to reproduce the issue:
1. Turn on, Hive run as end-user to true.
2. Connect to hive as some user other than admin, eg:- chiran
3. Create a database with external location
{code}
create database externaldbexample location '/user/chiran/externaldbexample'
{code}

The above statement fails with HDFS write permission denied error as below.

{code}
> create database externaldbexample location '/user/chiran/externaldbexample';
INFO  : Compiling 
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d): 
create database externaldbexample location '/user/chiran/externaldbexample'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d); Time 
taken: 1.377 seconds
INFO  : Executing 
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d): 
create database externaldbexample location '/user/chiran/externaldbexample'
INFO  : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. 
MetaException(message:java.lang.reflect.UndeclaredThrowableException)
INFO  : Completed executing 
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d); Time 
taken: 0.238 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.DDLTask. 
MetaException(message:java.lang.reflect.UndeclaredThrowableException) 
(state=08S01,code=1)
{code}

>From Hive Metastore service log, below is seen.

{code}
2020-01-22T04:36:27,870 WARN  [pool-6-thread-6]: metastore.ObjectStore 
(ObjectStore.java:getDatabase(1010)) - Failed to get database 
hive.externaldbexample, returning NoSuchObjectExcept
ion
2020-01-22T04:36:27,898 INFO  [pool-6-thread-6]: metastore.HiveMetaStore 
(HiveMetaStore.java:run(1339)) - Creating database path in managed directory 
hdfs://c470-node2.squadron.support.
hortonworks.com:8020/user/chiran/externaldbexample
2020-01-22T04:36:27,903 INFO  [pool-6-thread-6]: utils.FileUtils 
(FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist: 
hdfs://namenodeaddress:8020/user/chiran/externaldbexample
2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils 
(MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception: 
org.apache.hadoop.security.AccessControlException Permission denied: user=hive, 
access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at

Re: Review Request 71984: HIVE-22666

2020-01-21 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71984/
---

(Updated Jan. 22, 2020, 7:30 a.m.)


Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-22666
https://issues.apache.org/jira/browse/HIVE-22666


Repository: hive-git


Description
---

implement supporting partitioning in TopNKeyOperator
enable push down of partitioned TopNKeyOperator


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java b7c1250220 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java 
ce6efa4919 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 348fbb5faf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java f752967a96 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestTopNKeyFilter.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_in.q a5b3ce7951 
  ql/src/test/queries/clientpositive/subquery_notin.q f8636453c2 
  ql/src/test/queries/clientpositive/topnkey_windowing.q PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey_windowing_order.q PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_windowing_streaming.q e1011f9949 
  ql/src/test/queries/clientpositive/windowing_filter.q 2483c18416 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 4f0e75289f 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 67e349bb59 
  ql/src/test/results/clientpositive/llap/topnkey_windowing.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/topnkey_windowing_order.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/perf/tez/constraints/mv_query44.q.out 
786bbe12c8 
  ql/src/test/results/clientpositive/perf/tez/constraints/query44.q.out 
eace7a0685 
  ql/src/test/results/clientpositive/perf/tez/constraints/query67.q.out 
9ed7306c46 
  ql/src/test/results/clientpositive/perf/tez/constraints/query70.q.out 
40dfaa2f13 
  ql/src/test/results/clientpositive/perf/tez/query44.q.out a0defab8b3 
  ql/src/test/results/clientpositive/perf/tez/query67.q.out 54d76e9c1a 
  ql/src/test/results/clientpositive/perf/tez/query70.q.out 23f9166a4f 
  ql/src/test/results/clientpositive/topnkey_windowing.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/topnkey_windowing_order.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/71984/diff/5/

Changes: https://reviews.apache.org/r/71984/diff/4-5/


Testing
---

New q test were added
topnkey_windowing.q,topnkey_windowing_order.q

New UT:
TestTopNKeyFilter.java

existing:
windowing_filter.q,topnkey.q,topnkey_grouping_sets.q,topnkey_grouping_sets_functions.q,topnkey_grouping_sets_order.q

existing UT:
TestCommonKeyPrefix


Thanks,

Krisztian Kasa

[jira] [Created] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id

2020-01-21 Thread Slim Bouguerra (Jira) 
Slim Bouguerra created HIVE-22755:
-

 Summary: Cleaner/Compaction can skip the read locks and use the 
min open txn id
 Key: HIVE-22755
 URL: https://issues.apache.org/jira/browse/HIVE-22755
 Project: Hive
  Issue Type: Sub-task
Reporter: Slim Bouguerra






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore

2020-01-21 Thread Ramesh Kumar Thangarajan (Jira) 
Ramesh Kumar Thangarajan created HIVE-22756:
---

 Summary: Drop XAttrs options for distcp when dest is a blobstore
 Key: HIVE-22756
 URL: https://issues.apache.org/jira/browse/HIVE-22756
 Project: Hive
  Issue Type: Bug
Reporter: Ramesh Kumar Thangarajan
Assignee: Ramesh Kumar Thangarajan


Drop XAttrs options for distcp when dest is a blobstore



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TX watermark

2020-01-21 Thread Slim Bouguerra (Jira) 
Slim Bouguerra created HIVE-22754:
-

 Summary: Trim some extra HDFS find file name calls that can be 
deduced using current TX watermark
 Key: HIVE-22754
 URL: https://issues.apache.org/jira/browse/HIVE-22754
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Reporter: Slim Bouguerra
Assignee: Slim Bouguerra






--
This message was sent by Atlassian Jira
(v8.3.4#803005)