Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 74ef88545e 


Diff: https://reviews.apache.org/r/72290/diff/1/


Testing
---

Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/


Thanks,

Marton Bod

[DISCUSS] I propose a Hive 2.3.7 release

[Alan Gates]

Hope everyone is well.

There have been several requests on the Hive mailing list for a 2.3.7
release[1]  I've been working with Adrian Woodhead (aka Mass Dosage) on
testing the 2.3 branch.  We found two test failures there, which I've
posted a patch for at HIVE-23086 [2] (thanks Jésus for the review).

If everyone is agreeable I'll commit the changes to HIVE-23086 and put up a
release candidate for Hive 2.3.7.

Alan.

1.
https://lists.apache.org/thread.html/r1b5040489451e668275b94f2d2f67fca9050f83e94907674d19f864f%40%3Cdev.hive.apache.org%3E
 ,
https://lists.apache.org/thread.html/8473e04420b70d90502e221e663b341c877224cbc617515e40e8e787%40%3Cdev.hive.apache.org%3E

2. https://issues.apache.org/jira/browse/HIVE-23086

[jira] [Created] (HIVE-23118) Option for exposing compile time counters as tez counters

[Prasanth Jayachandran (Jira)]

Prasanth Jayachandran created HIVE-23118:


 Summary: Option for exposing compile time counters as tez counters
 Key: HIVE-23118
 URL: https://issues.apache.org/jira/browse/HIVE-23118
 Project: Hive
  Issue Type: Improvement
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


TezCounters currently are runtime only. Some compile time information from 
optimizer can be exposed as counters which can then be used by workload 
management to make runtime decisions. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23117) Review of Hive Statement

[David Mollitor (Jira)]

David Mollitor created HIVE-23117:
-

 Summary: Review of Hive Statement
 Key: HIVE-23117
 URL: https://issues.apache.org/jira/browse/HIVE-23117
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor
 Attachments: HIVE-23117.1.patch

* Remove unused instance variable(s)
 * Remove non-JavaDoc comments
 * Make inPlaceUpdateStream Optional (and remove NO-OP class)
 * {\{getQueryTimeout()}} returns incorrect value
 * Unify and improve Exception messages
 * Checkstyle fixes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72293: HIVE-23094

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72293/
---

Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.


Bugs: HIVE-23094
https://issues.apache.org/jira/browse/HIVE-23094


Repository: hive-git


Description
---

Implement Explain CBO of Update and Delete statements


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Context.java fd627c69e8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
179021e2b9 
  ql/src/test/queries/clientpositive/sort_acid.q 9d0b9d0663 
  ql/src/test/results/clientpositive/llap/sort_acid.q.out ff0a5adddc 


Diff: https://reviews.apache.org/r/72293/diff/1/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=sort_acid.q -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa

[jira] [Created] (HIVE-23116) get_partition_with_specs does not use filter spec when projection is empty

[Vihang Karajgaonkar (Jira)]

Vihang Karajgaonkar created HIVE-23116:
--

 Summary: get_partition_with_specs does not use filter spec when 
projection is empty
 Key: HIVE-23116
 URL: https://issues.apache.org/jira/browse/HIVE-23116
 Project: Hive
  Issue Type: Sub-task
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


The API implementation ignores the filter spec if the project spec is empty as 
seen here 
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3903]

if (fieldList == null || fieldList.isEmpty()) {
  // no fields are requested. Fallback to regular getPartitions 
implementation to return all the fields
  return getPartitionsInternal(table.getCatName(), table.getDbName(), 
table.getTableName(), -1,
  true, true);
}

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23115) SQL with "WITH" clause not suppoting Uppercase table name while creating Views

[Dilip Kumar (Jira)]

Dilip Kumar created HIVE-23115:
--

 Summary: SQL with "WITH" clause not suppoting Uppercase table name 
while creating Views
 Key: HIVE-23115
 URL: https://issues.apache.org/jira/browse/HIVE-23115
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 2.1.1
Reporter: Dilip Kumar


*Beeline version 2.1.1-mapr-1912*

SQL with "WITH" clause not suppoting Uppercase table name while creating Views

Below query creates the VIEW successfully, 

CREATE VIEW default.TEST_VIEW as
with TEMP_TABLE as (select col1,col2 from table1)
 SELECT col1 from TEMP_TABLE limit 1;

but when I execute select query on the view throws below error

{color:#FF}Error: Error while compiling statement: FAILED: 
SemanticException Line 3:90 Table not found 'TEMP_TABLE' in definition of VIEW 
TEST_VIEW{color}

 

When you replace uppercase TEMP_TABLE to lower case temp_table and query the 
Vew-TEST_VIEW it works,



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with ACID tables with direct insert and with insert-only tables

[Marta Kuczora (Jira)]

Marta Kuczora created HIVE-23114:


 Summary: Insert overwrite with dynamic partitioning is not working 
correctly with ACID tables with direct insert and with insert-only tables
 Key: HIVE-23114
 URL: https://issues.apache.org/jira/browse/HIVE-23114
 Project: Hive
  Issue Type: Bug
Reporter: Marta Kuczora
Assignee: Marta Kuczora






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72291: HIVE-23107: Remove MIN_HISTORY_LEVEL table

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72291/#review220153
---


Ship it!




Just a question, maybe not relevant.

Thanks for the patch LGTM +1 (pending tests :))


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 545-560 (original), 496-512 (patched)


How confident we are with this query?
Maybe delete immediately, like:
DELETE FROM TXN_TO_WRITE_ID WHERE T2W_TXNID < (SELECT MIN...)
The hard part is that it is hard to log what happened


- Peter Vary


On márc. 31, 2020, 1:51 du, Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72291/
> ---
> 
> (Updated márc. 31, 2020, 1:51 du)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23107: Remove MIN_HISTORY_LEVEL table
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
> 54b616e60c73fa1005c6d679ea76d65e01a0749d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  bf91ae704c83722502acaf445061bf297fde6a6f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  19a95b64db64b7f8bf8e82fbdedf5b54cd30aed3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  ef88240a791cbc553b6150b54701c7df8daf3b49 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e6fd680b311ca5c5e4d87f053af1026 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  41d2e7924b1286f82d212e051714107505fe9661 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  48ad67623143fcba2bb2507cc36ad46a58c13b75 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  7a230bde3a4dcd2504538bf8122db0bb2a59932f 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  9ed7f4f8192e1613f8942af3e9496c3ef7f1f04f 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  12d24e9a569310c57d86b1229deaa2b1d080e0b8 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  bc34b511a978af2a6704e47d8f73c9604c6cebf9 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  13f03bce6d579c18ff9a14e09133063fdcdbf7af 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  8482b5942c6284eda63ec7d2aa7c1202abb3db49 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  cbfdd861fd545ce786f5e267b284196f5fd9af03 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  aa35a7a7b392d4d46480c03ed95ffdec12b03b22 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  9462328a5f0c6dda8cffb683860153b8eb3aacec 
> 
> 
> Diff: https://reviews.apache.org/r/72291/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-23113) Clean Up HiveCallableStatement

[David Mollitor (Jira)]

David Mollitor created HIVE-23113:
-

 Summary: Clean Up HiveCallableStatement
 Key: HIVE-23113
 URL: https://issues.apache.org/jira/browse/HIVE-23113
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor


* Add a useful class comment
 * Remove all non-javadoc comments
 * Remove 'TODO' tags
 * Add {{@override}} tags
 * Checkstyle formatting



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72291: HIVE-23107: Remove MIN_HISTORY_LEVEL table

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72291/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-23107: Remove MIN_HISTORY_LEVEL table


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
54b616e60c73fa1005c6d679ea76d65e01a0749d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 bf91ae704c83722502acaf445061bf297fde6a6f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 19a95b64db64b7f8bf8e82fbdedf5b54cd30aed3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 ef88240a791cbc553b6150b54701c7df8daf3b49 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 74ef88545e6fd680b311ca5c5e4d87f053af1026 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 41d2e7924b1286f82d212e051714107505fe9661 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 48ad67623143fcba2bb2507cc36ad46a58c13b75 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 7a230bde3a4dcd2504538bf8122db0bb2a59932f 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 9ed7f4f8192e1613f8942af3e9496c3ef7f1f04f 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 12d24e9a569310c57d86b1229deaa2b1d080e0b8 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 bc34b511a978af2a6704e47d8f73c9604c6cebf9 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 13f03bce6d579c18ff9a14e09133063fdcdbf7af 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 8482b5942c6284eda63ec7d2aa7c1202abb3db49 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 cbfdd861fd545ce786f5e267b284196f5fd9af03 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 aa35a7a7b392d4d46480c03ed95ffdec12b03b22 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 9462328a5f0c6dda8cffb683860153b8eb3aacec 


Diff: https://reviews.apache.org/r/72291/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Added to Wiki

[David Mollitor]

Hello Gang,

Is anyone able to point me in the right direction on how to obtain access?

Thanks!

On Thu, Mar 26, 2020 at 3:15 PM David Mollitor  wrote:

> Hello Team,
>
> Can someone please add me to the Wiki so that I may edit?
>
> Thanks!
> David Mollitor (dmollitor)
>

Re: HIVE-21508 and Hive 2.3.7 question

[David Mollitor]

Hello Team,

Just to throw one more thing in there, awhile ago I put a good chunk of
time into shoring up the ZK Lock Manager because I worked with a lot of
folks on locking issues. HDP/CLDR moved away from ZK and is using a RDBMS
and therefore never paid it much mind. Any interest in rolling it into Hive
2?

HIVE-21469

On Tue, Mar 31, 2020, 5:20 AM Mass Dosage  wrote:

> Hey all,
>
> We've made some progress on this and are getting closer to a 2.3.7 release.
> Alan has identified 2 tests failing on the 2.3 branch that are fixed in
> newer versions of Hive so he is proposing to backport the fixes for them.
> The ticket for that is https://issues.apache.org/jira/browse/HIVE-23086 if
> you want to watch it and vote it up. Hopefully we can get that merged soon
> and then we'll be good to go.
>
> Thanks,
>
> Adrian
>
> On Sun, 8 Mar 2020 at 02:41, Hyukjin Kwon  wrote:
>
> > Thank you so much, Alan and all.
> >
> > 2020년 3월 8일 (일) 오전 10:36, Yuming Wang 님이 작성:
> >
> >> Great, thank you Alan and Adrian.
> >>
> >> On Sun, Mar 8, 2020 at 8:13 AM Alan Gates  wrote:
> >>
> >>> I'm working with Adrian on getting a 2.3.7 release out.  That will pick
> >>> up everything that is already on the 2.3 branch.
> >>>
> >>> Alan.
> >>>
> >>> On Sat, Mar 7, 2020 at 6:02 AM Yuming Wang  wrote:
> >>>
>  Hi Alan and Owen,
> 
>  Is there any plans to release Hive 2.3.7 or Hive 2.4.0? It may be the
>  only one that supports Java 11. Hive 3.x can not support it because of
>  HIVE-22097 .
> 
>  On Tue, Feb 11, 2020 at 7:32 PM Mass Dosage 
>  wrote:
> 
> > +1.
> >
> > At Expedia Group  we are big users of Hive and are also experiencing
> > issues with not being able to use Hive 2.3.x on Java >8 which is
> starting
> > to seriously impact some of our applications which require Java 11.
> We
> > worked on HIVE-21508 in order to get it merged into the various
> branches
> > and have been asking for a Hive 2.3.7 release for months with no
> replies to
> > our questions on this mailing list.
> >
> > Could someone from the Hive community please answer and let us know
> if
> > there is the possibility of a Hive 2.3.7 release? I've seen at least
> two
> > other requests for this on the list over the past few months.
> >
> > If not we will be forced to fork the current 2.3 branch and release
> > our own version of Hive 2.3.7 to Maven Central (with a different
> group id)
> > so that we can use it (it sounds like this would be useful to others
> out
> > there too). We'd really rather not do this but I don't see any other
> > solutions.
> >
> > Thanks,
> >
> > Adrian
> > --
> > Adrian Woodhead
> > Principal Engineer
> > Expedia Group - 407 St John Street, London, EC1V 4EX
> >
> >
> > On Thu, 30 Jan 2020 at 07:34, Hyukjin Kwon 
> > wrote:
> >
> >> Hi Hive dev team,
> >>
> >> As informed earlier, I, Yuming and many people from spark dev have
> >> made
> >> huge efforts
> >> to let Spark use official Hive release. Thanks Alan and all Hive dev
> >> for
> >> all the efforts for Hive 2.3.6 to make Spark support JDK 11.
> >>
> >> Few months ago, an unexpected problem was found. Spark throws
> >> ClassCastException when
> >> initializing HiveMetaStoreClient.
> >> Please see SPARK-29245 <
> >> https://issues.apache.org/jira/browse/SPARK-29245> for
> >> more details. This has fixed by HIVE-21508
> >> .
> >> We postponed the Hive release request to Spark code freeze schedule
> to
> >> avoid multiple requests.
> >>
> >> Spark is going to freeze code 31st January (tomorrow), and I
> currently
> >> foresee the RC starts around March. So, this will be hopefully the
> >> last
> >> request for Hive release for Spark 3.0.
> >>
> >> I was wondering if we could release Hive 2.3.7 soon so Spark can
> uses
> >> it.
> >>
> >> Thanks.
> >>
> >
>

Re: Review Request 72264: HIVE-23052: Optimize lock enqueueing in TxnHandler

[Marton Bod]

> On March 31, 2020, 9:35 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 199 (patched)
> > 
> >
> > %d could be problematic with different locales

I will change to be on the safe side


> On March 31, 2020, 9:35 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 2412 (patched)
> > 
> >
> > nit: unnecessary move, might make backporting harder

sure, makes sense


> On March 31, 2020, 9:35 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 2545 (patched)
> > 
> >
> > Is it possible to collect the all of the data before starting the RDBMS 
> > query, and get all of the data for a enqueueLockRequest in one go?

As discussed, good idea, but it would overcomplicate the code and the query 
structure, whereas usually only 1-2 dbName-tblName combos are locked in one go


> On March 31, 2020, 9:35 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 2599 (patched)
> > 
> >
> > Do we need this? We do not commit temp lock id, so we can have just a 
> > single value which definitely will not appear in the table

makes sense, will change to a cheaper placeholder value


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72264/#review220146
---


On March 24, 2020, 2:24 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72264/
> ---
> 
> (Updated March 24, 2020, 2:24 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23052: Optimize lock enqueueing in TxnHandler
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  f92ce7325e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
>  85719fdf84 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
>  76bbf3bc1e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72264/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21249/
> Custom benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23052
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: HIVE-21508 and Hive 2.3.7 question

[Yuming Wang]

Awesome, thanks for the update!

On Tue, Mar 31, 2020 at 5:20 PM Mass Dosage  wrote:

> Hey all,
>
> We've made some progress on this and are getting closer to a 2.3.7
> release. Alan has identified 2 tests failing on the 2.3 branch that are
> fixed in newer versions of Hive so he is proposing to backport the fixes
> for them. The ticket for that is
> https://issues.apache.org/jira/browse/HIVE-23086 if you want to watch it
> and vote it up. Hopefully we can get that merged soon and then we'll be
> good to go.
>
> Thanks,
>
> Adrian
>
> On Sun, 8 Mar 2020 at 02:41, Hyukjin Kwon  wrote:
>
>> Thank you so much, Alan and all.
>>
>> 2020년 3월 8일 (일) 오전 10:36, Yuming Wang 님이 작성:
>>
>>> Great, thank you Alan and Adrian.
>>>
>>> On Sun, Mar 8, 2020 at 8:13 AM Alan Gates  wrote:
>>>
 I'm working with Adrian on getting a 2.3.7 release out.  That will pick
 up everything that is already on the 2.3 branch.

 Alan.

 On Sat, Mar 7, 2020 at 6:02 AM Yuming Wang  wrote:

> Hi Alan and Owen,
>
> Is there any plans to release Hive 2.3.7 or Hive 2.4.0? It may be the
> only one that supports Java 11. Hive 3.x can not support it because of
> HIVE-22097 .
>
> On Tue, Feb 11, 2020 at 7:32 PM Mass Dosage 
> wrote:
>
>> +1.
>>
>> At Expedia Group  we are big users of Hive and are also experiencing
>> issues with not being able to use Hive 2.3.x on Java >8 which is starting
>> to seriously impact some of our applications which require Java 11. We
>> worked on HIVE-21508 in order to get it merged into the various branches
>> and have been asking for a Hive 2.3.7 release for months with no replies 
>> to
>> our questions on this mailing list.
>>
>> Could someone from the Hive community please answer and let us know
>> if there is the possibility of a Hive 2.3.7 release? I've seen at least 
>> two
>> other requests for this on the list over the past few months.
>>
>> If not we will be forced to fork the current 2.3 branch and release
>> our own version of Hive 2.3.7 to Maven Central (with a different group 
>> id)
>> so that we can use it (it sounds like this would be useful to others out
>> there too). We'd really rather not do this but I don't see any other
>> solutions.
>>
>> Thanks,
>>
>> Adrian
>> --
>> Adrian Woodhead
>> Principal Engineer
>> Expedia Group - 407 St John Street, London, EC1V 4EX
>>
>>
>> On Thu, 30 Jan 2020 at 07:34, Hyukjin Kwon 
>> wrote:
>>
>>> Hi Hive dev team,
>>>
>>> As informed earlier, I, Yuming and many people from spark dev have
>>> made
>>> huge efforts
>>> to let Spark use official Hive release. Thanks Alan and all Hive dev
>>> for
>>> all the efforts for Hive 2.3.6 to make Spark support JDK 11.
>>>
>>> Few months ago, an unexpected problem was found. Spark throws
>>> ClassCastException when
>>> initializing HiveMetaStoreClient.
>>> Please see SPARK-29245 <
>>> https://issues.apache.org/jira/browse/SPARK-29245> for
>>> more details. This has fixed by HIVE-21508
>>> .
>>> We postponed the Hive release request to Spark code freeze schedule
>>> to
>>> avoid multiple requests.
>>>
>>> Spark is going to freeze code 31st January (tomorrow), and I
>>> currently
>>> foresee the RC starts around March. So, this will be hopefully the
>>> last
>>> request for Hive release for Spark 3.0.
>>>
>>> I was wondering if we could release Hive 2.3.7 soon so Spark can
>>> uses it.
>>>
>>> Thanks.
>>>
>>

Re: Review Request 72234: HIVE-22785

[Krisztian Kasa]

> On March 30, 2020, 3:27 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite.q.out
> > Line 22 (original), 22 (patched)
> > 
> >
> > Can we make this comment more user friendly; the main reason for this 
> > is that the MV contains SORT BY but the new message can be cryptic for 
> > users. (We can defer this to a follow-up).

Order by works the same way. Will create a follow-up.


> On March 30, 2020, 3:27 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/test/results/clientnegative/update_notnull_constraint.q.out
> > Lines 29 (patched)
> > 
> >
> > This error message seems less descriptive too. Are we hitting same 
> > exception as we were hitting previously, and it is just a matter of 
> > printing the error correctly? If that is the case, we can tackle in 
> > follow-up.

Yes. The original error message still can be found in hive.log.
The follow up patch about handling constraints in CBO should fix this: 
https://issues.apache.org/jira/browse/HIVE-23089


> On March 30, 2020, 3:27 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/test/results/clientpositive/llap/vector_outer_reference_windowed.q.out
> > Line 458 (original), 458 (patched)
> > 
> >
> > Can you check the logs for this test to make sure CBO is executed now 
> > and that is the main reason for this change?

CBO is executed and succeeded. Ex:
```
explain vectorization detail
select sum(sum(c1)) over() from e011_01_n0
```
```
HiveProject(sum_window_0=[sum($0) OVER (PARTITION BY 0 ORDER BY 0 NULLS FIRST 
ROWS BETWEEN 2147483647 FOLLOWING AND 2147483647 PRECEDING)])
  HiveProject($f0=[$0])
HiveAggregate(group=[{}], agg#0=[sum($0)])
  HiveTableScan(table=[[default, e011_01_n0]], table:alias=[e011_01_n0])
```

Before this patch CBO plan was
```
HiveProject(sum_window_0=[$0])
  HiveProject(sum_window_0=[sum($0) OVER (PARTITION BY 0 ORDER BY 0 NULLS FIRST 
ROWS BETWEEN 2147483647 FOLLOWING AND 2147483647 PRECEDING)], window_col_0=[$0])
HiveProject($f0=[$0])
  HiveAggregate(group=[{}], agg#0=[sum($0)])
HiveTableScan(table=[[default, e011_01_n0]], table:alias=[e011_01_n0])
```

Cause: a `HiveProjectMergeRule` was added after 
`HiveProjectSortExchangeTransposeRule` in `applyPostJoinOrderingTransform`.
It was necessary because `HiveProjectSortExchangeTransposeRule` moves 
`Projects` and we can end up with a plan having more `Projects` next to each 
other and that would result an unnecessary subquery in the AST.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72234/#review220109
---


On March 26, 2020, 7:51 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72234/
> ---
> 
> (Updated March 26, 2020, 7:51 p.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-22785
> https://issues.apache.org/jira/browse/HIVE-22785
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Update/delete/merge statements not optimized through CBO
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/map_join_on_filter.q.out
>  653faab00a 
>   itests/src/test/resources/testconfiguration.properties 3510016c07 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 9c61b316e2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
>  e5f4c8492e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> 04b3888a25 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelJson.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelJsonImpl.java 
> 0d45eb0c61 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 
> e647b88961 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortExchange.java
>  880cae70f9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortExchangeTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  871c411e70 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  53d68e872a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  e51b2b6ebc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortPullUpConsta

Re: Review Request 72264: HIVE-23052: Optimize lock enqueueing in TxnHandler

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72264/#review220146
---



Thanks for the patch Marci!
A few questions below, feel free to correct me if I am wrong - this was just a 
cursory review.

Thanks,
Peter


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 199 (patched)


%d could be problematic with different locales



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 2412 (patched)


nit: unnecessary move, might make backporting harder



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 2545 (patched)


Is it possible to collect the all of the data before starting the RDBMS 
query, and get all of the data for a enqueueLockRequest in one go?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 2599 (patched)


Do we need this? We do not commit temp lock id, so we can have just a 
single value which definitely will not appear in the table


- Peter Vary


On márc. 24, 2020, 2:24 du, Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72264/
> ---
> 
> (Updated márc. 24, 2020, 2:24 du)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23052: Optimize lock enqueueing in TxnHandler
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  f92ce7325e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
>  85719fdf84 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
>  76bbf3bc1e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72264/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21249/
> Custom benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23052
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: HIVE-21508 and Hive 2.3.7 question

[Mass Dosage]

Hey all,

We've made some progress on this and are getting closer to a 2.3.7 release.
Alan has identified 2 tests failing on the 2.3 branch that are fixed in
newer versions of Hive so he is proposing to backport the fixes for them.
The ticket for that is https://issues.apache.org/jira/browse/HIVE-23086 if
you want to watch it and vote it up. Hopefully we can get that merged soon
and then we'll be good to go.

Thanks,

Adrian

On Sun, 8 Mar 2020 at 02:41, Hyukjin Kwon  wrote:

> Thank you so much, Alan and all.
>
> 2020년 3월 8일 (일) 오전 10:36, Yuming Wang 님이 작성:
>
>> Great, thank you Alan and Adrian.
>>
>> On Sun, Mar 8, 2020 at 8:13 AM Alan Gates  wrote:
>>
>>> I'm working with Adrian on getting a 2.3.7 release out.  That will pick
>>> up everything that is already on the 2.3 branch.
>>>
>>> Alan.
>>>
>>> On Sat, Mar 7, 2020 at 6:02 AM Yuming Wang  wrote:
>>>
 Hi Alan and Owen,

 Is there any plans to release Hive 2.3.7 or Hive 2.4.0? It may be the
 only one that supports Java 11. Hive 3.x can not support it because of
 HIVE-22097 .

 On Tue, Feb 11, 2020 at 7:32 PM Mass Dosage 
 wrote:

> +1.
>
> At Expedia Group  we are big users of Hive and are also experiencing
> issues with not being able to use Hive 2.3.x on Java >8 which is starting
> to seriously impact some of our applications which require Java 11. We
> worked on HIVE-21508 in order to get it merged into the various branches
> and have been asking for a Hive 2.3.7 release for months with no replies 
> to
> our questions on this mailing list.
>
> Could someone from the Hive community please answer and let us know if
> there is the possibility of a Hive 2.3.7 release? I've seen at least two
> other requests for this on the list over the past few months.
>
> If not we will be forced to fork the current 2.3 branch and release
> our own version of Hive 2.3.7 to Maven Central (with a different group id)
> so that we can use it (it sounds like this would be useful to others out
> there too). We'd really rather not do this but I don't see any other
> solutions.
>
> Thanks,
>
> Adrian
> --
> Adrian Woodhead
> Principal Engineer
> Expedia Group - 407 St John Street, London, EC1V 4EX
>
>
> On Thu, 30 Jan 2020 at 07:34, Hyukjin Kwon 
> wrote:
>
>> Hi Hive dev team,
>>
>> As informed earlier, I, Yuming and many people from spark dev have
>> made
>> huge efforts
>> to let Spark use official Hive release. Thanks Alan and all Hive dev
>> for
>> all the efforts for Hive 2.3.6 to make Spark support JDK 11.
>>
>> Few months ago, an unexpected problem was found. Spark throws
>> ClassCastException when
>> initializing HiveMetaStoreClient.
>> Please see SPARK-29245 <
>> https://issues.apache.org/jira/browse/SPARK-29245> for
>> more details. This has fixed by HIVE-21508
>> .
>> We postponed the Hive release request to Spark code freeze schedule to
>> avoid multiple requests.
>>
>> Spark is going to freeze code 31st January (tomorrow), and I currently
>> foresee the RC starts around March. So, this will be hopefully the
>> last
>> request for Hive release for Spark 3.0.
>>
>> I was wondering if we could release Hive 2.3.7 soon so Spark can uses
>> it.
>>
>> Thanks.
>>
>

Re: Review Request 72264: HIVE-23052: Optimize lock enqueueing in TxnHandler

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72264/#review220139
---



LGTM, 1 small comment, for your consideration, otherwise - ship it!


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2443 (original), 2496 (patched)


Could we combine this block with above check?


- Denys Kuzmenko


On March 24, 2020, 2:24 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72264/
> ---
> 
> (Updated March 24, 2020, 2:24 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23052: Optimize lock enqueueing in TxnHandler
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  f92ce7325e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
>  85719fdf84 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
>  76bbf3bc1e 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72264/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21249/
> Custom benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23052
> 
> 
> Thanks,
> 
> Marton Bod
> 
>