[jira] [Created] (HIVE-23129) Cast invalid string to date returns incorrect result

[Yuming Wang (Jira)]

Yuming Wang created HIVE-23129:
--

 Summary: Cast invalid string to date returns incorrect result
 Key: HIVE-23129
 URL: https://issues.apache.org/jira/browse/HIVE-23129
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.2
Reporter: Yuming Wang


{noformat}
hive> select cast('2020-20-20' as date);
OK
2021-08-20
Time taken: 4.436 seconds, Fetched: 1 row(s)
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Inconsistent Hive Object Names

[David Mollitor]

Hello Gang,

I came across an interesting problem and I think the entire community
should be mindful of this situation.

There needs to be better consistency with handling of Object names
(database, tables, column, view, function, etc.).  I think it makes sense
to standardize on the same rules which MySQL/MariaDB uses for their column
names so that Hive can be more of a drop-in replacement for these.

The two important things to keep in mind are:

1// Permitted characters in quoted identifiers include the full Unicode
Basic Multilingual Plane (BMP), except U+

2// If any components of a multiple-part name require quoting, quote them
individually rather than quoting the name as a whole. For example, write
`my-table`.`my-column`, not `my-table.my-column`.

https://dev.mysql.com/doc/refman/8.0/en/identifiers.html
https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html

That is to say:

-- Select all rows from a table named `default.mytable`
-- (Yes, the table name itself has a period in it. This is valid)
SELECT * FROM `default.mytable`;

-- Select all rows from database `default`, table `mytable`
SELECT * FROM `default`.`mytable`;

This plays out in a couple of ways.  There may be more, but these are the
ones I know about already:

1// Hive generates incorrect syntax
https://issues.apache.org/jira/browse/HIVE-23128

2// Hive throws exception if there is a period in the table name.  This is
an invalid response.  Table name may have a period in them. More likely
than not, it will throw 'table not found' exception since the user most
likely accidentally used backticks incorrectly and meant to specify a db
and a table separately.
https://issues.apache.org/jira/browse/HIVE-16907

Thanks.

[jira] [Created] (HIVE-23128) SHOW CREATE TABLE Creates Invalid Syntax When Database Specified

[David Mollitor (Jira)]

David Mollitor created HIVE-23128:
-

 Summary: SHOW CREATE TABLE Creates Invalid Syntax When Database 
Specified
 Key: HIVE-23128
 URL: https://issues.apache.org/jira/browse/HIVE-23128
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.2, 2.4.0
Reporter: David Mollitor


{code:sql}
show create table `sample_07`;
show create table `default`.`sample_07`;
show create table `default.sample_07`;
{code}

{code:none|title=Results}
CREATE TABLE `sample_07`(...)
CREATE TABLE `default.sample_07`(...)
CREATE TABLE `default.sample_07`(...);
{code}

All three queries complete in Hive 2.x (not sure about 3.x) but the results are 
consistent in both versions.  The first result is correct because it does not 
include the database name, however, the subsequent two results are invalid: 
each field must be quoted individually.

{quote}
If any components of a multiple-part name require quoting, quote them 
individually rather than quoting the name as a whole. For example, write 
`my-table`.`my-column`, not `my-table.my-column`.
* https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23127) Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner

[Vineet Garg (Jira)]

Vineet Garg created HIVE-23127:
--

 Summary: Replace listPartitionsByExpr with GetPartitionsWithSpecs 
in Partition pruner
 Key: HIVE-23127
 URL: https://issues.apache.org/jira/browse/HIVE-23127
 Project: Hive
  Issue Type: Task
  Components: HiveServer2
Reporter: Vineet Garg
Assignee: Vineet Garg


GetPartitionsWithSpecs reduces data transfer by deduplicating storage descriptor



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23126) Add TXNS table to sysdb

[Ramesh Kumar Thangarajan (Jira)]

Ramesh Kumar Thangarajan created HIVE-23126:
---

 Summary: Add TXNS table to sysdb
 Key: HIVE-23126
 URL: https://issues.apache.org/jira/browse/HIVE-23126
 Project: Hive
  Issue Type: Task
Reporter: Ramesh Kumar Thangarajan
Assignee: Ramesh Kumar Thangarajan


Add TXNS table to sysdb



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23125) YARN service placement constraints is broken

[Eric Yang (Jira)]

Eric Yang created HIVE-23125:


 Summary: YARN service placement constraints is broken
 Key: HIVE-23125
 URL: https://issues.apache.org/jira/browse/HIVE-23125
 Project: Hive
  Issue Type: Task
Reporter: Eric Yang


YARN service placement constraint does not work with node label nor node 
attributes. Example of placement constraints: 
{code} 
  "placement_policy": {
"constraints": [
  {
"type": "AFFINITY",
"scope": "NODE",
"node_attributes": {
  "label":["genfile"]
},
"target_tags": [
  "ping"
] 
  }
]
  },
{code}

Node attribute added: 
{code} ./bin/yarn nodeattributes -add "host-3.example.com:label=genfile" {code} 

Scheduling activities shows: 
{code}  Node does not match partition or placement constraints, 
unsatisfied PC expression="in,node,ping", target-type=ALLOCATION_TAG 

 1
 host-3.example.com:45454{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72301: HIVE-23089

[Krisztian Kasa]

> On April 1, 2020, 3:10 p.m., Jesús Camacho Rodríguez wrote:
> > Krisztian, could we hold pushing this patch? HIVE-23100 changes the way 
> > that the expressions are generated in Calcite planner (directly as RexNode 
> > from the AST) so we cannot rely on method from SemanticAnalyzer + 
> > RexConverter. Since this is a non-blocking optimization, it seems we do not 
> > need to push it immediately. Please, let me know what you think.

Sure. It can be hold.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72301/#review220172
---


On April 1, 2020, 2:16 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72301/
> ---
> 
> (Updated April 1, 2020, 2:16 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.
> 
> 
> Bugs: HIVE-23089
> https://issues.apache.org/jira/browse/HIVE-23089
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add constraint checks to CBO plan
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c39cb20bb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c87f2d2292 
>   ql/src/test/results/clientnegative/update_notnull_constraint.q.out 
> 32905378e7 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 3ef0744c7b 
> 
> 
> Diff: https://reviews.apache.org/r/72301/diff/1/
> 
> 
> Testing
> ---
> 
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestMiniLlapLocalCliDriver -Dqfile=check_constraint.q,sort_acid.q -pl 
> itests/qtest -Pitests
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestNegativeCliDriver -Dqfile=update_notnull_constraint.q -pl 
> itests/qtest -Pitests
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

Re: Review Request 72301: HIVE-23089

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72301/#review220172
---



Krisztian, could we hold pushing this patch? HIVE-23100 changes the way that 
the expressions are generated in Calcite planner (directly as RexNode from the 
AST) so we cannot rely on method from SemanticAnalyzer + RexConverter. Since 
this is a non-blocking optimization, it seems we do not need to push it 
immediately. Please, let me know what you think.

- Jesús Camacho Rodríguez


On April 1, 2020, 2:16 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72301/
> ---
> 
> (Updated April 1, 2020, 2:16 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.
> 
> 
> Bugs: HIVE-23089
> https://issues.apache.org/jira/browse/HIVE-23089
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add constraint checks to CBO plan
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c39cb20bb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c87f2d2292 
>   ql/src/test/results/clientnegative/update_notnull_constraint.q.out 
> 32905378e7 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 3ef0744c7b 
> 
> 
> Diff: https://reviews.apache.org/r/72301/diff/1/
> 
> 
> Testing
> ---
> 
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestMiniLlapLocalCliDriver -Dqfile=check_constraint.q,sort_acid.q -pl 
> itests/qtest -Pitests
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestNegativeCliDriver -Dqfile=update_notnull_constraint.q -pl 
> itests/qtest -Pitests
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

[jira] [Created] (HIVE-23124) Review of SQLOperation

[David Mollitor (Jira)]

David Mollitor created HIVE-23124:
-

 Summary: Review of SQLOperation 
 Key: HIVE-23124
 URL: https://issues.apache.org/jira/browse/HIVE-23124
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
 Attachments: HIVE-23124.1.patch





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72301: HIVE-23089

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72301/
---

Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.


Bugs: HIVE-23089
https://issues.apache.org/jira/browse/HIVE-23089


Repository: hive-git


Description
---

Add constraint checks to CBO plan


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c39cb20bb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c87f2d2292 
  ql/src/test/results/clientnegative/update_notnull_constraint.q.out 32905378e7 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 3ef0744c7b 


Diff: https://reviews.apache.org/r/72301/diff/1/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=check_constraint.q,sort_acid.q -pl 
itests/qtest -Pitests
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestNegativeCliDriver 
-Dqfile=update_notnull_constraint.q -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa

[jira] [Created] (HIVE-23123) Export/Import of views and materialized views

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23123:
-

 Summary: Export/Import of views and materialized views
 Key: HIVE-23123
 URL: https://issues.apache.org/jira/browse/HIVE-23123
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely


According to 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport 
import and export can be done by using the

 
{code:java}
export table ...
import table ... 
{code}
commands. The document doesn't mention views or materialized views at all, and 
in fact we don't support commands like

 
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ... 
{code}
they can not be parsed at all. The word table is often used though in a broader 
sense, when it means all table like entities, including views and materialized 
views. For example the various Table classes may represent any of these as well.

If I try to export a view with the export table ... command, it goes fine. A 
_metadata file will be created, but no data directory, which is what we'd 
expect. If I try to import it back, an exception is thrown due to the lack of 
the data dir:

 
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
 at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
 at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
 at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
 at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
 at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
 at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485) 
{code}
Still the view gets imported successfully, as data movement wasn't even 
necessary.

If we try to export a materialized view which is transactional, then this 
exception occurs:

 
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException: 
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
 at 
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
 at 
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
 at 
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
 at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
 at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
 at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
 at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
 at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
 at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541) 
{code}
So the export process can not handle it, as the temporary table is not getting 
created.

 

The import command handling have a lot of codes dedicated to importing views 
and materialized views, which suggests that we support the importing (and thus 
also suggests implicitly that we support the exporting) of views and 
materialiezed views.

 

So the conclusion is that we have to decide if we support exporting/importing 
of views and materialized views.
 #  If we decide not to support them then:
 - export process should throw an exception if a view or materialized view is 
the subject
 - the codes specific to view imports should be removed
 # If we decide to support them, then:
 - the commands mentioned above above should be introduced
 - exception should be thrown if not the proper command used (e.g. export view 
on a table)
 - the exceptions mentioned above should be fixed

I prefer #1, I don't think we should support the exporting / importing of 
views. The point of exporting / importing is the transfer of data, not DDL, it 
causes more issues than it solves. Our current documentation also suggests that 
it is only supported for tables.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72283: HIVE-23076 Add batching for openTxn

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72283/
---

(Updated ápr. 1, 2020, 1:12 du)


Review request for hive, Denys Kuzmenko and Marton Bod.


Changes
---

addressed review comments


Bugs: HIVE-23076
https://issues.apache.org/jira/browse/HIVE-23076


Repository: hive-git


Description
---

Add batching for openTxn request for better performance


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 74ef88545e 


Diff: https://reviews.apache.org/r/72283/diff/2/

Changes: https://reviews.apache.org/r/72283/diff/1-2/


Testing
---

Tested it locally against all of the supported RDBMS types:
mysql no patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 2.0941.8211.4624.78631.06   
openTxn0-2 2.4192.1611.7205.86732.43   
openTxn0-102.5782.2891.9737.20428.74   
openTxn0-100   6.9486.8355.25411.0315.91   
openTxn0-1000  51.3150.4933.5693.1016.27   
openTxn115k-1  26.9423.6922.24169.656.13   
openTxn115k-2  25.2623.8122.4250.6816.90   
openTxn115k-10 26.2024.2923.0160.7321.94   
openTxn125k-10029.1428.1825.8143.6311.16 

mysql patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 2.2641.9641.6526.02335.59   
openTxn0-2 2.5382.2891.9326.01329.41   
openTxn0-102.9822.6412.1778.82932.54   
openTxn0-100   6.7756.3865.01221.7327.10   
openTxn0-1000  42.9642.9330.8961.9214.46   
openTxn115k-1  24.2923.2722.4073.6221.64   
openTxn115k-2  24.0523.5822.4628.605.651   
openTxn115k-10 24.4824.0222.9429.976.075   
openTxn125k-10027.9127.5125.7842.506.905   

postgres no patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 3.7342.8832.50611.4655.16   
openTxn0-2 3.8343.1112.63315.5053.22   
openTxn0-105.0054.1783.44916.8047.56   
openTxn0-100   9.8237.7556.83379.3479.96   
openTxn0-1000  75.5172.0358.62207.923.98   
openTxn115k-1  21.7919.4518.4366.7629.10   
openTxn115k-2  21.9120.1418.8851.4220.92   
openTxn115k-10 22.4320.8519.3845.1818.58   
openTxn125k-10027.7125.3623.1954.9921.46   

postgres patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 1.6881.4231.1307.81455.91   
openTxn0-2 1.9821.6621.3067.78647.13   
openTxn0-102.6802.5641.7615.06926.93   
openTxn0-100   8.3407.5355.35130.0037.97   
openTxn0-1000  41.7337.5524.38107.833.87   
openTxn115k-1  12.2411.6510.2126.2319.75   
openTxn115k-2  13.0711.8610.7668.9547.37   
openTxn115k-10 13.0312.2311.0654.8834.23   
openTxn125k-10015.6214.0312.46102.958.21   

Oracle no patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 14.8513.9111.5027.2619.49   
openTxn0-2 17.8917.1314.5627.0013.53   
openTxn0-1023.1221.3817.9167.3725.46   
openTxn0-100   114.199.0382.62214.035.61   
openTxn0-1000  4123 3952 3593 5790 15.96   
openTxn115k-1  16.7416.8814.0121.7514.52   
openTxn115k-2  20.2818.3416.5130.3423.09   
openTxn115k-10 22.4221.0719.8731.3915.74   
openTxn125k-10088.1387.8878.95100.47.990   

Oracle patch
Operation  Mean Med  Min  Max  Err%
openTxn0-1 15.8714.0712.2180.4448.32   
openTxn0-2 17.0616.1412.8033.5219.47   
openTxn0-10  

Re: Review Request 72234: HIVE-22785

[Krisztian Kasa]

> On March 30, 2020, 3:27 p.m., Jesús Camacho Rodríguez wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Lines 18 (patched)
> > 
> >
> > Instead of adding it here, we should add it directly to 
> > `minillaplocal.query.files`: Those are only executed in 
> > MiniLlapLocalCliDriver.

Fixin in a follow-up: https://issues.apache.org/jira/browse/HIVE-23119


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72234/#review220109
---


On March 26, 2020, 7:51 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72234/
> ---
> 
> (Updated March 26, 2020, 7:51 p.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-22785
> https://issues.apache.org/jira/browse/HIVE-22785
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Update/delete/merge statements not optimized through CBO
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/map_join_on_filter.q.out
>  653faab00a 
>   itests/src/test/resources/testconfiguration.properties 3510016c07 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 9c61b316e2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
>  e5f4c8492e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> 04b3888a25 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelJson.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelJsonImpl.java 
> 0d45eb0c61 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 
> e647b88961 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortExchange.java
>  880cae70f9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortExchangeTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  871c411e70 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  53d68e872a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  e51b2b6ebc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortPullUpConstantsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  e03e96ff12 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/PlanModifierForASTConv.java
>  31619c0314 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/opconventer/HiveSortExchangeVisitor.java
>  68227db1ee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/opconventer/JoinVisitor.java
>  0286d54ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 6589eeb39b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RewriteSemanticAnalyzer.java 
> 31068cb8c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 679ae2e1e6 
>   ql/src/test/queries/clientpositive/authorization_view_disable_cbo_1.q 
> be50b69830 
>   ql/src/test/queries/clientpositive/sort.q cab2712810 
>   ql/src/test/queries/clientpositive/sort_acid.q PRE-CREATION 
>   ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite.q.out 
> 2b7ff65c7a 
>   ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite_2.q.out 
> 6850290412 
>   ql/src/test/results/clientnegative/update_notnull_constraint.q.out 
> 86bfc67480 
>   ql/src/test/results/clientpositive/acid_view_delete.q.out 3771c3ba63 
>   ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out 
> b609982bb9 
>   ql/src/test/results/clientpositive/auto_join0.q.out 665cf28dea 
>   ql/src/test/results/clientpositive/auto_join15.q.out 6ea9db28f0 
>   ql/src/test/results/clientpositive/auto_join20.q.out 6bbcb47084 
>   ql/src/test/results/clientpositive/auto_join21.q.out b0af07c93e 
>   ql/src/test/results/clientpositive/auto_join23.q.out 9dcfc1aa6f 
>   ql/src/test/results/clientpositive/auto_join28.q.out 6b27398f1e 
>   ql/src/test/results/clientpositive/auto_join29.q.out ade39bdc1a 
>   ql/src/test/results/clientpositive/auto_join31.q.out 73651d36f9 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join0.q.out 72e041e767 
>   ql/src/test/results/clientpositive/correlationoptimizer14.q.out b8d764841b 
>   ql/src/test/results/clientpositive/identity_project_remove_skip.q.out 
> 1176c7c0d4 
>   ql/src/test/results/clientpositive/input_part7.q.out 

[jira] [Created] (HIVE-23122) LLAP TaskSchedulerService should log details about task eviction decision details

[Jira]

László Bodor created HIVE-23122:
---

 Summary: LLAP TaskSchedulerService should log details about task 
eviction decision details
 Key: HIVE-23122
 URL: https://issues.apache.org/jira/browse/HIVE-23122
 Project: Hive
  Issue Type: Improvement
Reporter: László Bodor






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: HIVE-21508 and Hive 2.3.7 question

[Mass Dosage]

I think, given that we're so close to potentially cutting a 2.3.7 release
(see Alan's separate post to the mailing list) that we shouldn't add
anything else at this this stage. This could potentially be of interest for
a 2.3.8 or 2.4.0 release if the rest of the Hive community agrees.

Thanks,

Adrian

On Tue, 31 Mar 2020 at 13:24, David Mollitor  wrote:

> Hello Team,
>
> Just to throw one more thing in there, awhile ago I put a good chunk of
> time into shoring up the ZK Lock Manager because I worked with a lot of
> folks on locking issues. HDP/CLDR moved away from ZK and is using a RDBMS
> and therefore never paid it much mind. Any interest in rolling it into Hive
> 2?
>
> HIVE-21469
>
> On Tue, Mar 31, 2020, 5:20 AM Mass Dosage  wrote:
>
> > Hey all,
> >
> > We've made some progress on this and are getting closer to a 2.3.7
> release.
> > Alan has identified 2 tests failing on the 2.3 branch that are fixed in
> > newer versions of Hive so he is proposing to backport the fixes for them.
> > The ticket for that is https://issues.apache.org/jira/browse/HIVE-23086
> if
> > you want to watch it and vote it up. Hopefully we can get that merged
> soon
> > and then we'll be good to go.
> >
> > Thanks,
> >
> > Adrian
> >
> > On Sun, 8 Mar 2020 at 02:41, Hyukjin Kwon  wrote:
> >
> > > Thank you so much, Alan and all.
> > >
> > > 2020년 3월 8일 (일) 오전 10:36, Yuming Wang 님이 작성:
> > >
> > >> Great, thank you Alan and Adrian.
> > >>
> > >> On Sun, Mar 8, 2020 at 8:13 AM Alan Gates 
> wrote:
> > >>
> > >>> I'm working with Adrian on getting a 2.3.7 release out.  That will
> pick
> > >>> up everything that is already on the 2.3 branch.
> > >>>
> > >>> Alan.
> > >>>
> > >>> On Sat, Mar 7, 2020 at 6:02 AM Yuming Wang  wrote:
> > >>>
> >  Hi Alan and Owen,
> > 
> >  Is there any plans to release Hive 2.3.7 or Hive 2.4.0? It may be
> the
> >  only one that supports Java 11. Hive 3.x can not support it because
> of
> >  HIVE-22097 .
> > 
> >  On Tue, Feb 11, 2020 at 7:32 PM Mass Dosage 
> >  wrote:
> > 
> > > +1.
> > >
> > > At Expedia Group  we are big users of Hive and are also
> experiencing
> > > issues with not being able to use Hive 2.3.x on Java >8 which is
> > starting
> > > to seriously impact some of our applications which require Java 11.
> > We
> > > worked on HIVE-21508 in order to get it merged into the various
> > branches
> > > and have been asking for a Hive 2.3.7 release for months with no
> > replies to
> > > our questions on this mailing list.
> > >
> > > Could someone from the Hive community please answer and let us know
> > if
> > > there is the possibility of a Hive 2.3.7 release? I've seen at
> least
> > two
> > > other requests for this on the list over the past few months.
> > >
> > > If not we will be forced to fork the current 2.3 branch and release
> > > our own version of Hive 2.3.7 to Maven Central (with a different
> > group id)
> > > so that we can use it (it sounds like this would be useful to
> others
> > out
> > > there too). We'd really rather not do this but I don't see any
> other
> > > solutions.
> > >
> > > Thanks,
> > >
> > > Adrian
> > > --
> > > Adrian Woodhead
> > > Principal Engineer
> > > Expedia Group - 407 St John Street, London, EC1V 4EX
> > >
> > >
> > > On Thu, 30 Jan 2020 at 07:34, Hyukjin Kwon 
> > > wrote:
> > >
> > >> Hi Hive dev team,
> > >>
> > >> As informed earlier, I, Yuming and many people from spark dev have
> > >> made
> > >> huge efforts
> > >> to let Spark use official Hive release. Thanks Alan and all Hive
> dev
> > >> for
> > >> all the efforts for Hive 2.3.6 to make Spark support JDK 11.
> > >>
> > >> Few months ago, an unexpected problem was found. Spark throws
> > >> ClassCastException when
> > >> initializing HiveMetaStoreClient.
> > >> Please see SPARK-29245 <
> > >> https://issues.apache.org/jira/browse/SPARK-29245> for
> > >> more details. This has fixed by HIVE-21508
> > >> .
> > >> We postponed the Hive release request to Spark code freeze
> schedule
> > to
> > >> avoid multiple requests.
> > >>
> > >> Spark is going to freeze code 31st January (tomorrow), and I
> > currently
> > >> foresee the RC starts around March. So, this will be hopefully the
> > >> last
> > >> request for Hive release for Spark 3.0.
> > >>
> > >> I was wondering if we could release Hive 2.3.7 soon so Spark can
> > uses
> > >> it.
> > >>
> > >> Thanks.
> > >>
> > >
> >
>

[jira] [Created] (HIVE-23121) Re-examine TestWarehouseExternalDir to see if it uses HMS translation.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23121:


 Summary: Re-examine TestWarehouseExternalDir to see if it uses HMS 
translation.
 Key: HIVE-23121
 URL: https://issues.apache.org/jira/browse/HIVE-23121
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


TestWarehouseExternalDir currently passes with just one change related to 
HIVE-22995. But that change was assuming it was using HMS Translation to 
convert non-acid managed table to external. 
Ensure that it still does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Marton Bod]

> On April 1, 2020, 9:02 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1379 (patched)
> > 
> >
> > Will this issue unnecessary queries for read only queries? For Oracle 
> > this could increase exexution time
> > 
> > Also why not use executeQueryiesInBatch for this?

As discussed, in the current state, these delete queries are executed every 
time during commitTxn (not during abortTxn). Regarding reusing 
executeQueryiesInBatch, that's a good point, will do.


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/#review220167
---


On April 1, 2020, 6:53 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72290/
> ---
> 
> (Updated April 1, 2020, 6:53 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72290/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72276: HIVE-23084: Implement kill query in multiple HS2 environment

[Peter Varga via Review Board]

> On March 30, 2020, 9:38 a.m., Adam Szita wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithServiceDiscovery.java
> > Lines 94 (patched)
> > 
> >
> > Can be private if not used elsewhere

I reverted this one, since its needed during the query execution and must be 
public. Now every tests passes.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72276/#review220102
---


On March 27, 2020, 10:08 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72276/
> ---
> 
> (Updated March 27, 2020, 10:08 a.m.)
> 
> 
> Review request for hive and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> KILL  command was implemented in:
> 
> https://issues.apache.org/jira/browse/HIVE-17483
> https://issues.apache.org/jira/browse/HIVE-20549
> But it is not working in an environment where service discovery is enabled 
> and more than one HS2 instance is running (except for manually sending the 
> kill query to all HS2 instance).
> 
> Solution:
> 
> If a HS2 instance can't kill a query locally, it should post a kill query 
> request to the Zookeeper
> Every HS2 should watch the Zookeeper for kill query requests and if its 
> running on that instance kill it
> Authorization of kill query should work the same
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 34df01e60e 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java 
> 3973ec9270 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java
>  68a515ccbe 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithServiceDiscovery.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/thrift/TestMiniHS2StateWithNoZookeeper.java
>  99e681e5b2 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestKillQueryZookeeperManager.java
>  PRE-CREATION 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1b60a51ebd 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java db965e7a22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/process/kill/KillQueriesOperation.java
>  afde1a4762 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 8becef1cd3 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 9e497545b5 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 277519cba5 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 181ea5d6d5 
>   service/src/java/org/apache/hive/service/server/KillQueryImpl.java 
> 883e32bd2e 
>   
> service/src/java/org/apache/hive/service/server/KillQueryZookeeperManager.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ZooKeeperHiveHelper.java
>  71d8651712 
> 
> 
> Diff: https://reviews.apache.org/r/72276/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

Re: Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/#review220167
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1379 (patched)


Will this issue unnecessary queries for read only queries? For Oracle this 
could increase exexution time

Also why not use executeQueryiesInBatch for this?


- Peter Vary


On ápr. 1, 2020, 6:53 de, Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72290/
> ---
> 
> (Updated ápr. 1, 2020, 6:53 de)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72290/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Marton Bod]

> On April 1, 2020, 7:38 a.m., Denys Kuzmenko wrote:
> > LGTM, just a few comments.

Thanks for the review!


> On April 1, 2020, 7:38 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 1300 (original), 1305 (patched)
> > 
> >
> > Use setLong, setObject would require type inference

sure


> On April 1, 2020, 7:38 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1390 (patched)
> > 
> >
> > Not 100% sure, I think Laci P was working on removal of 
> > MIN_HISTORY_LEVEL table (HIVE-23107), please check with him.

yes he is, I will rebase onto his change once it's committed


> On April 1, 2020, 7:38 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 4295 (patched)
> > 
> >
> > Please extract this into 
> > org.apache.hadoop.hive.metastore.tools.SQLGenerator

as discussed, let's move it to TxnDbUtil


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/#review220164
---


On April 1, 2020, 6:53 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72290/
> ---
> 
> (Updated April 1, 2020, 6:53 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72290/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/#review220164
---



LGTM, just a few comments.


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 1300 (original), 1305 (patched)


Use setLong, setObject would require type inference



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1390 (patched)


Not 100% sure, I think Laci P was working on removal of MIN_HISTORY_LEVEL 
table (HIVE-23107), please check with him.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 4295 (patched)


Please extract this into org.apache.hadoop.hive.metastore.tools.SQLGenerator


- Denys Kuzmenko


On April 1, 2020, 6:53 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72290/
> ---
> 
> (Updated April 1, 2020, 6:53 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  74ef88545e 
> 
> 
> Diff: https://reviews.apache.org/r/72290/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

[jira] [Created] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-23120:
-

 Summary: TopNKey related tests should be run by 
TestMiniLlapLocalCliDriver only
 Key: HIVE-23120
 URL: https://issues.apache.org/jira/browse/HIVE-23120
 Project: Hive
  Issue Type: Task
  Components: Physical Optimizer
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


TopNKey optimization is only used when the execution framework is Tez.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23119) Test sort_acid should be run by TestMiniLlapLocalCliDriver only

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-23119:
-

 Summary: Test sort_acid should be run by 
TestMiniLlapLocalCliDriver only
 Key: HIVE-23119
 URL: https://issues.apache.org/jira/browse/HIVE-23119
 Project: Hive
  Issue Type: Task
  Components: CBO
Reporter: Krisztian Kasa






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72282: HIVE-23101

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72282/
---

(Updated April 1, 2020, 7:14 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-23101
https://issues.apache.org/jira/browse/HIVE-23101


Repository: hive-git


Description
---

Fix topnkey_grouping_sets


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties cfe83d6cb1 
  ql/src/test/queries/clientpositive/topnkey_grouping_sets.q e8c5401ee5 
  ql/src/test/results/clientpositive/llap/topnkey_grouping_sets.q.out 
41a8c3a5b2 
  ql/src/test/results/clientpositive/topnkey_grouping_sets.q.out 27998efafc 


Diff: https://reviews.apache.org/r/72282/diff/2/

Changes: https://reviews.apache.org/r/72282/diff/1-2/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=topnkey_grouping_sets.q -pl 
itests/qtest -Pitests


Thanks,

Krisztian Kasa

Review Request 72290: HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72290/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

HIVE-23067: Use batch DB calls in TxnHandler for commitTxn and abortTxns


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 74ef88545e 


Diff: https://reviews.apache.org/r/72290/diff/1/


Testing
---

Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21347/


Thanks,

Marton Bod