[
https://issues.apache.org/jira/browse/HIVE-23965?focusedWorklogId=517195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517195
]
ASF GitHub Bot logged work on HIVE-23965:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Nov/20 22:19
Start Date: 26/Nov/20 22:19
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #1714:
URL: https://github.com/apache/hive/pull/1714
### What changes were proposed in this pull request and why?
1. Add new perf driver, TestTezTPCDS30TBCliDriver, relying on a dockerized
metastore.
2. Use Dockerized postgres metastore with TPC-DS 30TB dump
3. Remove old drivers (with and without constraints), related classes
(e.g., MetastoreDumpUtility), and resources.
4. Use Hive config properties obtained and curated from real-life usages
5. Allow AbstractCliConfig to override metastore DB type
6. Rework CorePerfCliDriver to allow pre-initialized metastores
7. Remove redundant logs in System.err. Logging and throwing an exception
is an anti-pattern.
8. Replace assertions with exceptions and improve the messages.
9. Upgrade postgres JDBC driver to version 42.2.14 to be compatible
with the docker image used
10. Disable queries 14 (HIVE-24167), 30 (HIVE-23964)
11. Re-enable CBO plan tests for queries 44, 45, 67, 70, 86
The queries were disabled as part of HIVE-20718. They were supposed to
be fixed in Calcite 1.18.0 and currently Hive is in 1.21.0 so it is not
surprising that they pass.
12. Add missing queries: cbo_query41, cbo_query62, query62
### Does this PR introduce _any_ user-facing change?
No, except the fact that old TestTezPerf drivers no longer exist.
### How was this patch tested?
`mvn test -Dtest=TestTezTPCDS30TBPerfCliDriver`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 517195)
Time Spent: 5.5h (was: 5h 20m)
> Improve plan regression tests using TPCDS30TB metastore dump and custom
> configs
> -------------------------------------------------------------------------------
>
> Key: HIVE-23965
> URL: https://issues.apache.org/jira/browse/HIVE-23965
> Project: Hive
> Issue Type: Improvement
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: master355.tgz
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> The existing regression tests (HIVE-12586) based on TPC-DS have certain
> shortcomings:
> The table statistics do not reflect cardinalities from a specific TPC-DS
> scale factor (SF). Some tables are from a 30TB dataset, others from 200GB
> dataset, and others from a 3GB dataset. This mix leads to plans that may
> never appear when using an actual TPC-DS dataset.
> The existing statistics do not contain information about partitions something
> that can have a big impact on the resulting plans.
> The existing regression tests rely on more or less on the default
> configuration (hive-site.xml). In real-life scenarios though some of the
> configurations differ and may impact the choices of the optimizer.
> This issue aims to address the above shortcomings by using a curated
> TPCDS30TB metastore dump along with some custom hive configurations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
