Re: Review Request 69432: HIVE-20964 Create a test that checks the level of the parallel compilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69432/#review210800 --- Ship it! Ship It! - Denys Kuzmenko On Nov. 22, 2018, 3:19 p.m., Peter Vary wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69432/ > --- > > (Updated Nov. 22, 2018, 3:19 p.m.) > > > Review request for hive, Denys Kuzmenko, Marta Kuczora, and Adam Szita. > > > Bugs: HIVE-20964 > https://issues.apache.org/jira/browse/HIVE-20964 > > > Repository: hive-git > > > Description > --- > > * Created 2 query types in the TestCompileLock mock driver. The original > SHORT_QUERY is finishing in 0.5s as before, but the new LONG_QUERY will > finish only after 5s. > * With using the new 5s query I have created a new test where the compile > quota is 4 and the parallel request number is 10. So the test expects that 6 > query will fail with timeout. > * Added a new verifyThatTimedOutCompileOpsCount method to validate the number > of the timed out queries. > * The other changes are just pushing down the query string so the > compileAndRespond method can decide which query to run. > > > Diffs > - > > ql/src/test/org/apache/hadoop/hive/ql/TestCompileLock.java 8dc05ff480 > > > Diff: https://reviews.apache.org/r/69432/diff/1/ > > > Testing > --- > > Run the new test, and all the old tests in TestCompileLock > > > Thanks, > > Peter Vary > >
Review Request 69432: HIVE-20964 Create a test that checks the level of the parallel compilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69432/ --- Review request for hive, Denys Kuzmenko, Marta Kuczora, and Adam Szita. Bugs: HIVE-20964 https://issues.apache.org/jira/browse/HIVE-20964 Repository: hive-git Description --- * Created 2 query types in the TestCompileLock mock driver. The original SHORT_QUERY is finishing in 0.5s as before, but the new LONG_QUERY will finish only after 5s. * With using the new 5s query I have created a new test where the compile quota is 4 and the parallel request number is 10. So the test expects that 6 query will fail with timeout. * Added a new verifyThatTimedOutCompileOpsCount method to validate the number of the timed out queries. * The other changes are just pushing down the query string so the compileAndRespond method can decide which query to run. Diffs - ql/src/test/org/apache/hadoop/hive/ql/TestCompileLock.java 8dc05ff480 Diff: https://reviews.apache.org/r/69432/diff/1/ Testing --- Run the new test, and all the old tests in TestCompileLock Thanks, Peter Vary
[jira] [Created] (HIVE-20964) Create a test that check the level of the parallel compilation
Peter Vary created HIVE-20964:
-
Summary: Create a test that check the level of the parallel
compilation
Key: HIVE-20964
URL: https://issues.apache.org/jira/browse/HIVE-20964
Project: Hive
Issue Type: Test
Components: Test
Reporter: Peter Vary
Assignee: Peter Vary
Create a test which verifies that the
{{hive.driver.parallel.compilation.global.limit}} is honored and creates a hard
limit for the number of parallel compilations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 69410: HIVE-20330: HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69410/#review210792 --- My only concen is that some other components might use HCAT_KEY_JOB_INFO property values as well? Was this a public property key? Otherwise nicely done! - Peter Vary On nov. 20, 2018, 12:53 du, Adam Szita wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69410/ > --- > > (Updated nov. 20, 2018, 12:53 du) > > > Review request for hive, Nandor Kollar and Peter Vary. > > > Bugs: HIVE-20330 > https://issues.apache.org/jira/browse/HIVE-20330 > > > Repository: hive-git > > > Description > --- > > The change in this patch is that we're not just serializing and putting one > InputJobInfo into JobConf, but rather always append to a list (or create it > on the first occurrence) of InputJobInfo instances in it. > This ensures that if multiple tables serve as inputs in a job, Pig can > retrieve information for each of the tables, not just the last one added. > > I've also discovered a bug in InputJobInfo.writeObject() where the > ObjectOutputStream was closed by mistake after writing partition information > in a compressed manner. Closing the compressed writer inevitably closed the > OOS on the context and prevented any other objects to be written into OOS - I > had to fix that because it prevented serializing InputJobInfo instances > inside a list. > > > Diffs > - > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java > 8e72a1275a5cdcc2d778080fff6bb82198395f5f > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java > 195eaa367933990e3ef0ef879f34049c65822aee > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java > 8d7a8f9df9412105ec7d77fad9af0d7dd18f4323 > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java > ad6f3eb9f93338023863c6239d6af0449b20ff9c > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java > 364382d9ccf6eb9fc29689b0eb5f973f422051b4 > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InputJobInfo.java > ac1dd54be821d32aa008d41514df05a41f16223c > > hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java > 91aa4fa2693e0b0bd65c1667210af340619f552d > > hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java > c3bde2d2a3cbd09fb0b1ed758bf4f2b1041a23cb > > hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/AbstractHCatLoaderTest.java > 58981f88ef6abfbf7a4b7ffc3116c53d47e86fde > > > Diff: https://reviews.apache.org/r/69410/diff/1/ > > > Testing > --- > > Added (true) unit tests to verify my method of adding/retrieving InputJobInfo > instances to/from config instances. > Added (integration-like) unit tests to mock Pig calling HCatLoader for > multiple input tables, and checking the reported input sizes. > > > Thanks, > > Adam Szita > >
[jira] [Created] (HIVE-20963) Handle C-Style comments in hive query
Shubhangi Pardeshi created HIVE-20963: - Summary: Handle C-Style comments in hive query Key: HIVE-20963 URL: https://issues.apache.org/jira/browse/HIVE-20963 Project: Hive Issue Type: Bug Components: Parser Reporter: Shubhangi Pardeshi h3. Problem Currently only Std. SQL. style comment i.e. "–" can be used in query. Requesting to add support for C-Style single line as well as multiline comments. 1. /* */ 2. /* */ 3. // -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20962) CommonMergeJoinOperator cannot join on complex keys
Clemens Valiente created HIVE-20962:
---
Summary: CommonMergeJoinOperator cannot join on complex keys
Key: HIVE-20962
URL: https://issues.apache.org/jira/browse/HIVE-20962
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 2.3.4
Reporter: Clemens Valiente
CommonMergeJoinOperator fails to perform joins on complex keys, e.g.
{code:sql}
CREATE TABLE complex_key (
`key` struct
value int)
PARTITIONED BY (date int);
SELECT t1.key, t1.value, t2.value FROM
complex_key t1
FULL OUTER JOIN
complex_key t2
ON (t1.date=20181121 and t2.date =20181122 AND t1.key=t2.key);
{code}
This causes a ClassCastException:
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=1)
{"key":{"reducesinkkey0":{"id":1,"country":"DK"}},"value":{"_col0":1489}}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
... 22 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to
org.apache.hadoop.io.WritableComparable
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.compareKeys(CommonMergeJoinOperator.java:543)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:516)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:212)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
{code}
Because the compareKeys() method tries to cast each key to a WritableComparable
but e.g. the StandardStructObjectInspector would return our key field as an
Arraylist.
https://github.com/apache/hive/blob/66f97da9de65b1c7151ec57bdf9ada937855bd75/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L590
Proper way to do it would probably be to use the KeyWrapperFactory to convert
the keys to something easily comparable?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20961) Retire NVL implementation
Zoltan Haindrich created HIVE-20961: --- Summary: Retire NVL implementation Key: HIVE-20961 URL: https://issues.apache.org/jira/browse/HIVE-20961 Project: Hive Issue Type: Improvement Reporter: Zoltan Haindrich Right now we have coalesce and nvl implemented separetly; it might be better to remove one of them as they are doing the same. Because Coalesce is in the standard - I think NVL have to go...and became an alias to Coalesce. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
