[jira] [Commented] (HIVE-4943) An explode function that includes the item's position in the array
[ https://issues.apache.org/jira/browse/HIVE-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812298#comment-13812298 ] Niko Stahl commented on HIVE-4943: -- Thanks, Lars! An explode function that includes the item's position in the array -- Key: HIVE-4943 URL: https://issues.apache.org/jira/browse/HIVE-4943 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.11.0 Reporter: Niko Stahl Labels: patch Fix For: 0.13.0 Attachments: HIVE-4943.1.patch, HIVE-4943.2.patch, HIVE-4943.3.patch Original Estimate: 8h Remaining Estimate: 8h A function that explodes an array and includes an output column with the position of each item in the original array. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[ https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5356: -- Attachment: HIVE-5356.2.patch Patch #3 included additional tests, fixes. Move arithmatic UDFs to generic UDF implementations --- Key: HIVE-5356 URL: https://issues.apache.org/jira/browse/HIVE-5356 Project: Hive Issue Type: Task Components: UDF Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.patch Currently, all of the arithmetic operators, such as add/sub/mult/div, are implemented as old-style UDFs and java reflection is used to determine the return type TypeInfos/ObjectInspectors, based on the return type of the evaluate() method chosen for the expression. This works fine for types that don't have type params. Hive decimal type participates in these operations just like int or double. Different from double or int, however, decimal has precision and scale, which cannot be determined by just looking at the return type (decimal) of the UDF evaluate() method, even though the operands have certain precision/scale. With the default of decimal without precision/scale, then (10, 0) will be the type params. This is certainly not desirable. To solve this problem, all of the arithmetic operators would need to be implemented as GenericUDFs, which allow returning ObjectInspector during the initialize() method. The object inspectors returned can carry type params, from which the exact return type can be determined. It's worth mentioning that, for user UDF implemented in non-generic way, if the return type of the chosen evaluate() method is decimal, the return type actually has (10,0) as precision/scale, which might not be desirable. This needs to be documented. This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit the scope of review. The remaining ones will be covered under HIVE-5706. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[ https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5356: -- Status: Patch Available (was: Open) Move arithmatic UDFs to generic UDF implementations --- Key: HIVE-5356 URL: https://issues.apache.org/jira/browse/HIVE-5356 Project: Hive Issue Type: Task Components: UDF Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.patch Currently, all of the arithmetic operators, such as add/sub/mult/div, are implemented as old-style UDFs and java reflection is used to determine the return type TypeInfos/ObjectInspectors, based on the return type of the evaluate() method chosen for the expression. This works fine for types that don't have type params. Hive decimal type participates in these operations just like int or double. Different from double or int, however, decimal has precision and scale, which cannot be determined by just looking at the return type (decimal) of the UDF evaluate() method, even though the operands have certain precision/scale. With the default of decimal without precision/scale, then (10, 0) will be the type params. This is certainly not desirable. To solve this problem, all of the arithmetic operators would need to be implemented as GenericUDFs, which allow returning ObjectInspector during the initialize() method. The object inspectors returned can carry type params, from which the exact return type can be determined. It's worth mentioning that, for user UDF implemented in non-generic way, if the return type of the chosen evaluate() method is decimal, the return type actually has (10,0) as precision/scale, which might not be desirable. This needs to be documented. This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit the scope of review. The remaining ones will be covered under HIVE-5706. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15113: HIVE-5356: Move arithmatic UDFs to generic UDF implementations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15113/ --- (Updated Nov. 3, 2013, 7:26 p.m.) Review request for hive. Bugs: HIVe-5356 https://issues.apache.org/jira/browse/HIVe-5356 Repository: hive-git Description --- Replace plus, minus, and so on 6 old UDFs with generic UDF implementations. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 898b6a5 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java dd8d7dd ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 47d7995 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseNumericOp.java 1e74fce ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPDivide.java da01934 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMinus.java 0996231 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMod.java 0942ac3 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMultiply.java 07c1957 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPPlus.java cfe7b2b ql/src/java/org/apache/hadoop/hive/ql/udf/UDFPosMod.java 0da7eae ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDivide.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPosMod.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorSelectOperator.java 4aeb4e6 ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 1e49a2e ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFOPMod.java 4c2e722 ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFPosMod.java 2a301bc ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPDivide.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMinus.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMod.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMultiply.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPPlus.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFPosMod.java PRE-CREATION ql/src/test/queries/clientpositive/ctas_hadoop20.q 4961b97 ql/src/test/results/clientpositive/ctas_hadoop20.q.out 885e985 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 22e5ec5 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/AbstractPrimitiveObjectInspector.java c19f5aa serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java 0915562 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveDecimalObjectInspector.java a9ca7bd serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java 942a178 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java ad3a063 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java 6b67dac serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/HiveDecimalUtils.java 420509c Diff: https://reviews.apache.org/r/15113/diff/ Testing --- Full set of unit tests is to be run. Old testcases are also migrated. Thanks, Xuefu Zhang
[jira] [Commented] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[ https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812458#comment-13812458 ] Xuefu Zhang commented on HIVE-5356: --- Please note that patch contains that for HIVE-5726, which is required to run the test suite. Move arithmatic UDFs to generic UDF implementations --- Key: HIVE-5356 URL: https://issues.apache.org/jira/browse/HIVE-5356 Project: Hive Issue Type: Task Components: UDF Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.patch Currently, all of the arithmetic operators, such as add/sub/mult/div, are implemented as old-style UDFs and java reflection is used to determine the return type TypeInfos/ObjectInspectors, based on the return type of the evaluate() method chosen for the expression. This works fine for types that don't have type params. Hive decimal type participates in these operations just like int or double. Different from double or int, however, decimal has precision and scale, which cannot be determined by just looking at the return type (decimal) of the UDF evaluate() method, even though the operands have certain precision/scale. With the default of decimal without precision/scale, then (10, 0) will be the type params. This is certainly not desirable. To solve this problem, all of the arithmetic operators would need to be implemented as GenericUDFs, which allow returning ObjectInspector during the initialize() method. The object inspectors returned can carry type params, from which the exact return type can be determined. It's worth mentioning that, for user UDF implemented in non-generic way, if the return type of the chosen evaluate() method is decimal, the return type actually has (10,0) as precision/scale, which might not be desirable. This needs to be documented. This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit the scope of review. The remaining ones will be covered under HIVE-5706. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[
https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812478#comment-13812478
]
Hive QA commented on HIVE-5356:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12611833/HIVE-5356.2.patch
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 4589 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_merge1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_merge2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_round
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_round_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_arithmetic_type
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
org.apache.hadoop.hive.ql.exec.TestFunctionRegistry.testCommonClassComparison
org.apache.hadoop.hive.ql.exec.TestFunctionRegistry.testCommonClassUnionAll
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testArithmeticExpressionVectorization
org.apache.hadoop.hive.ql.parse.TestParse.testParse_cast1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf4
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/124/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/124/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 53 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12611833
Move arithmatic UDFs to generic UDF implementations
---
Key: HIVE-5356
URL: https://issues.apache.org/jira/browse/HIVE-5356
Project: Hive
Issue Type: Task
[jira] [Updated] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[ https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5356: -- Status: Open (was: Patch Available) Move arithmatic UDFs to generic UDF implementations --- Key: HIVE-5356 URL: https://issues.apache.org/jira/browse/HIVE-5356 Project: Hive Issue Type: Task Components: UDF Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.patch Currently, all of the arithmetic operators, such as add/sub/mult/div, are implemented as old-style UDFs and java reflection is used to determine the return type TypeInfos/ObjectInspectors, based on the return type of the evaluate() method chosen for the expression. This works fine for types that don't have type params. Hive decimal type participates in these operations just like int or double. Different from double or int, however, decimal has precision and scale, which cannot be determined by just looking at the return type (decimal) of the UDF evaluate() method, even though the operands have certain precision/scale. With the default of decimal without precision/scale, then (10, 0) will be the type params. This is certainly not desirable. To solve this problem, all of the arithmetic operators would need to be implemented as GenericUDFs, which allow returning ObjectInspector during the initialize() method. The object inspectors returned can carry type params, from which the exact return type can be determined. It's worth mentioning that, for user UDF implemented in non-generic way, if the return type of the chosen evaluate() method is decimal, the return type actually has (10,0) as precision/scale, which might not be desirable. This needs to be documented. This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit the scope of review. The remaining ones will be covered under HIVE-5706. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5733) Publish hive-exec artifact without all the dependencies
Jarek Jarcec Cecho created HIVE-5733:
Summary: Publish hive-exec artifact without all the dependencies
Key: HIVE-5733
URL: https://issues.apache.org/jira/browse/HIVE-5733
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Currently the artifact {{hive-exec}} that is available in
[maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
is shadowing all the dependencies (= the jar contains all Hive's
dependencies). As other projects that are depending on Hive might be use
slightly different version of the dependencies, it can easily happens that
Hive's shadowed version will be used instead which leads to very time consuming
debugging of what is happening (for example SQOOP-1198).
Would it be feasible publish {{hive-exec}} jar that will be build without
shadowing any dependency? For example
[avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
is having classifier nodeps that represents artifact without any
dependencies.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-4523) round() function with specified decimal places not consistent with mysql
[ https://issues.apache.org/jira/browse/HIVE-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812519#comment-13812519 ] Brock Noland commented on HIVE-4523: +1 round() function with specified decimal places not consistent with mysql - Key: HIVE-4523 URL: https://issues.apache.org/jira/browse/HIVE-4523 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.7.1 Reporter: Fred Desing Assignee: Xuefu Zhang Priority: Minor Attachments: HIVE-4523.1.patch, HIVE-4523.2.patch, HIVE-4523.3.patch, HIVE-4523.4.patch, HIVE-4523.5.patch, HIVE-4523.6.patch, HIVE-4523.7.patch, HIVE-4523.8.patch, HIVE-4523.patch // hive hive select round(150.000, 2) from temp limit 1; 150.0 hive select round(150, 2) from temp limit 1; 150.0 // mysql mysql select round(150.000, 2) from DUAL limit 1; round(150.000, 2) 150.00 mysql select round(150, 2) from DUAL limit 1; round(150, 2) 150 http://dev.mysql.com/doc/refman/5.1/en/mathematical-functions.html#function_round -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5354) Decimal precision/scale support in ORC file
[ https://issues.apache.org/jira/browse/HIVE-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812521#comment-13812521 ] Brock Noland commented on HIVE-5354: +1 Decimal precision/scale support in ORC file --- Key: HIVE-5354 URL: https://issues.apache.org/jira/browse/HIVE-5354 Project: Hive Issue Type: Task Components: Serializers/Deserializers Affects Versions: 0.10.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5354.1.patch, HIVE-5354.2.patch, HIVE-5354.3.patch, HIVE-5354.4.patch, HIVE-5354.patch A subtask of HIVE-3976. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5355) JDBC support for decimal precision/scale
[ https://issues.apache.org/jira/browse/HIVE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812522#comment-13812522 ] Brock Noland commented on HIVE-5355: +1 JDBC support for decimal precision/scale Key: HIVE-5355 URL: https://issues.apache.org/jira/browse/HIVE-5355 Project: Hive Issue Type: Task Components: Serializers/Deserializers Affects Versions: 0.10.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5355.1.patch, HIVE-5355.2.patch, HIVE-5355.patch A subtask of HIVE-3976. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Hive compilation error
Hi Mohammad, I can't find the mybucket_*.q files on trunk. Are you using a different branch? 1. Change the code in TestCliDriver.vm to consider this use cases during source generation. I think replacing the call to indexOf() with one to lastIndexOf() will fix the problem. If you want to submit a patch I'll review it. Thanks. Carl
[jira] [Created] (HIVE-5734) Enable merge/move tasks for Tez
Gunther Hagleitner created HIVE-5734: Summary: Enable merge/move tasks for Tez Key: HIVE-5734 URL: https://issues.apache.org/jira/browse/HIVE-5734 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-5719) Remove some overly noisy perflogger statements from Tez codepath
[ https://issues.apache.org/jira/browse/HIVE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5719. -- Resolution: Fixed Committed to branch. Remove some overly noisy perflogger statements from Tez codepath Key: HIVE-5719 URL: https://issues.apache.org/jira/browse/HIVE-5719 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5719.1.patch Some of these fire so often they turn into hotspots in profiler runs... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5734) Enable merge/move tasks for Tez
[ https://issues.apache.org/jira/browse/HIVE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5734: - Attachment: HIVE-5734.1.patch Enable merge/move tasks for Tez --- Key: HIVE-5734 URL: https://issues.apache.org/jira/browse/HIVE-5734 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5734.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[ https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5718: Status: Open (was: Patch Available) Support direct fetch for lateral views, sub queries, etc. - Key: HIVE-5718 URL: https://issues.apache.org/jira/browse/HIVE-5718 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13857.1.patch Extend HIVE-2925 with LV and SubQ. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[ https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-5718: -- Attachment: D13857.2.patch navis updated the revision HIVE-5718 [jira] Support direct fetch for lateral views, sub queries, etc.. 1. Fixed trivial test changes 2. Fixed missing view dependency 3. Use OperatorFactory for creating ListSinkOperator Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D13857 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D13857?vs=42867id=42957#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/explain_logical.q.out ql/src/test/results/clientpositive/lateral_view_noalias.q.out ql/src/test/results/clientpositive/nonmr_fetch.q.out ql/src/test/results/clientpositive/udf_explode.q.out ql/src/test/results/clientpositive/udf_inline.q.out ql/src/test/results/clientpositive/udf_reflect2.q.out ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out ql/src/test/results/clientpositive/udtf_explode.q.out To: JIRA, navis Support direct fetch for lateral views, sub queries, etc. - Key: HIVE-5718 URL: https://issues.apache.org/jira/browse/HIVE-5718 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13857.1.patch, D13857.2.patch Extend HIVE-2925 with LV and SubQ. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[ https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5718: Status: Patch Available (was: Open) Support direct fetch for lateral views, sub queries, etc. - Key: HIVE-5718 URL: https://issues.apache.org/jira/browse/HIVE-5718 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13857.1.patch, D13857.2.patch Extend HIVE-2925 with LV and SubQ. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5723) Add dependency plugin
[ https://issues.apache.org/jira/browse/HIVE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reassigned HIVE-5723: --- Assignee: (was: Navis) Add dependency plugin - Key: HIVE-5723 URL: https://issues.apache.org/jira/browse/HIVE-5723 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Navis Priority: Trivial Attachments: HIVE-5723.1.patch.txt NO PRECOMMIT TESTS For easy gathering of required libraries. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5723) Add dependency plugin
[ https://issues.apache.org/jira/browse/HIVE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812561#comment-13812561 ] Navis commented on HIVE-5723: - Firstly, I'm new to maven (and whatsoever build systems) and what I've really wanted was just a directory containing libraries enough for running hive shell (something like ivy/lib/default) for debugging. It would be really appreciated if someone suggest alternative way to do that. With transitive dependencies=true, the lib directory is flooded with various versions of libraries and hive shell does not work with them. Without transitive, there are some missing libraries like common-http or http-components, which might be added to ql and jdbc pom file but worked good for my purpose. I should have investigated this more but I was almost certain that there would be more simple and cool way to do this. Add dependency plugin - Key: HIVE-5723 URL: https://issues.apache.org/jira/browse/HIVE-5723 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5723.1.patch.txt NO PRECOMMIT TESTS For easy gathering of required libraries. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812572#comment-13812572
]
Hive QA commented on HIVE-5718:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12611852/D13857.2.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/125/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/125/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-125/source-prep.txt
+ [[ true == \t\r\u\e ]]
+ rm -rf ivy maven
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/HiveDecimalUtils.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/AbstractPrimitiveObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveDecimalObjectInspector.java'
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java'
Reverted 'ql/src/test/results/clientpositive/ctas_hadoop20.q.out'
Reverted
'ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java'
Reverted
'ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorSelectOperator.java'
Reverted 'ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFPosMod.java'
Reverted 'ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFOPMod.java'
Reverted 'ql/src/test/queries/clientpositive/ctas_hadoop20.q'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMultiply.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseNumericOp.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFPosMod.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPDivide.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMod.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPPlus.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPMinus.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/common/target
shims/common-secure/target packaging/target hbase-handler/target
testutils/target jdbc/target metastore/target itests/target
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/storage-handlers/hbase/target
hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen
contrib/target service/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPDivide.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMinus.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMultiply.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMod.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFPosMod.java
HIVE_HOME after mavenization
I am working on a patch and refreshed it recently after mavenization. I am trying to run HIVE cli but not sure what to set for HIVE_HOME? Is it documented anywhere? Thanks Prasanth Jayachandran -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-5723) Add dependency plugin
[ https://issues.apache.org/jira/browse/HIVE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812597#comment-13812597 ] Brock Noland commented on HIVE-5723: +1 Add dependency plugin - Key: HIVE-5723 URL: https://issues.apache.org/jira/browse/HIVE-5723 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Navis Priority: Trivial Attachments: HIVE-5723.1.patch.txt NO PRECOMMIT TESTS For easy gathering of required libraries. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5723) Add dependency plugin
[ https://issues.apache.org/jira/browse/HIVE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812596#comment-13812596 ] Brock Noland commented on HIVE-5723: OK understood. I think we can commit this as is for now and then improve upon it later. Add dependency plugin - Key: HIVE-5723 URL: https://issues.apache.org/jira/browse/HIVE-5723 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Navis Priority: Trivial Attachments: HIVE-5723.1.patch.txt NO PRECOMMIT TESTS For easy gathering of required libraries. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5611) Add assembly (i.e.) tar creation to pom
[ https://issues.apache.org/jira/browse/HIVE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812600#comment-13812600 ] Brock Noland commented on HIVE-5611: Thank you! We'll have to do some followup work for hcatalog but this good for now. +1 Add assembly (i.e.) tar creation to pom --- Key: HIVE-5611 URL: https://issues.apache.org/jira/browse/HIVE-5611 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Reporter: Brock Noland Assignee: Szehon Ho Labels: Maven Attachments: HIVE-5611.1.patch, HIVE-5611.2.patch, HIVE-5611.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-5734) Enable merge/move tasks for Tez
[ https://issues.apache.org/jira/browse/HIVE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5734. -- Resolution: Fixed Committed to branch. Enable merge/move tasks for Tez --- Key: HIVE-5734 URL: https://issues.apache.org/jira/browse/HIVE-5734 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5734.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5735) Enable noscan/partialscan on Tez
[ https://issues.apache.org/jira/browse/HIVE-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5735: - Attachment: HIVE-5735.1.patch Enable noscan/partialscan on Tez Key: HIVE-5735 URL: https://issues.apache.org/jira/browse/HIVE-5735 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5735.1.patch analyze ... noscan or partialscan is currently implemented as a physical optimization (GenMRTable1). There's little benefit in porting this over to tez, but we should automatically fall back to MR even if optimize.tez is set. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5735) Enable noscan/partialscan on Tez
Gunther Hagleitner created HIVE-5735: Summary: Enable noscan/partialscan on Tez Key: HIVE-5735 URL: https://issues.apache.org/jira/browse/HIVE-5735 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5735.1.patch analyze ... noscan or partialscan is currently implemented as a physical optimization (GenMRTable1). There's little benefit in porting this over to tez, but we should automatically fall back to MR even if optimize.tez is set. -- This message was sent by Atlassian JIRA (v6.1#6144)
[ANNOUNCE] New Hive Committer - Xuefu Zhang
The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl
Re: [ANNOUNCE] New Hive Committer - Xuefu Zhang
Congratulations! Xuefu, well deserved! Johnny On Sun, Nov 3, 2013 at 8:06 PM, Carl Steinbach cwsteinb...@gmail.comwrote: The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl
[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[ https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-5718: -- Attachment: D13857.3.patch navis updated the revision HIVE-5718 [jira] Support direct fetch for lateral views, sub queries, etc.. Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D13857 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D13857?vs=42957id=42963#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/explain_logical.q.out ql/src/test/results/clientpositive/lateral_view_noalias.q.out ql/src/test/results/clientpositive/nonmr_fetch.q.out ql/src/test/results/clientpositive/udf_explode.q.out ql/src/test/results/clientpositive/udf_inline.q.out ql/src/test/results/clientpositive/udf_reflect2.q.out ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out ql/src/test/results/clientpositive/udtf_explode.q.out To: JIRA, navis Support direct fetch for lateral views, sub queries, etc. - Key: HIVE-5718 URL: https://issues.apache.org/jira/browse/HIVE-5718 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch Extend HIVE-2925 with LV and SubQ. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-5735) Enable noscan/partialscan on Tez
[ https://issues.apache.org/jira/browse/HIVE-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5735. -- Resolution: Fixed Committed to branch. Enable noscan/partialscan on Tez Key: HIVE-5735 URL: https://issues.apache.org/jira/browse/HIVE-5735 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5735.1.patch analyze ... noscan or partialscan is currently implemented as a physical optimization (GenMRTable1). There's little benefit in porting this over to tez, but we should automatically fall back to MR even if optimize.tez is set. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [ANNOUNCE] New Hive Committer - Xuefu Zhang
Bravo Xuefu! -- Lefty On Sun, Nov 3, 2013 at 11:09 PM, Zhang Xiaoyu zhangxiaoyu...@gmail.comwrote: Congratulations! Xuefu, well deserved! Johnny On Sun, Nov 3, 2013 at 8:06 PM, Carl Steinbach cwsteinb...@gmail.comwrote: The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl
Re: [ANNOUNCE] New Hive Committer - Xuefu Zhang
Congrats Xuefu! Gunther. On Sun, Nov 3, 2013 at 8:23 PM, Lefty Leverenz leftylever...@gmail.comwrote: Bravo Xuefu! -- Lefty On Sun, Nov 3, 2013 at 11:09 PM, Zhang Xiaoyu zhangxiaoyu...@gmail.com wrote: Congratulations! Xuefu, well deserved! Johnny On Sun, Nov 3, 2013 at 8:06 PM, Carl Steinbach cwsteinb...@gmail.com wrote: The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-5657) TopN produces incorrect results with count(distinct)
[ https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812617#comment-13812617 ] Navis commented on HIVE-5657: - HIVE-5503 is committed and I cannot rebase on that. Was it necessary to refactor the codes which is not yet confirmed solid and has on-going issues on it? I appreciate progresses made by members of the company. But not like this. TopN produces incorrect results with count(distinct) Key: HIVE-5657 URL: https://issues.apache.org/jira/browse/HIVE-5657 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Navis Priority: Critical Attachments: D13797.1.patch, D13797.2.patch, HIVE-5657.1.patch.txt, example.patch Attached patch illustrates the problem. limit_pushdown test has various other cases of aggregations and distincts, incl. count-distinct, that work correctly (that said, src dataset is bad for testing these things because every count, for example, produces one record only), so something must be special about this. I am not very familiar with distinct- code and these nuances; if someone knows a quick fix feel free to take this, otherwise I will probably start looking next week. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [ANNOUNCE] New Hive Committer - Xuefu Zhang
Congrats Xuefu! Thanks, Hari On Nov 3, 2013, at 8:28 PM, Gunther Hagleitner ghagleit...@hortonworks.com wrote: Congrats Xuefu! Gunther. On Sun, Nov 3, 2013 at 8:23 PM, Lefty Leverenz leftylever...@gmail.comwrote: Bravo Xuefu! -- Lefty On Sun, Nov 3, 2013 at 11:09 PM, Zhang Xiaoyu zhangxiaoyu...@gmail.com wrote: Congratulations! Xuefu, well deserved! Johnny On Sun, Nov 3, 2013 at 8:06 PM, Carl Steinbach cwsteinb...@gmail.com wrote: The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HIVE-5736) Fix assertion in Operator.java
Gunther Hagleitner created HIVE-5736: Summary: Fix assertion in Operator.java Key: HIVE-5736 URL: https://issues.apache.org/jira/browse/HIVE-5736 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5736) Fix assertion in Operator.java
[ https://issues.apache.org/jira/browse/HIVE-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5736: - Attachment: HIVE-5736.1.patch Fix assertion in Operator.java -- Key: HIVE-5736 URL: https://issues.apache.org/jira/browse/HIVE-5736 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5736.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5736) Fix assertion in Operator.java
[ https://issues.apache.org/jira/browse/HIVE-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812627#comment-13812627 ] Gunther Hagleitner commented on HIVE-5736: -- the cached operator array in operator.java can be null in two cases if the childoperator list is null or if it's empty. Fix assertion in Operator.java -- Key: HIVE-5736 URL: https://issues.apache.org/jira/browse/HIVE-5736 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5736.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-5043) Re-enable analyze command for Tez
[ https://issues.apache.org/jira/browse/HIVE-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5043. -- Resolution: Fixed Already fixed. Re-enable analyze command for Tez - Key: HIVE-5043 URL: https://issues.apache.org/jira/browse/HIVE-5043 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Fix For: tez-branch The analyze command is setup in the physical optimizer (table scan + follow up task). This doesn't happen for Tez yet. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-5736) Fix assertion in Operator.java
[ https://issues.apache.org/jira/browse/HIVE-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5736. -- Resolution: Fixed Committed to branch. Fix assertion in Operator.java -- Key: HIVE-5736 URL: https://issues.apache.org/jira/browse/HIVE-5736 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5736.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812639#comment-13812639
]
Hive QA commented on HIVE-5718:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12611862/D13857.3.patch
{color:green}SUCCESS:{color} +1 4547 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/126/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/126/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12611862
Support direct fetch for lateral views, sub queries, etc.
-
Key: HIVE-5718
URL: https://issues.apache.org/jira/browse/HIVE-5718
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch
Extend HIVE-2925 with LV and SubQ.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5711) m2eclipse does not work and eclipse:eclipse requires a manual fix
[ https://issues.apache.org/jira/browse/HIVE-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5711: - Attachment: HIVE-5711.1.patch m2eclipse does not work and eclipse:eclipse requires a manual fix - Key: HIVE-5711 URL: https://issues.apache.org/jira/browse/HIVE-5711 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Reporter: Brock Noland Priority: Critical Labels: Eclipse, Maven Attachments: HIVE-5711.1.patch As discussed [here|https://issues.apache.org/jira/browse/HIVE-5610?focusedCommentId=13809855page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13809855] m2eclipse doesn't work with the new maven change. Additionally as discussed [here|https://issues.apache.org/jira/browse/HIVE-5610?focusedCommentId=13809910page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13809910] eclipse:eclipse requires removing the hive-shims reference from all classpath files. We should figure out how to resolve this. -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 15203: HIVE-5711: m2eclipse does not work and eclipse:eclipse requires a manual fix
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15203/ --- Review request for hive and Brock Noland. Bugs: HIVE-5711 https://issues.apache.org/jira/browse/HIVE-5711 Repository: hive-git Description --- I verified that this patch works on Eclipse 4.3. YMMV with other versions of Eclipse. Diffs - hcatalog/webhcat/svr/pom.xml 30c987f metastore/pom.xml 7cf3d59 pom.xml 0f115c4 shims/assembly/pom.xml PRE-CREATION shims/assembly/src/assemble/uberjar.xml PRE-CREATION shims/pom.xml e41a190 shims/src/assembly/uberjar.xml 67ce50b Diff: https://reviews.apache.org/r/15203/diff/ Testing --- Thanks, Carl Steinbach
[jira] [Commented] (HIVE-5711) m2eclipse does not work and eclipse:eclipse requires a manual fix
[ https://issues.apache.org/jira/browse/HIVE-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812650#comment-13812650 ] Carl Steinbach commented on HIVE-5711: -- Review request: https://reviews.apache.org/r/15203/ m2eclipse does not work and eclipse:eclipse requires a manual fix - Key: HIVE-5711 URL: https://issues.apache.org/jira/browse/HIVE-5711 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Reporter: Brock Noland Priority: Critical Labels: Eclipse, Maven Attachments: HIVE-5711.1.patch As discussed [here|https://issues.apache.org/jira/browse/HIVE-5610?focusedCommentId=13809855page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13809855] m2eclipse doesn't work with the new maven change. Additionally as discussed [here|https://issues.apache.org/jira/browse/HIVE-5610?focusedCommentId=13809910page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13809910] eclipse:eclipse requires removing the hive-shims reference from all classpath files. We should figure out how to resolve this. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Hive compilation error
Thanks Carl for the reply. Yes these were my own .q files. Since there are no such current files, we can skip it for the time being. Need to make sure no future q filenames would contain multiple dots. Regards, Mohammad On Sunday, November 3, 2013 4:59 PM, Carl Steinbach cwsteinb...@gmail.com wrote: Hi Mohammad, I can't find the mybucket_*.q files on trunk. Are you using a different branch? 1. Change the code in TestCliDriver.vm to consider this use cases during source generation. I think replacing the call to indexOf() with one to lastIndexOf() will fix the problem. If you want to submit a patch I'll review it. Thanks. Carl
Re: [ANNOUNCE] New Hive Committer - Xuefu Zhang
Congrats Xuefu! --Mohammad On Sunday, November 3, 2013 9:07 PM, hsubramani...@hortonworks.com hsubramani...@hortonworks.com wrote: Congrats Xuefu! Thanks, Hari On Nov 3, 2013, at 8:28 PM, Gunther Hagleitner ghagleit...@hortonworks.com wrote: Congrats Xuefu! Gunther. On Sun, Nov 3, 2013 at 8:23 PM, Lefty Leverenz leftylever...@gmail.comwrote: Bravo Xuefu! -- Lefty On Sun, Nov 3, 2013 at 11:09 PM, Zhang Xiaoyu zhangxiaoyu...@gmail.com wrote: Congratulations! Xuefu, well deserved! Johnny On Sun, Nov 3, 2013 at 8:06 PM, Carl Steinbach cwsteinb...@gmail.com wrote: The Apache Hive PMC has voted to make Xuefu Zhang a committer on the Apache Hive project. Please join me in congratulating Xuefu! Thanks. Carl -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HIVE-5737) Provide StructObjectInspector for UDTFs rather than ObjectInspect[]
Navis created HIVE-5737:
---
Summary: Provide StructObjectInspector for UDTFs rather than
ObjectInspect[]
Key: HIVE-5737
URL: https://issues.apache.org/jira/browse/HIVE-5737
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Navis
Assignee: Navis
Priority: Trivial
In UDTF, column names can be useful sometimes. For example, complex function
with many optional parameters something like,
xml_explode('\t' as field, '\n' as line, '=' as mapkey, ':' as items, input)
Without column name, it's not easy to discern each parameter is for what.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5737) Provide StructObjectInspector for UDTFs rather than ObjectInspect[]
[
https://issues.apache.org/jira/browse/HIVE-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-5737:
Status: Patch Available (was: Open)
Provide StructObjectInspector for UDTFs rather than ObjectInspect[]
---
Key: HIVE-5737
URL: https://issues.apache.org/jira/browse/HIVE-5737
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5737.1.patch.txt
In UDTF, column names can be useful sometimes. For example, complex function
with many optional parameters something like,
xml_explode('\t' as field, '\n' as line, '=' as mapkey, ':' as items, input)
Without column name, it's not easy to discern each parameter is for what.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5737) Provide StructObjectInspector for UDTFs rather than ObjectInspect[]
[
https://issues.apache.org/jira/browse/HIVE-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-5737:
Attachment: HIVE-5737.1.patch.txt
Provide StructObjectInspector for UDTFs rather than ObjectInspect[]
---
Key: HIVE-5737
URL: https://issues.apache.org/jira/browse/HIVE-5737
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5737.1.patch.txt
In UDTF, column names can be useful sometimes. For example, complex function
with many optional parameters something like,
xml_explode('\t' as field, '\n' as line, '=' as mapkey, ':' as items, input)
Without column name, it's not easy to discern each parameter is for what.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-4880) Rearrange explain order of stages simpler
[
https://issues.apache.org/jira/browse/HIVE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4880:
--
Attachment: D11733.4.patch
navis updated the revision HIVE-4880 [jira] Rearrange explain order of stages
simpler.
Rebased to trunk fixed test result (was partially merged with other
patch(HIVE-3925))
Reviewers: hagleitn, JIRA
REVISION DETAIL
https://reviews.facebook.net/D11733
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D11733?vs=42009id=42975#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/common/ObjectPair.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
ql/src/test/queries/clientpositive/explain_rearrange.q
ql/src/test/results/clientpositive/explain_rearrange.q.out
To: JIRA, hagleitn, navis
Cc: brock
Rearrange explain order of stages simpler
-
Key: HIVE-4880
URL: https://issues.apache.org/jira/browse/HIVE-4880
Project: Hive
Issue Type: Improvement
Components: Diagnosability
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: D11733.2.patch, D11733.3.patch, D11733.4.patch,
HIVE-4880.D11733.1.patch
Current explain order of stages is not best form to read (and to debug)
{noformat}
Before:
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
Stage-4
Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
Stage-9 depends on stages: Stage-0
Stage-2 depends on stages: Stage-9
Stage-3
Stage-5
Stage-6 depends on stages: Stage-5
After:
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1 , consists of Stage-3, Stage-4, Stage-5
Stage-3
Stage-4
Stage-5
Stage-6 depends on stages: Stage-5
Stage-7 depends on stages: Stage-3, Stage-4, Stage-6
Stage-8 depends on stages: Stage-7
Stage-9 depends on stages: Stage-8
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-3925) dependencies of fetch task are not shown by explain
[ https://issues.apache.org/jira/browse/HIVE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812662#comment-13812662 ] Navis commented on HIVE-3925: - [~ashutoshc] I've missed your comment. I'll check that. dependencies of fetch task are not shown by explain --- Key: HIVE-3925 URL: https://issues.apache.org/jira/browse/HIVE-3925 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Navis Attachments: HIVE-3925.4.patch.txt, HIVE-3925.D8577.1.patch, HIVE-3925.D8577.2.patch, HIVE-3925.D8577.3.patch A simple query like: hive explain select * from src order by key; OK ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEASC (TOK_TABLE_OR_COL key) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage Stage: Stage-0 Fetch Operator limit: -1 Stage-0 is not a root stage and depends on stage-1. -- This message was sent by Atlassian JIRA (v6.1#6144)
