[jira] [Updated] (HIVE-4924) Support query timeout for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4924:
Status: Open (was: Patch Available)
Support query timeout for jdbc2
---
Key: HIVE-4924
URL: https://issues.apache.org/jira/browse/HIVE-4924
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
support
{code}
Statement.setQueryTimeout(int timeout)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4924) Support query timeout for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4924:
Status: Patch Available (was: Open)
The failed test seemed flaky which is described in HIVE-4851.
Support query timeout for jdbc2
---
Key: HIVE-4924
URL: https://issues.apache.org/jira/browse/HIVE-4924
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
support
{code}
Statement.setQueryTimeout(int timeout)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4123) The RLE encoding for ORC can be improved
[ https://issues.apache.org/jira/browse/HIVE-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737891#comment-13737891 ] Hudson commented on HIVE-4123: -- ABORTED: Integrated in Hive-trunk-hadoop2 #354 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/354/]) HIVE-4123 Improved ORC integer RLE version 2. (Prasanth Jayachandran via omalley) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513155) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerWriter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java.orig * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestNewIntegerEncoding.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out The RLE encoding for ORC can be improved Key: HIVE-4123 URL: https://issues.apache.org/jira/browse/HIVE-4123 Project: Hive Issue Type: New Feature Components: File Formats Affects Versions: 0.12.0 Reporter: Owen O'Malley Assignee: Prasanth J Labels: orcfile Fix For: 0.12.0 Attachments: HIVE-4123.1.git.patch.txt, HIVE-4123.2.git.patch.txt, HIVE-4123.3.patch.txt, HIVE-4123.4.patch.txt, HIVE-4123.5.txt, HIVE-4123.6.txt, HIVE-4123.7.txt, HIVE-4123-8.patch, HIVE-4123.8.txt, HIVE-4123.8.txt, HIVE-4123.patch.txt, ORC-Compression-Ratio-Comparison.xlsx The run length encoding of integers can be improved: * tighter bit packing * allow delta encoding * allow longer runs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4246) Implement predicate pushdown for ORC
[
https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737892#comment-13737892
]
Hive QA commented on HIVE-4246:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597610/HIVE-4246.D11415.3.patch
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 2868 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_cast1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join8
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input_testxpath
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input_part1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_subq
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf_when
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input_testsequencefile
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input_testxpath2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample3
org.apache.hadoop.hive.ql.io.orc.TestRecordReaderImpl.testPartialPlan
org.apache.hadoop.hive.ql.parse.TestParse.testParse_case_sensitivity
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input9
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby4
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf_case
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join3
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/411/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/411/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 46 tests failed
{noformat}
This message is automatically generated.
Implement predicate pushdown for ORC
Key: HIVE-4246
URL: https://issues.apache.org/jira/browse/HIVE-4246
Project: Hive
Issue Type: New Feature
Components: File Formats
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch,
HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch
By using the push down predicates from the table scan operator, ORC can skip
over 10,000 rows at a time that won't satisfy the predicate. This will help a
lot, especially if the file is sorted by the column that is used in the
predicate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-1708) make hive history file configurable
[ https://issues.apache.org/jira/browse/HIVE-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-1708. - Resolution: Duplicate make hive history file configurable --- Key: HIVE-1708 URL: https://issues.apache.org/jira/browse/HIVE-1708 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Currentlly, it is derived from System.getProperty(user.home)/.hivehistory; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3779) An empty value to hive.logquery.location can't disable the creation of hive history log files
[ https://issues.apache.org/jira/browse/HIVE-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-3779. - Resolution: Fixed An empty value to hive.logquery.location can't disable the creation of hive history log files - Key: HIVE-3779 URL: https://issues.apache.org/jira/browse/HIVE-3779 Project: Hive Issue Type: Bug Components: Documentation Affects Versions: 0.9.0 Reporter: Bing Li Priority: Minor In AdminManual Configuration (https://cwiki.apache.org/Hive/adminmanual-configuration.html), the description of hive.querylog.location mentioned that if the variable set to empty string structured log will not be created. But it fails with the following setting, property namehive.querylog.location/name value/value /property It seems that it can NOT get an empty value from HiveConf.ConfVars.HIVEHISTORYFILELOC, but the default value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5022) Decimal Arithmetic generates NULL value
[ https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737917#comment-13737917 ] Teddy Choi commented on HIVE-5022: -- The result of division has the maximum precision. The following multiplication makes a result with a greater precision than the maximum precision, so it causes an error. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27838462#LanguageManualTypes-FloatingPointTypes provides more information. Please use ROUND() to reduce precisions or use DOUBLE to be tolerant of precision issues. Decimal Arithmetic generates NULL value --- Key: HIVE-5022 URL: https://issues.apache.org/jira/browse/HIVE-5022 Project: Hive Issue Type: Bug Components: Types Affects Versions: 0.11.0 Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107 Reporter: Kevin Soo Hoo When a decimal division is the first operation, the quotient cannot be multiplied in a subsequent calculation. Instead, a NULL is returned. The following yield NULL results: select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as decimal) from tablename limit 1; select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as decimal) from tablename limit 1; If we move the multiplication operation to be first, then it will successfully calculate the result. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-5022) Decimal Arithmetic generates NULL value
[ https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi reassigned HIVE-5022: Assignee: Teddy Choi Decimal Arithmetic generates NULL value --- Key: HIVE-5022 URL: https://issues.apache.org/jira/browse/HIVE-5022 Project: Hive Issue Type: Bug Components: Types Affects Versions: 0.11.0 Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107 Reporter: Kevin Soo Hoo Assignee: Teddy Choi When a decimal division is the first operation, the quotient cannot be multiplied in a subsequent calculation. Instead, a NULL is returned. The following yield NULL results: select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as decimal) from tablename limit 1; select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as decimal) from tablename limit 1; If we move the multiplication operation to be first, then it will successfully calculate the result. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737920#comment-13737920 ] Hudson commented on HIVE-4863: -- FAILURE: Integrated in Hive-trunk-hadoop2 #355 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/355/]) HIVE-4863: Fix parallel order by on hadoop2 (Edward Capriolo via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513351) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-4863.1.patch.txt, HIVE-4863.2.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737922#comment-13737922 ] Thejas M Nair commented on HIVE-4513: - [~thiruvel] Thanks for the feedback and pointers to the duplicate jiras. Yes, I think it is better to address thread safety issue in another jira, as it is not a regression introduced by this patch, the issue was there when the function was in HiveHistory.java (but I should have noticed the bug!) disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5022) Decimal Arithmetic generates NULL value
[ https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737923#comment-13737923 ] Teddy Choi commented on HIVE-5022: -- I don't know how other database handles this issue. If it works differently from others, please tell me. It may be a bug. Decimal Arithmetic generates NULL value --- Key: HIVE-5022 URL: https://issues.apache.org/jira/browse/HIVE-5022 Project: Hive Issue Type: Bug Components: Types Affects Versions: 0.11.0 Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107 Reporter: Kevin Soo Hoo Assignee: Teddy Choi When a decimal division is the first operation, the quotient cannot be multiplied in a subsequent calculation. Instead, a NULL is returned. The following yield NULL results: select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as decimal) from tablename limit 1; select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as decimal) from tablename limit 1; If we move the multiplication operation to be first, then it will successfully calculate the result. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5071) Address thread safety issues with HiveHistoryUtil
[ https://issues.apache.org/jira/browse/HIVE-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737926#comment-13737926 ] Thejas M Nair commented on HIVE-5071: - From HIVE-4513 - parseLine() method is not thread-safe. It uses parseBuffer which could be modified by multiple threads. Looks like there is no reason why parseBuffer should be static, it be a variable local to the function. Address thread safety issues with HiveHistoryUtil - Key: HIVE-5071 URL: https://issues.apache.org/jira/browse/HIVE-5071 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Thiruvel Thirumoolan Priority: Minor Fix For: 0.12.0 HiveHistoryUtil.parseLine() is not thread safe, it could be used by multiple clients of HWA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737927#comment-13737927 ] Navis commented on HIVE-4851: - TestNotificationListener.testAMQListener seemed checking arrived messages too fast (on busy, fast machine). Just pre-pending Thread.sleep(100) in tearDown() might fix that. Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat * TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5073) Fix problem with multiple root tasks in tez
Gunther Hagleitner created HIVE-5073: Summary: Fix problem with multiple root tasks in tez Key: HIVE-5073 URL: https://issues.apache.org/jira/browse/HIVE-5073 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Input splits are always created in the same directory. That won't work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5073) Fix problem with multiple root tasks in tez
[ https://issues.apache.org/jira/browse/HIVE-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5073: - Description: Input splits are always created in the same directory. That won't work. NO PRECOMMIT TESTS (this is wip for the tez branch) was:Input splits are always created in the same directory. That won't work. Fix problem with multiple root tasks in tez --- Key: HIVE-5073 URL: https://issues.apache.org/jira/browse/HIVE-5073 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Input splits are always created in the same directory. That won't work. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737951#comment-13737951 ] Gunther Hagleitner commented on HIVE-4246: -- [~owen.omalley] looks like you need to update the golden files for most of the TestParse tests. The neededColumns you added to the TableScan show up in all the serialized queries now. Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5074) Additional information for mini-mr tests
Navis created HIVE-5074: --- Summary: Additional information for mini-mr tests Key: HIVE-5074 URL: https://issues.apache.org/jira/browse/HIVE-5074 Project: Hive Issue Type: Test Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Flaky tests of Test(Negative)MinimrCliDriver is hard to track. Test results for diff error and exception trace for unexpected exception would be helpful for debugging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5022) Decimal Arithmetic generates NULL value
[
https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737956#comment-13737956
]
Teddy Choi commented on HIVE-5022:
--
To avoid this problem, I changed HiveDecimal#multiply as following. It produced
a non-null result, with loss of precision. I'm not sure whether it reflects the
original purpose or not.
{code}
public HiveDecimal multiply(HiveDecimal dec) {
return new HiveDecimal(bd.multiply(dec.bd), true);
}
{code}
Decimal Arithmetic generates NULL value
---
Key: HIVE-5022
URL: https://issues.apache.org/jira/browse/HIVE-5022
Project: Hive
Issue Type: Bug
Components: Types
Affects Versions: 0.11.0
Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107
Reporter: Kevin Soo Hoo
Assignee: Teddy Choi
When a decimal division is the first operation, the quotient cannot be
multiplied in a subsequent calculation. Instead, a NULL is returned.
The following yield NULL results:
select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as
decimal) from tablename limit 1;
select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as
decimal) from tablename limit 1;
If we move the multiplication operation to be first, then it will
successfully calculate the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5022) Decimal Arithmetic generates NULL value
[ https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-5022: - Attachment: HIVE-5022.1.patch.txt Decimal Arithmetic generates NULL value --- Key: HIVE-5022 URL: https://issues.apache.org/jira/browse/HIVE-5022 Project: Hive Issue Type: Bug Components: Types Affects Versions: 0.11.0 Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107 Reporter: Kevin Soo Hoo Assignee: Teddy Choi Attachments: HIVE-5022.1.patch.txt When a decimal division is the first operation, the quotient cannot be multiplied in a subsequent calculation. Instead, a NULL is returned. The following yield NULL results: select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as decimal) from tablename limit 1; select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as decimal) from tablename limit 1; If we move the multiplication operation to be first, then it will successfully calculate the result. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4943) An explode function that includes the item's position in the array
[ https://issues.apache.org/jira/browse/HIVE-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737968#comment-13737968 ] Niko Stahl commented on HIVE-4943: -- Please review. Thanks! An explode function that includes the item's position in the array -- Key: HIVE-4943 URL: https://issues.apache.org/jira/browse/HIVE-4943 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.11.0 Reporter: Niko Stahl Labels: patch Fix For: 0.11.0 Original Estimate: 8h Remaining Estimate: 8h A function that explodes an array and includes an output column with the position of each item in the original array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[
https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737972#comment-13737972
]
Hive QA commented on HIVE-4513:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597289/HIVE-4513.6.patch
{color:green}SUCCESS:{color} +1 2850 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/412/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/412/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
disable hivehistory logs by default
---
Key: HIVE-4513
URL: https://issues.apache.org/jira/browse/HIVE-4513
Project: Hive
Issue Type: Bug
Components: Configuration, Logging
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch,
HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch
HiveHistory log files (hive_job_log_hive_*.txt files) store information about
hive query such as query string, plan , counters and MR job progress
information.
There is no mechanism to delete these files and as a result they get
accumulated over time, using up lot of disk space.
I don't think this is used by most people, so I think it would better to turn
this off by default. Jobtracker logs already capture most of this
information, though it is not as structured as history logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4100) Improve regex_replace UDF to allow non-ascii characters
[
https://issues.apache.org/jira/browse/HIVE-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737987#comment-13737987
]
Teddy Choi commented on HIVE-4100:
--
If we allow \u form, then \UDFBa in hive\ql\udf\UDFBack.java may be
parsed as a unicode character. It is ambiguous.
How about this way?
{code}REGEXP_REPLACE(some_column, [^\\u-\\u],
UNESCAPE_UNICODE(\ufffd)){code}
In this example, UNESCAPE_UNICODE is just a random name.
Improve regex_replace UDF to allow non-ascii characters
---
Key: HIVE-4100
URL: https://issues.apache.org/jira/browse/HIVE-4100
Project: Hive
Issue Type: Improvement
Components: UDF
Affects Versions: 0.10.0
Reporter: Mark Grover
Assignee: Mark Grover
There have a been a few email threads on the user mailing list regarding
regex_replace UDF not supporting non-ASCII characters. We should validate
that and improve the UDF to allow it. Translate UDF will be a good reference
since it does that by using code points instead of characters
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4601) WebHCat, Templeton need to support proxy users
[
https://issues.apache.org/jira/browse/HIVE-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738053#comment-13738053
]
Hive QA commented on HIVE-4601:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597627/HIVE-4601.patch
{color:green}SUCCESS:{color} +1 2848 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/413/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
WebHCat, Templeton need to support proxy users
--
Key: HIVE-4601
URL: https://issues.apache.org/jira/browse/HIVE-4601
Project: Hive
Issue Type: Improvement
Components: HCatalog
Affects Versions: 0.11.0
Reporter: Dilli Arumugam
Assignee: Eugene Koifman
Labels: gateay, proxy, templeton
Fix For: 0.12.0
Attachments: HIVE-4601.patch
We have a use case where a Gateway would provide unified and controlled
access to secure hadoop cluster.
The Gateway itself would authenticate to secure WebHDFS, Oozie and Templeton
with SPNego.
The Gateway would authenticate the end user with http basic and would assert
the end user identity as douser argument in the calls to downstream WebHDFS,
Oozie and Templeton.
This works fine with WebHDFS and Oozie. But, does not work for Templeton as
Templeton does not support proxy users.
Hence, request to add this improvement to Templeton.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5069) Tests on list bucketing are failing again in hadoop2
[
https://issues.apache.org/jira/browse/HIVE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738109#comment-13738109
]
Hive QA commented on HIVE-5069:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597631/HIVE-5069.D12201.1.patch
{color:green}SUCCESS:{color} +1 2848 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/414/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Tests on list bucketing are failing again in hadoop2
Key: HIVE-5069
URL: https://issues.apache.org/jira/browse/HIVE-5069
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-5069.D12201.1.patch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4513: --- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Thejas! disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5069) Tests on list bucketing are failing again in hadoop2
[ https://issues.apache.org/jira/browse/HIVE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738117#comment-13738117 ] Ashutosh Chauhan commented on HIVE-5069: [~sershe] Do you want to take a look at this one? Tests on list bucketing are failing again in hadoop2 Key: HIVE-5069 URL: https://issues.apache.org/jira/browse/HIVE-5069 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5069.D12201.1.patch org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4899) Hive returns non-meanful error message for ill-formed fs.default.name
[
https://issues.apache.org/jira/browse/HIVE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738118#comment-13738118
]
Hudson commented on HIVE-4899:
--
SUCCESS: Integrated in Hive-trunk-h0.21 #2264 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2264/])
HIVE-4899 - Hive returns non-meanful error message for ill-formed
fs.default.name (Xuefu Zhang, Reviewed By: Ashutosh Chauhan) (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513229)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/test/results/clientnegative/fs_default_name1.q.out
* /hive/trunk/ql/src/test/results/clientnegative/fs_default_name2.q.out
Hive returns non-meanful error message for ill-formed fs.default.name
-
Key: HIVE-4899
URL: https://issues.apache.org/jira/browse/HIVE-4899
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.10.0, 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Priority: Minor
Fix For: 0.12.0
Attachments: HIVE-4899.patch
For query in test case fs_default_name1.q:
{code}
set fs.default.name='http://www.example.com;
show tables;
{code}
The following error message is returned:
{code}
FAILED: IllegalArgumentException null
{code}
The message is not very meaningful, and has null in it.
It would be better if we can provide detailed error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5051) StorageBasedAuthorizationProvider masks lower level exception with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5051:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Eugene!
StorageBasedAuthorizationProvider masks lower level exception with
IllegalStateException
Key: HIVE-5051
URL: https://issues.apache.org/jira/browse/HIVE-5051
Project: Hive
Issue Type: Bug
Components: Authorization
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: HIVE-5051.patch
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
has the following 2 methods
private HiveException hiveException(Exception e) {
HiveException ex = new HiveException(e);
ex.initCause(e);
return ex;
}
private AuthorizationException authorizationException(Exception e) {
AuthorizationException ex = new AuthorizationException(e);
ex.initCause(e);
return ex;
}
both, when called, will cause
throw new IllegalStateException(Can't overwrite cause); to be thrown and
the original Exception is lost.
See
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/Throwable.java#Throwable.initCause%28java.lang.Throwable%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5023) Hive get wrong result when partition has the same path but different schema or authority
[
https://issues.apache.org/jira/browse/HIVE-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5023:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Shuaishuai!
Hive get wrong result when partition has the same path but different schema
or authority
Key: HIVE-5023
URL: https://issues.apache.org/jira/browse/HIVE-5023
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Fix For: 0.12.0
Attachments: HIVE-5023.1.patch, HIVE-5023.2.patch
Hive does not differentiate scheme and authority in file uris which cause
wrong result when partition has the same path but different schema or
authority. Here is a simple repro
partition file path:
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text1.txt
with content 2013-08-05 00:00:00
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text2.txt
with content 2013-08-05 00:00:20
{noformat}
CREATE EXTERNAL TABLE IF NOT EXISTS T1 (t STRING) PARTITIONED BY (ProcessDate
STRING, Hour STRING, ClusterName STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED by '\t' STORED AS TEXTFILE;
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='CLusterA');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterA') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='ClusterB');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterB') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
{noformat}
the expect output of the hive query
{noformat}
SELECT ClusterName, t FROM T1 WHERE ProcessDate=’2013-08-05’ AND Hour=’00’;
{noformat}
should be
{noformat}
ClusterA2013-08-05 00:00:00
ClusterB2013-08-05 00:00:20
{noformat}
However it is
{noformat}
ClusterA2013-08-05 00:00:00
ClusterA2013-08-05 00:00:20
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev
[ https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738133#comment-13738133 ] Ashutosh Chauhan commented on HIVE-4690: I am really finding it hard to believe that mapred.max.split.size is not effective on 20S. I have used this config to configure number of splits on 20S many times. Though your tests indeed indicate otherwise. Do you by any chance happen to know hadoop jira which has introduced this change in behavior? stats_partscan_1.q makes different result with different hadhoop.mr.rev Key: HIVE-4690 URL: https://issues.apache.org/jira/browse/HIVE-4690 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-4690.D11163.1.patch stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, which can be different with different hadoop.mr.rev. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[ https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738136#comment-13738136 ] Ashutosh Chauhan commented on HIVE-4985: +1 Looks like HIVE QA didnt kick in. If you have run the tests, can you post the results here? refactor/clean up partition name pruning to be usable inside metastore server -- Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch, HIVE-4985.D11961.3.patch, HIVE-4985.D11961.4.patch, HIVE-4985.D11961.5.patch Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3189) cast ( string type as bigint) returning null values
[ https://issues.apache.org/jira/browse/HIVE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738153#comment-13738153 ] Ashutosh Chauhan commented on HIVE-3189: +1 cast ( string type as bigint) returning null values - Key: HIVE-3189 URL: https://issues.apache.org/jira/browse/HIVE-3189 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Attachments: Hive-3189.patch.txt select rnum, c1, cast(c1 as bigint) from cert.tsdchar tsdchar where rnum in (0,1,2) create table if not exists CERT.TSDCHAR ( RNUM int , C1 string) row format sequencefile rnum c1 _c2 0 -1 null 1 0 null 2 10 null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4885) Alternative object serialization for execution plan in hive testing
[
https://issues.apache.org/jira/browse/HIVE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4885:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Xuefu!
Alternative object serialization for execution plan in hive testing
Key: HIVE-4885
URL: https://issues.apache.org/jira/browse/HIVE-4885
Project: Hive
Issue Type: Improvement
Components: CLI
Affects Versions: 0.10.0, 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.12.0
Attachments: HIVE-4885.patch
Currently there are a lot of test cases involving in comparing execution
plan, such as those in TestParse suite. XmlEncoder is used to serialize the
generated plan by hive, and store it in the file for file diff comparison.
However, XmlEncoder is tied with Java compiler, whose implementation may
change from version to version. Thus, upgrade the compiler can generate a lot
of fake test failures. The following is an example of diff generated when
running hive with JDK7:
{code}
Begin query: case_sensitivity.q
diff -a
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.out
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/parse/case_sensitivity.q.out
diff -a -b
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.xml
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/plan/case_sensitivity.q.xml
3c3
object class=org.apache.hadoop.hive.ql.exec.MapRedTask id=MapRedTask0
---
object id=MapRedTask0
class=org.apache.hadoop.hive.ql.exec.MapRedTask
12c12
object class=java.util.ArrayList id=ArrayList0
---
object id=ArrayList0 class=java.util.ArrayList
14c14
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask0
---
object id=MoveTask0
class=org.apache.hadoop.hive.ql.exec.MoveTask
18c18
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask1
---
object id=MoveTask1
class=org.apache.hadoop.hive.ql.exec.MoveTask
22c22
object class=org.apache.hadoop.hive.ql.exec.StatsTask
id=StatsTask0
---
object id=StatsTask0
class=org.apache.hadoop.hive.ql.exec.StatsTask
60c60
object class=org.apache.hadoop.hive.ql.exec.MapRedTask
id=MapRedTask1
---
object id=MapRedTask1
class=org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
As it can be seen, the only difference is the order of the attributes in the
serialized XML doc, yet it brings 50+ test failures in Hive.
We need to have a better plan comparison, or object serialization to improve
the situation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4003: --- Status: Patch Available (was: Open) Marking Patch Available to get a test run. NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - Key: HIVE-4003 URL: https://issues.apache.org/jira/browse/HIVE-4003 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Thomas Adam Assignee: Mark Grover Attachments: HIVE-4003.patch Utilities.java seems to be throwing a NPE. Change contributed by Thomas Adam. Reference: https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4940) udaf_percentile_approx.q is not deterministic
[ https://issues.apache.org/jira/browse/HIVE-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738176#comment-13738176 ] Phabricator commented on HIVE-4940: --- brock has commented on the revision HIVE-4940 [jira] udaf_percentile_approx.q is not deterministic. Thanks for doing this work Navis! I am just curious as to what the difference was between the two versions test outputs? INLINE COMMENTS ql/src/test/queries/clientpositive/udaf_percentile_approx_20.q:42 Should these queries have an order by to ensure they are deterministic in future revs of hadoop and JVM versions? REVISION DETAIL https://reviews.facebook.net/D12189 To: JIRA, navis Cc: brock udaf_percentile_approx.q is not deterministic - Key: HIVE-4940 URL: https://issues.apache.org/jira/browse/HIVE-4940 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4940.D12189.1.patch Makes different result for 20(S) and 23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2482: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Mark! Convenience UDFs for binary data type - Key: HIVE-2482 URL: https://issues.apache.org/jira/browse/HIVE-2482 Project: Hive Issue Type: New Feature Reporter: Ashutosh Chauhan Assignee: Mark Wagner Fix For: 0.12.0 Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch, HIVE-2482.3.patch, HIVE-2482.4.patch HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[
https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738185#comment-13738185
]
Hive QA commented on HIVE-4985:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597640/HIVE-4985.D11961.5.patch
{color:green}SUCCESS:{color} +1 2848 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/415/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/415/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
refactor/clean up partition name pruning to be usable inside metastore server
--
Key: HIVE-4985
URL: https://issues.apache.org/jira/browse/HIVE-4985
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch,
HIVE-4985.D11961.3.patch, HIVE-4985.D11961.4.patch, HIVE-4985.D11961.5.patch
Preliminary for HIVE-4914.
The patch is going to be large already, so some refactoring and dead code
removal that is non-controversial can be done in advance in a separate patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5056) MapJoinProcessor ignores order of values in removing RS
[ https://issues.apache.org/jira/browse/HIVE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738186#comment-13738186 ] Ashutosh Chauhan commented on HIVE-5056: [~navis] As requested, would you like to describe the bug and subsequent fix in some detail? MapJoinProcessor ignores order of values in removing RS --- Key: HIVE-5056 URL: https://issues.apache.org/jira/browse/HIVE-5056 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-5056.D12147.1.patch, HIVE-5056.D12147.2.patch http://www.mail-archive.com/user@hive.apache.org/msg09073.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5022) Decimal Arithmetic generates NULL value
[ https://issues.apache.org/jira/browse/HIVE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Soo Hoo updated HIVE-5022: The answer cannot be anymore precise than the most precise operand. If this were a scientific calculation, then it would be the least precise operand that would dictate the precision. I have not actually looked at how other databases handle the situation, but they certainly do not throw nulls. I would use the most precise operand as my guide just to be safe. As you said in your note, people can always round up if they want fewer significant digits. Thank! Decimal Arithmetic generates NULL value --- Key: HIVE-5022 URL: https://issues.apache.org/jira/browse/HIVE-5022 Project: Hive Issue Type: Bug Components: Types Affects Versions: 0.11.0 Environment: Hortonworks 1.3 running Hive 0.11.0.1.3.0.0-107 Reporter: Kevin Soo Hoo Assignee: Teddy Choi Attachments: HIVE-5022.1.patch.txt When a decimal division is the first operation, the quotient cannot be multiplied in a subsequent calculation. Instead, a NULL is returned. The following yield NULL results: select (cast (4.53 as decimal) / cast(25.86 as decimal)) * cast(0.087 as decimal) from tablename limit 1; select cast (4.53 as decimal) / cast(25.86 as decimal) * cast(0.087 as decimal) from tablename limit 1; If we move the multiplication operation to be first, then it will successfully calculate the result. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5062) Insert + orderby + limit does not need additional RS for limiting rows
[
https://issues.apache.org/jira/browse/HIVE-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738204#comment-13738204
]
Phabricator commented on HIVE-5062:
---
ashutoshc has accepted the revision HIVE-5062 [jira] Insert + orderby + limit
does not need additional RS for limiting rows.
+1
REVISION DETAIL
https://reviews.facebook.net/D12171
BRANCH
HIVE-5062
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Insert + orderby + limit does not need additional RS for limiting rows
--
Key: HIVE-5062
URL: https://issues.apache.org/jira/browse/HIVE-5062
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5062.D12171.1.patch
The query,
{noformat}
insert overwrite table dummy select * from src order by key limit 10;
{noformat}
runs two MR but single MR is enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5061) Row sampling throws NPE when used in sub-query
[
https://issues.apache.org/jira/browse/HIVE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738206#comment-13738206
]
Phabricator commented on HIVE-5061:
---
ashutoshc has accepted the revision HIVE-5061 [jira] Row sampling throws NPE
when used in sub-query.
+1
REVISION DETAIL
https://reviews.facebook.net/D12165
BRANCH
HIVE-5061
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Row sampling throws NPE when used in sub-query
--
Key: HIVE-5061
URL: https://issues.apache.org/jira/browse/HIVE-5061
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-5061.D12165.1.patch
select * from (select * from src TABLESAMPLE (1 ROWS)) x;
{noformat}
ava.lang.NullPointerException
at
org.apache.hadoop.hive.ql.parse.SplitSample.getTargetSize(SplitSample.java:103)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.sampleSplits(CombineHiveInputFormat.java:487)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:405)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1025)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1017)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:928)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:881)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:881)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:855)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1424)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1204)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:878)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3077) Insert overwrite table doesn't fail for bucketed tables and breaks bucketing
[ https://issues.apache.org/jira/browse/HIVE-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned HIVE-3077: - Assignee: Xuefu Zhang Insert overwrite table doesn't fail for bucketed tables and breaks bucketing Key: HIVE-3077 URL: https://issues.apache.org/jira/browse/HIVE-3077 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0 Environment: java version 1.6.0_30 hive version 0.9.0 hadoop version 0.20.205.0 Reporter: Mark Grover Assignee: Xuefu Zhang If table my_table is bucketed, the command insert into table my_table ... is supposed to give an error stating Bucketized tables do not support INSERT INTO. However, it doesn't seem to do that in all cases. Consider the following example on Hive 0.9.0: create table src(x string) clustered by( x ) sorted by ( x ) into 32 buckets; create table dest(x string) clustered by( x ) sorted by ( x ) into 32 buckets; Now, put some data into x (after enable hive.enforce.bucketing and hive.enforce.sorting to be true). Then, do: insert into table dest select * from src; This should fail since dest is a bucketized table. However, this succeeds creating a 33rd file inside the HDFS folder for the table, thereby corrupting it. This happens regardless of whether the src table is bucketed or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3819) Creating a table on Hive without Hadoop daemons running returns a misleading error
[
https://issues.apache.org/jira/browse/HIVE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang reassigned HIVE-3819:
-
Assignee: Xuefu Zhang
Creating a table on Hive without Hadoop daemons running returns a misleading
error
--
Key: HIVE-3819
URL: https://issues.apache.org/jira/browse/HIVE-3819
Project: Hive
Issue Type: Bug
Components: CLI, Metastore
Reporter: Mark Grover
Assignee: Xuefu Zhang
I was running hive without running the underlying hadoop daemon's running.
Hadoop was configured to run in pseudo-distributed mode. However, when I
tried to create a hive table, I got this rather misleading error:
{code}
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
{code}
We should look into making this error message less misleading (more about
hadoop daemons not running instead of metastore client not being
instantiable).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5063) Fix some non-deterministic or not-updated tests
[ https://issues.apache.org/jira/browse/HIVE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738224#comment-13738224 ] Phabricator commented on HIVE-5063: --- ashutoshc has accepted the revision HIVE-5063 [jira] Fix some non-deterministic or not-updated tests. +1 REVISION DETAIL https://reviews.facebook.net/D12177 BRANCH HIVE-5063 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Fix some non-deterministic or not-updated tests --- Key: HIVE-5063 URL: https://issues.apache.org/jira/browse/HIVE-5063 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5063.D12177.1.patch update result auto_join14.q,input12.q,join14.q,union_remove_19.q fix non-determinisitcs partition_date.q,partition_date2.q,ppd_vc.q,nonblock_op_deduplicate.q -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3751) Handle enclosing characters in delimited fields
[
https://issues.apache.org/jira/browse/HIVE-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang reassigned HIVE-3751:
-
Assignee: Xuefu Zhang
Handle enclosing characters in delimited fields
---
Key: HIVE-3751
URL: https://issues.apache.org/jira/browse/HIVE-3751
Project: Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Mark Grover
Assignee: Xuefu Zhang
People have requested in the past to have Hive support enclosing characters
in delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{code}
DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]
{code}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping
of enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5063) Fix some non-deterministic or not-updated tests
[ https://issues.apache.org/jira/browse/HIVE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738231#comment-13738231 ] Brock Noland commented on HIVE-5063: Thanks the +1 Ashutosh! I just ran the tests changed on hadoop2 and they pass so I am planning on committing this. I'll wait a few minutes just in case you already are. :) Fix some non-deterministic or not-updated tests --- Key: HIVE-5063 URL: https://issues.apache.org/jira/browse/HIVE-5063 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5063.D12177.1.patch update result auto_join14.q,input12.q,join14.q,union_remove_19.q fix non-determinisitcs partition_date.q,partition_date2.q,ppd_vc.q,nonblock_op_deduplicate.q -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3104) Predicate pushdown doesn't work with multi-insert statements using LATERAL VIEW
[ https://issues.apache.org/jira/browse/HIVE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned HIVE-3104: - Assignee: Xuefu Zhang Predicate pushdown doesn't work with multi-insert statements using LATERAL VIEW --- Key: HIVE-3104 URL: https://issues.apache.org/jira/browse/HIVE-3104 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.9.0 Environment: Apache Hive 0.9.0, Apache Hadoop 0.20.205.0 Reporter: Mark Grover Assignee: Xuefu Zhang Predicate pushdown seems to work for single-insert queries using LATERAL VIEW. It also seems to work for multi-insert queries *not* using LATERAL VIEW. However, it doesn't work for multi-insert queries using LATERAL VIEW. Here are some examples. In the below examples, I make use of the fact that a query with no partition filtering when run under hive.mapred.mode=strict fails. --Table creation and population DROP TABLE IF EXISTS test; CREATE TABLE test (col1 arrayint, col2 int) PARTITIONED BY (part_col int); INSERT OVERWRITE TABLE test PARTITION (part_col=1) SELECT array(1,2), count(*) FROM test; INSERT OVERWRITE TABLE test PARTITION (part_col=2) SELECT array(2,4,6), count(*) FROM test; -- Query 1 -- This succeeds (using LATERAL VIEW with single insert) set hive.mapred.mode=strict; FROM partition_test LATERAL VIEW explode(col1) tmp AS exp_col1 INSERT OVERWRITE DIRECTORY '/test/1' SELECT exp_col1 WHERE (part_col=2); -- Query 2 -- This succeeds (NOT using LATERAL VIEW with multi-insert) set hive.mapred.mode=strict; FROM partition_test INSERT OVERWRITE DIRECTORY '/test/1' SELECT col1 WHERE (part_col=2) INSERT OVERWRITE DIRECTORY '/test/2' SELECT col1 WHERE (part_col=2); -- Query 3 -- This fails (using LATERAL VIEW with multi-insert) set hive.mapred.mode=strict; FROM test LATERAL VIEW explode(col1) tmp AS exp_col1 INSERT OVERWRITE DIRECTORY '/test/1' SELECT exp_col1 WHERE (part_col=2) INSERT OVERWRITE DIRECTORY '/test/2' SELECT exp_col1 WHERE (part_col=2); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5059) Meaningless warning message from TypeCheckProcFactory
[
https://issues.apache.org/jira/browse/HIVE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738233#comment-13738233
]
Phabricator commented on HIVE-5059:
---
ashutoshc has accepted the revision HIVE-5059 [jira] Meaningless warning
message from TypeCheckProcFactory.
+1
REVISION DETAIL
https://reviews.facebook.net/D12159
BRANCH
HIVE-5059
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Meaningless warning message from TypeCheckProcFactory
-
Key: HIVE-5059
URL: https://issues.apache.org/jira/browse/HIVE-5059
Project: Hive
Issue Type: Task
Components: Logging
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5059.D12159.1.patch
Regression from HIVE-3849, hive logs meaningless messages as warning like
below,
{noformat}
WARN parse.TypeCheckProcFactory (TypeCheckProcFactory.java:convert(180)) -
Invalid type entry TOK_TABLE_OR_COL=null
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5063) Fix some non-deterministic or not-updated tests
[ https://issues.apache.org/jira/browse/HIVE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5063: --- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk! Thank you very much for you patch! Fix some non-deterministic or not-updated tests --- Key: HIVE-5063 URL: https://issues.apache.org/jira/browse/HIVE-5063 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5063.D12177.1.patch update result auto_join14.q,input12.q,join14.q,union_remove_19.q fix non-determinisitcs partition_date.q,partition_date2.q,ppd_vc.q,nonblock_op_deduplicate.q -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3107) Improve semantic analyzer to better handle column name references in group by/sort by clauses
[
https://issues.apache.org/jira/browse/HIVE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738241#comment-13738241
]
Xuefu Zhang commented on HIVE-3107:
---
I plan to make this improvement, so assign it to me. Please feel free to take
it from me if anyone likes to work on it.
Improve semantic analyzer to better handle column name references in group
by/sort by clauses
-
Key: HIVE-3107
URL: https://issues.apache.org/jira/browse/HIVE-3107
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.9.0
Reporter: Richard Ding
This is related to HIVE-1922.
Following queries all fail with various SemanticExceptions:
{code}
explain select t.c from t group by c;
explain select t.c from t group by c sort by t.c;
explain select t.c as c0 from t group by c0;
explain select t.c from t group by t.c sort by t.c;
{code}
It is true that one could always find a version of any of above queries that
works. But one has to try to find out and it doesn't work well with machine
generated SQL queries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3107) Improve semantic analyzer to better handle column name references in group by/sort by clauses
[
https://issues.apache.org/jira/browse/HIVE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang reassigned HIVE-3107:
-
Assignee: Xuefu Zhang
Improve semantic analyzer to better handle column name references in group
by/sort by clauses
-
Key: HIVE-3107
URL: https://issues.apache.org/jira/browse/HIVE-3107
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.9.0
Reporter: Richard Ding
Assignee: Xuefu Zhang
This is related to HIVE-1922.
Following queries all fail with various SemanticExceptions:
{code}
explain select t.c from t group by c;
explain select t.c from t group by c sort by t.c;
explain select t.c as c0 from t group by c0;
explain select t.c from t group by t.c sort by t.c;
{code}
It is true that one could always find a version of any of above queries that
works. But one has to try to find out and it doesn't work well with machine
generated SQL queries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4925) Modify Hive build to enable compiling and running Hive with JDK7
[ https://issues.apache.org/jira/browse/HIVE-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738253#comment-13738253 ] Xuefu Zhang commented on HIVE-4925: --- It's found that compiling and running hive with JDK7 (or JDK6) requires no change on hive build and it's just a matter of setting up the JDK environment. Thus, I plan to close this issue as not an issue. Please let me know if anyone disagree. Modify Hive build to enable compiling and running Hive with JDK7 Key: HIVE-4925 URL: https://issues.apache.org/jira/browse/HIVE-4925 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4925) Modify Hive build to enable compiling and running Hive with JDK7
[ https://issues.apache.org/jira/browse/HIVE-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738255#comment-13738255 ] Ashutosh Chauhan commented on HIVE-4925: [~xuefuz] Can you please document what (if any) changes in JDK environment are required to get hive to compile/run on JDK 7? Modify Hive build to enable compiling and running Hive with JDK7 Key: HIVE-4925 URL: https://issues.apache.org/jira/browse/HIVE-4925 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4925) Modify Hive build to enable compiling and running Hive with JDK7
[ https://issues.apache.org/jira/browse/HIVE-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738272#comment-13738272 ] Brock Noland commented on HIVE-4925: Doesn't it require -Djavac.version=1.7 as an ant arg? Modify Hive build to enable compiling and running Hive with JDK7 Key: HIVE-4925 URL: https://issues.apache.org/jira/browse/HIVE-4925 Project: Hive Issue Type: Sub-task Components: Build Infrastructure Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3189) cast ( string type as bigint) returning null values
[ https://issues.apache.org/jira/browse/HIVE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738293#comment-13738293 ] pandeeswaran commented on HIVE-3189: I am unable to reproduce this in 0.8.1.6 — Sent from Mailbox for iPad On Tue, Aug 13, 2013 at 6:21 PM, Ashutosh Chauhan (JIRA) j...@apache.org cast ( string type as bigint) returning null values - Key: HIVE-3189 URL: https://issues.apache.org/jira/browse/HIVE-3189 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Attachments: Hive-3189.patch.txt select rnum, c1, cast(c1 as bigint) from cert.tsdchar tsdchar where rnum in (0,1,2) create table if not exists CERT.TSDCHAR ( RNUM int , C1 string) row format sequencefile rnum c1 _c2 0 -1 null 1 0 null 2 10 null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev
[
https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738324#comment-13738324
]
Hive QA commented on HIVE-4690:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12586918/HIVE-4690.D11163.1.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/417/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/417/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-417/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTF.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
ql/src/test/results/clientpositive/udf_col_names.q.out
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDFColumnNameTest.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFColumnNameTest.java
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFColumnNameTest.java
ql/src/test/queries/clientpositive/udf_col_names.q
+ svn update
Uql/src/test/queries/clientpositive/partition_date2.q
Uql/src/test/queries/clientpositive/ppd_vc.q
Uql/src/test/queries/clientpositive/nonblock_op_deduplicate.q
Uql/src/test/queries/clientpositive/partition_date.q
Uql/src/test/results/clientpositive/ppd_vc.q.out
Uql/src/test/results/clientpositive/partition_date2.q.out
Uql/src/test/results/clientpositive/nonblock_op_deduplicate.q.out
Uql/src/test/results/clientpositive/input12.q.out
Uql/src/test/results/clientpositive/auto_join14.q.out
Uql/src/test/results/clientpositive/union_remove_19.q.out
Uql/src/test/results/clientpositive/join14.q.out
Uql/src/test/results/clientpositive/partition_date.q.out
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1513535.
Updated to revision 1513534.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
stats_partscan_1.q makes different result with different hadhoop.mr.rev
Key: HIVE-4690
URL: https://issues.apache.org/jira/browse/HIVE-4690
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.11.0
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-4690.D11163.1.patch
[jira] [Commented] (HIVE-5025) Column aliases for input argument of GenericUDFs
[
https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738319#comment-13738319
]
Hive QA commented on HIVE-5025:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596782/HIVE-5025.D12093.1.patch
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 2857 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_multipartitioning
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_general_queries
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_seqfile
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_rank
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_InvalidValueBoundary
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_rcfile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_reuse_memstore
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_columnPruning
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_ntile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby1
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/416/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/416/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
Column aliases for input argument of GenericUDFs
-
Key: HIVE-5025
URL: https://issues.apache.org/jira/browse/HIVE-5025
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5025.D12093.1.patch
In some cases, column aliases for input argument are very useful to know. But
I cannot sure of this in the sense that UDFs should not be dependent to
contextual information like column alias.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5018) Avoiding object instantiation in loops (issue 6)
[ https://issues.apache.org/jira/browse/HIVE-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Jakobus updated HIVE-5018: --- Status: Open (was: Patch Available) Avoiding object instantiation in loops (issue 6) Key: HIVE-5018 URL: https://issues.apache.org/jira/browse/HIVE-5018 Project: Hive Issue Type: Sub-task Reporter: Benjamin Jakobus Assignee: Benjamin Jakobus Priority: Minor Fix For: 0.12.0 Object instantiation inside loops is very expensive. Where possible, object references should be created outside the loop so that they can be reused. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4003: --- Status: Open (was: Patch Available) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - Key: HIVE-4003 URL: https://issues.apache.org/jira/browse/HIVE-4003 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Thomas Adam Assignee: Mark Grover Attachments: HIVE-4003.patch Utilities.java seems to be throwing a NPE. Change contributed by Thomas Adam. Reference: https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[
https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738325#comment-13738325
]
Hive QA commented on HIVE-4003:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12568605/HIVE-4003.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/419/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/419/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-419/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1513535.
At revision 1513535.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
NullPointerException in
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
-
Key: HIVE-4003
URL: https://issues.apache.org/jira/browse/HIVE-4003
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Thomas Adam
Assignee: Mark Grover
Attachments: HIVE-4003.patch
Utilities.java seems to be throwing a NPE.
Change contributed by Thomas Adam.
Reference:
https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5018) Avoiding object instantiation in loops (issue 6)
[ https://issues.apache.org/jira/browse/HIVE-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Jakobus updated HIVE-5018: --- Attachment: (was: HIVE-5018.1.patch.txt) Avoiding object instantiation in loops (issue 6) Key: HIVE-5018 URL: https://issues.apache.org/jira/browse/HIVE-5018 Project: Hive Issue Type: Sub-task Reporter: Benjamin Jakobus Assignee: Benjamin Jakobus Priority: Minor Fix For: 0.12.0 Object instantiation inside loops is very expensive. Where possible, object references should be created outside the loop so that they can be reused. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738340#comment-13738340 ] Brock Noland commented on HIVE-4003: Hey [~mgrover] thanks for the patch! Since it doesn't apply any longer can you rebase? NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - Key: HIVE-4003 URL: https://issues.apache.org/jira/browse/HIVE-4003 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Thomas Adam Assignee: Mark Grover Attachments: HIVE-4003.patch Utilities.java seems to be throwing a NPE. Change contributed by Thomas Adam. Reference: https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3751) Handle enclosing characters in delimited fields
[
https://issues.apache.org/jira/browse/HIVE-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738348#comment-13738348
]
Xuefu Zhang commented on HIVE-3751:
---
I plan to work on this. However, if anyone else wants it, please feel free.
Handle enclosing characters in delimited fields
---
Key: HIVE-3751
URL: https://issues.apache.org/jira/browse/HIVE-3751
Project: Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Mark Grover
Assignee: Xuefu Zhang
People have requested in the past to have Hive support enclosing characters
in delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{code}
DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]
{code}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping
of enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3077) Insert overwrite table doesn't fail for bucketed tables and breaks bucketing
[ https://issues.apache.org/jira/browse/HIVE-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738349#comment-13738349 ] Xuefu Zhang commented on HIVE-3077: --- I plan to work on this(plus a few others). However, if anyone else wants it, please feel free. Insert overwrite table doesn't fail for bucketed tables and breaks bucketing Key: HIVE-3077 URL: https://issues.apache.org/jira/browse/HIVE-3077 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0 Environment: java version 1.6.0_30 hive version 0.9.0 hadoop version 0.20.205.0 Reporter: Mark Grover Assignee: Xuefu Zhang If table my_table is bucketed, the command insert into table my_table ... is supposed to give an error stating Bucketized tables do not support INSERT INTO. However, it doesn't seem to do that in all cases. Consider the following example on Hive 0.9.0: create table src(x string) clustered by( x ) sorted by ( x ) into 32 buckets; create table dest(x string) clustered by( x ) sorted by ( x ) into 32 buckets; Now, put some data into x (after enable hive.enforce.bucketing and hive.enforce.sorting to be true). Then, do: insert into table dest select * from src; This should fail since dest is a bucketized table. However, this succeeds creating a 33rd file inside the HDFS folder for the table, thereby corrupting it. This happens regardless of whether the src table is bucketed or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2609) NPE when pruning partitions by thrift method get_partitions_by_filter
[
https://issues.apache.org/jira/browse/HIVE-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738346#comment-13738346
]
Brock Noland commented on HIVE-2609:
I tried reproducing this after HIVE-3632 and cannot. Therefore I think we can
mark it resolved.
NPE when pruning partitions by thrift method get_partitions_by_filter
-
Key: HIVE-2609
URL: https://issues.apache.org/jira/browse/HIVE-2609
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.7.1
Reporter: Min Zhou
It's a datanucleus bug indeed.
try this code:
{code}
boolean open = false;
for (int i = 0; i 5 !open; ++i) {
try {
transport.open();
open = true;
} catch (TTransportException e) {
System.out.println(failed to connect to MetaStore, re-trying...);
try {
Thread.sleep(1000);
} catch (InterruptedException ignore) {}
}
}
try {
ListPartition parts =
client.get_partitions_by_filter(default, partitioned_nation,
pt '2', (short) -1);
for (Partition part : parts) {
System.out.println(part.getSd().getLocation());
}
} catch (Exception te) {
te.printStackTrace();
}
{code}
A NPEexception would be thrown on the thrift server side
{noformat}
11/11/25 13:11:55 ERROR api.ThriftHiveMetastore$Processor: Internal error
processing get_partitions_by_filter
java.lang.NullPointerException
at
org.datanucleus.store.mapped.mapping.MappingHelper.getMappingIndices(MappingHelper.java:35)
at
org.datanucleus.store.mapped.expression.StatementText.applyParametersToStatement(StatementText.java:194)
at
org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:233)
at
org.datanucleus.store.rdbms.query.legacy.SQLEvaluator.evaluate(SQLEvaluator.java:115)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.performExecute(JDOQLQuery.java:288)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1657)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithMap(Query.java:1526)
at org.datanucleus.jdo.JDOQuery.executeWithMap(JDOQuery.java:334)
at
org.apache.hadoop.hive.metastore.ObjectStore.listMPartitionsByFilter(ObjectStore.java:1329)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:1241)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2369)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2366)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:2366)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.process(ThriftHiveMetastore.j
ava:6099)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor.process(ThriftHiveMetastore.java:4789)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$TLoggingProcessor.process(HiveMetaStore.java:3167)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
A null JavaTypeMapping was passed into
org.datanucleus.store.mapped.mapping.MappingHelper.(int initialPosition,
JavaTypeMapping mapping), that caused NPE.
After digged into the datanucleus source, I found that the null value was
born in the constructor of
org.datanucleus.store.mapped.expression.SubstringExpression. see
{code}
/**
* Constructs the substring
* @param str the String Expression
* @param begin The start position
* @param end The end position expression
**/
public SubstringExpression(StringExpression str, NumericExpression begin,
NumericExpression end)
{
super(str.getQueryExpression());
st.append(SUBSTRING().append(str).append( FROM )
.append(begin.add(new IntegerLiteral(qs, mapping,
BigInteger.ONE)))
.append( FOR ).append(end.sub(begin)).append(')');
}
{code}
The field mapping hasn't been instanced at that moment.
How do you deal with such a external bug?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
[jira] [Commented] (HIVE-4355) HCatalog test TestPigHCatUtil might fail on JDK7
[
https://issues.apache.org/jira/browse/HIVE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738356#comment-13738356
]
Jarek Jarcec Cecho commented on HIVE-4355:
--
Hi [~ashutoshc],
thank you very much for taking a look on this patch! Please accept my apologies
for late response, I've unfortunately completely missed the updates.
The issue here is that method {{testGetBagSubSchemaConfigured}} is setting
System properties in order to test the behavior, but it's not cleaning them
after the test. This do not appear to be an issue on pure JDK6 as the method
{{testGetBagSubSchema}} is being executed before
{{testGetBagSubSchemaConfigured}}. However when I compiled the tests with JDK6
and then executed them on JDK7, the order in which the two testing methods are
executed has changed - the method {{testGetBagSubSchemaConfigured}} was
executed before {{testGetBagSubSchema}}. As the
{{testGetBagSubSchemaConfigured}} was not cleaning up the System properties,
the second method {{testGetBagSubSchema}} has picked them up and the test
itself has failed. Hope that it helps!
HCatalog test TestPigHCatUtil might fail on JDK7
Key: HIVE-4355
URL: https://issues.apache.org/jira/browse/HIVE-4355
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Attachments: HIVE-4355.patch
I’ve tried interesting scenario. I’ve compiled hcatalog with JDK 6 (including
tests) and run the tests itself on JDK 7. My motivation was to see what will
happen to users that will download official Apache release (usually compiled
on JDK 6) and will run it on JDK 7. I’ve seen {{TestPigHCatUtil}} failing
because the order of method calls was different then when compiling and
running the tests only on JDK 6 or only on JDK 7.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738355#comment-13738355 ] Mark Grover commented on HIVE-4003: --- Thanks, Brock, for taking a look. I will rebase this. NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - Key: HIVE-4003 URL: https://issues.apache.org/jira/browse/HIVE-4003 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Thomas Adam Assignee: Mark Grover Attachments: HIVE-4003.patch Utilities.java seems to be throwing a NPE. Change contributed by Thomas Adam. Reference: https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4355) HCatalog test TestPigHCatUtil might fail on JDK7
[
https://issues.apache.org/jira/browse/HIVE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Jarcec Cecho updated HIVE-4355:
-
Status: Patch Available (was: Open)
Putting back status Patch available as the explanation was provided.
HCatalog test TestPigHCatUtil might fail on JDK7
Key: HIVE-4355
URL: https://issues.apache.org/jira/browse/HIVE-4355
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Attachments: HIVE-4355.patch
I’ve tried interesting scenario. I’ve compiled hcatalog with JDK 6 (including
tests) and run the tests itself on JDK 7. My motivation was to see what will
happen to users that will download official Apache release (usually compiled
on JDK 6) and will run it on JDK 7. I’ve seen {{TestPigHCatUtil}} failing
because the order of method calls was different then when compiling and
running the tests only on JDK 6 or only on JDK 7.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4778) hive.server2.authentication CUSTOM not working
[ https://issues.apache.org/jira/browse/HIVE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4778: -- Attachment: HIVE-4778.D12207.1.patch azrael requested code review of HIVE-4778 [jira] hive.server2.authentication CUSTOM not working. Reviewers: JIRA TEST PLAN unit test and manual test REVISION DETAIL https://reviews.facebook.net/D12207 AFFECTED FILES build-common.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template hbase-handler/src/test/templates/TestHBaseCliDriver.vm ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerWriter.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java.orig ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriter.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFDecode.java ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBase64.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnbase64.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnhex.java ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestNewIntegerEncoding.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDecode.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFEncode.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFBase64.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFHex.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnbase64.java ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnhex.java ql/src/test/queries/clientpositive/ba_table_udfs.q ql/src/test/queries/clientpositive/nonblock_op_deduplicate.q ql/src/test/queries/clientpositive/partition_date.q ql/src/test/queries/clientpositive/partition_date2.q ql/src/test/queries/clientpositive/ppd_vc.q ql/src/test/queries/clientpositive/schemeAuthority2.q ql/src/test/queries/clientpositive/udf_sentences.q ql/src/test/resources/orc-file-dump-dictionary-threshold.out ql/src/test/resources/orc-file-dump.out ql/src/test/results/clientnegative/fs_default_name1.q.out ql/src/test/results/clientnegative/fs_default_name2.q.out ql/src/test/results/clientpositive/auto_join14.q.out ql/src/test/results/clientpositive/ba_table_udfs.q.out ql/src/test/results/clientpositive/input12.q.out ql/src/test/results/clientpositive/join14.q.out ql/src/test/results/clientpositive/nonblock_op_deduplicate.q.out ql/src/test/results/clientpositive/partition_date.q.out ql/src/test/results/clientpositive/partition_date2.q.out ql/src/test/results/clientpositive/ppd_vc.q.out ql/src/test/results/clientpositive/schemeAuthority2.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_hex.q.out ql/src/test/results/clientpositive/udf_sentences.q.out ql/src/test/results/clientpositive/udf_unhex.q.out ql/src/test/results/clientpositive/union_remove_19.q.out ql/src/test/templates/TestCliDriver.vm service/src/java/org/apache/hive/service/auth/CustomAuthenticationProviderImpl.java
[jira] [Commented] (HIVE-4778) hive.server2.authentication CUSTOM not working
[ https://issues.apache.org/jira/browse/HIVE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738376#comment-13738376 ] Azrael Park commented on HIVE-4778: --- I rebased wrong, I will update patch. hive.server2.authentication CUSTOM not working -- Key: HIVE-4778 URL: https://issues.apache.org/jira/browse/HIVE-4778 Project: Hive Issue Type: Bug Components: Authentication Affects Versions: 0.11.0 Environment: CentOS release 6.2 x86_64 java version 1.6.0_31 Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Reporter: Zdenek Ott Assignee: Azrael Park Attachments: HIVE-4778.D12207.1.patch I have created my own class PamAuthenticationProvider that implements PasswdAuthenticationProvider interface. I have puted jar into hive lib directory and have configured hive-site.xml in following way: property namehive.server2.authentication/name valueCUSTOM/value /property property namehive.server2.custom.authentication.class/name valuecom.avast.ff.hive.PamAuthenticationProvider/value /property I use SQuireL and jdbc drivers to connect to hive. During authentication Hive throws following exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128) at org.apache.hive.service.auth.CustomAuthenticationProviderImpl.init(CustomAuthenticationProviderImpl.java:20) at org.apache.hive.service.auth.AuthenticationProviderFactory.getAuthenticationProvider(AuthenticationProviderFactory.java:57) at org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(PlainSaslHelper.java:61) at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:127) at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:509) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.init() at java.lang.Class.getConstructor0(Class.java:2706) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:122) ... 12 more I have done small patch for org.apache.hive.service.auth.CustomAuthenticationProviderImpl , that have solved my problem, but I'm not sure if it's the best solution. Here is the patch: --- CustomAuthenticationProviderImpl.java 2013-06-20 14:55:22.473995184 +0200 +++ CustomAuthenticationProviderImpl.java.new 2013-06-20 14:57:36.549012966 +0200 @@ -33,7 +33,7 @@ HiveConf conf = new HiveConf(); this.customHandlerClass = (Class? extends PasswdAuthenticationProvider) conf.getClass( - HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.name(), + HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.varname, PasswdAuthenticationProvider.class); this.customProvider = ReflectionUtils.newInstance(this.customHandlerClass, conf); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5018) Avoiding object instantiation in loops (issue 6)
[ https://issues.apache.org/jira/browse/HIVE-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Jakobus updated HIVE-5018: --- Attachment: HIVE-5018.1.patch.txt Avoiding object instantiation in loops (issue 6) Key: HIVE-5018 URL: https://issues.apache.org/jira/browse/HIVE-5018 Project: Hive Issue Type: Sub-task Reporter: Benjamin Jakobus Assignee: Benjamin Jakobus Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5018.1.patch.txt Object instantiation inside loops is very expensive. Where possible, object references should be created outside the loop so that they can be reused. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5018) Avoiding object instantiation in loops (issue 6)
[ https://issues.apache.org/jira/browse/HIVE-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Jakobus updated HIVE-5018: --- Release Note: Minor performance improvements: I ran arithmetic operations on a small dataset consisting of 30,000,000 records (standalone mode) and the patched version was 2.6% faster than the unpatched version. Status: Patch Available (was: Open) Avoiding object instantiation in loops (issue 6) Key: HIVE-5018 URL: https://issues.apache.org/jira/browse/HIVE-5018 Project: Hive Issue Type: Sub-task Reporter: Benjamin Jakobus Assignee: Benjamin Jakobus Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5018.1.patch.txt Object instantiation inside loops is very expensive. Where possible, object references should be created outside the loop so that they can be reused. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4355) HCatalog test TestPigHCatUtil might fail on JDK7
[
https://issues.apache.org/jira/browse/HIVE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738405#comment-13738405
]
Xuefu Zhang commented on HIVE-4355:
---
[~jarcec] would you think it's reasonable to unset the system property in the
same test where such property is set rather than unsetting it in other methods?
HCatalog test TestPigHCatUtil might fail on JDK7
Key: HIVE-4355
URL: https://issues.apache.org/jira/browse/HIVE-4355
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Attachments: HIVE-4355.patch
I’ve tried interesting scenario. I’ve compiled hcatalog with JDK 6 (including
tests) and run the tests itself on JDK 7. My motivation was to see what will
happen to users that will download official Apache release (usually compiled
on JDK 6) and will run it on JDK 7. I’ve seen {{TestPigHCatUtil}} failing
because the order of method calls was different then when compiling and
running the tests only on JDK 6 or only on JDK 7.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738440#comment-13738440 ] Hudson commented on HIVE-4513: -- FAILURE: Integrated in Hive-trunk-h0.21 #2265 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2265/]) HIVE-4513 : disable hivehistory logs by default (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513445) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java * /hive/trunk/ql/src/test/templates/TestCliDriver.vm disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5051) StorageBasedAuthorizationProvider masks lower level exception with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738439#comment-13738439
]
Hudson commented on HIVE-5051:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #356 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/356/])
HIVE-5051 : StorageBasedAuthorizationProvider masks lower level exception with
IllegalStateException (Eugene Koifman via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513450)
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
StorageBasedAuthorizationProvider masks lower level exception with
IllegalStateException
Key: HIVE-5051
URL: https://issues.apache.org/jira/browse/HIVE-5051
Project: Hive
Issue Type: Bug
Components: Authorization
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: HIVE-5051.patch
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
has the following 2 methods
private HiveException hiveException(Exception e) {
HiveException ex = new HiveException(e);
ex.initCause(e);
return ex;
}
private AuthorizationException authorizationException(Exception e) {
AuthorizationException ex = new AuthorizationException(e);
ex.initCause(e);
return ex;
}
both, when called, will cause
throw new IllegalStateException(Can't overwrite cause); to be thrown and
the original Exception is lost.
See
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/Throwable.java#Throwable.initCause%28java.lang.Throwable%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738444#comment-13738444 ] Hudson commented on HIVE-4863: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/]) HIVE-4863: Fix parallel order by on hadoop2 (Edward Capriolo via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513351) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-4863.1.patch.txt, HIVE-4863.2.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738438#comment-13738438 ] Hudson commented on HIVE-4863: -- FAILURE: Integrated in Hive-trunk-h0.21 #2265 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2265/]) HIVE-4863: Fix parallel order by on hadoop2 (Edward Capriolo via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513351) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-4863.1.patch.txt, HIVE-4863.2.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5023) Hive get wrong result when partition has the same path but different schema or authority
[
https://issues.apache.org/jira/browse/HIVE-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738445#comment-13738445
]
Hudson commented on HIVE-5023:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/])
HIVE-5023 : Hive get wrong result when partition has the same path but
different schema or authority (Shuaishuai Nie via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513451)
* /hive/trunk/build-common.xml
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
* /hive/trunk/ql/src/test/queries/clientpositive/schemeAuthority2.q
* /hive/trunk/ql/src/test/results/clientpositive/schemeAuthority2.q.out
Hive get wrong result when partition has the same path but different schema
or authority
Key: HIVE-5023
URL: https://issues.apache.org/jira/browse/HIVE-5023
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Fix For: 0.12.0
Attachments: HIVE-5023.1.patch, HIVE-5023.2.patch
Hive does not differentiate scheme and authority in file uris which cause
wrong result when partition has the same path but different schema or
authority. Here is a simple repro
partition file path:
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text1.txt
with content 2013-08-05 00:00:00
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text2.txt
with content 2013-08-05 00:00:20
{noformat}
CREATE EXTERNAL TABLE IF NOT EXISTS T1 (t STRING) PARTITIONED BY (ProcessDate
STRING, Hour STRING, ClusterName STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED by '\t' STORED AS TEXTFILE;
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='CLusterA');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterA') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='ClusterB');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterB') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
{noformat}
the expect output of the hive query
{noformat}
SELECT ClusterName, t FROM T1 WHERE ProcessDate=’2013-08-05’ AND Hour=’00’;
{noformat}
should be
{noformat}
ClusterA2013-08-05 00:00:00
ClusterB2013-08-05 00:00:20
{noformat}
However it is
{noformat}
ClusterA2013-08-05 00:00:00
ClusterA2013-08-05 00:00:20
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4885) Alternative object serialization for execution plan in hive testing
[
https://issues.apache.org/jira/browse/HIVE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738449#comment-13738449
]
Hudson commented on HIVE-4885:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/])
HIVE-4885 : Alternative object serialization for execution plan in hive testing
(Xuefu Zhang via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513465)
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
Alternative object serialization for execution plan in hive testing
Key: HIVE-4885
URL: https://issues.apache.org/jira/browse/HIVE-4885
Project: Hive
Issue Type: Improvement
Components: CLI
Affects Versions: 0.10.0, 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.12.0
Attachments: HIVE-4885.patch
Currently there are a lot of test cases involving in comparing execution
plan, such as those in TestParse suite. XmlEncoder is used to serialize the
generated plan by hive, and store it in the file for file diff comparison.
However, XmlEncoder is tied with Java compiler, whose implementation may
change from version to version. Thus, upgrade the compiler can generate a lot
of fake test failures. The following is an example of diff generated when
running hive with JDK7:
{code}
Begin query: case_sensitivity.q
diff -a
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.out
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/parse/case_sensitivity.q.out
diff -a -b
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.xml
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/plan/case_sensitivity.q.xml
3c3
object class=org.apache.hadoop.hive.ql.exec.MapRedTask id=MapRedTask0
---
object id=MapRedTask0
class=org.apache.hadoop.hive.ql.exec.MapRedTask
12c12
object class=java.util.ArrayList id=ArrayList0
---
object id=ArrayList0 class=java.util.ArrayList
14c14
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask0
---
object id=MoveTask0
class=org.apache.hadoop.hive.ql.exec.MoveTask
18c18
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask1
---
object id=MoveTask1
class=org.apache.hadoop.hive.ql.exec.MoveTask
22c22
object class=org.apache.hadoop.hive.ql.exec.StatsTask
id=StatsTask0
---
object id=StatsTask0
class=org.apache.hadoop.hive.ql.exec.StatsTask
60c60
object class=org.apache.hadoop.hive.ql.exec.MapRedTask
id=MapRedTask1
---
object id=MapRedTask1
class=org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
As it can be seen, the only difference is the order of the attributes in the
serialized XML doc, yet it brings 50+ test failures in Hive.
We need to have a better plan comparison, or object serialization to improve
the situation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5063) Fix some non-deterministic or not-updated tests
[ https://issues.apache.org/jira/browse/HIVE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738446#comment-13738446 ] Hudson commented on HIVE-5063: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/]) HIVE-5063: Fix some non-deterministic or not-updated tests (Navis via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513495) * /hive/trunk/ql/src/test/queries/clientpositive/nonblock_op_deduplicate.q * /hive/trunk/ql/src/test/queries/clientpositive/partition_date.q * /hive/trunk/ql/src/test/queries/clientpositive/partition_date2.q * /hive/trunk/ql/src/test/queries/clientpositive/ppd_vc.q * /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out * /hive/trunk/ql/src/test/results/clientpositive/input12.q.out * /hive/trunk/ql/src/test/results/clientpositive/join14.q.out * /hive/trunk/ql/src/test/results/clientpositive/nonblock_op_deduplicate.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_date.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_date2.q.out * /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out * /hive/trunk/ql/src/test/results/clientpositive/union_remove_19.q.out Fix some non-deterministic or not-updated tests --- Key: HIVE-5063 URL: https://issues.apache.org/jira/browse/HIVE-5063 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5063.D12177.1.patch update result auto_join14.q,input12.q,join14.q,union_remove_19.q fix non-determinisitcs partition_date.q,partition_date2.q,ppd_vc.q,nonblock_op_deduplicate.q -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738448#comment-13738448 ] Hudson commented on HIVE-2482: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/]) HIVE-2482 : Convenience UDFs for binary data type (Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513474) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFDecode.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBase64.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnbase64.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnhex.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDecode.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFEncode.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFBase64.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFHex.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnbase64.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnhex.java * /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q * /hive/trunk/ql/src/test/queries/clientpositive/udf_sentences.q * /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out * /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_hex.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_sentences.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_unhex.q.out Convenience UDFs for binary data type - Key: HIVE-2482 URL: https://issues.apache.org/jira/browse/HIVE-2482 Project: Hive Issue Type: New Feature Reporter: Ashutosh Chauhan Assignee: Mark Wagner Fix For: 0.12.0 Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch, HIVE-2482.3.patch, HIVE-2482.4.patch HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738442#comment-13738442 ] Hudson commented on HIVE-4513: -- FAILURE: Integrated in Hive-trunk-hadoop2 #356 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/356/]) HIVE-4513 : disable hivehistory logs by default (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513445) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java * /hive/trunk/ql/src/test/templates/TestCliDriver.vm disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738447#comment-13738447 ] Hudson commented on HIVE-4513: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #56 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/56/]) HIVE-4513 : disable hivehistory logs by default (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513445) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java * /hive/trunk/ql/src/test/templates/TestCliDriver.vm disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5023) Hive get wrong result when partition has the same path but different schema or authority
[
https://issues.apache.org/jira/browse/HIVE-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738441#comment-13738441
]
Hudson commented on HIVE-5023:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #356 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/356/])
HIVE-5023 : Hive get wrong result when partition has the same path but
different schema or authority (Shuaishuai Nie via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513451)
* /hive/trunk/build-common.xml
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
* /hive/trunk/ql/src/test/queries/clientpositive/schemeAuthority2.q
* /hive/trunk/ql/src/test/results/clientpositive/schemeAuthority2.q.out
Hive get wrong result when partition has the same path but different schema
or authority
Key: HIVE-5023
URL: https://issues.apache.org/jira/browse/HIVE-5023
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Fix For: 0.12.0
Attachments: HIVE-5023.1.patch, HIVE-5023.2.patch
Hive does not differentiate scheme and authority in file uris which cause
wrong result when partition has the same path but different schema or
authority. Here is a simple repro
partition file path:
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text1.txt
with content 2013-08-05 00:00:00
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text2.txt
with content 2013-08-05 00:00:20
{noformat}
CREATE EXTERNAL TABLE IF NOT EXISTS T1 (t STRING) PARTITIONED BY (ProcessDate
STRING, Hour STRING, ClusterName STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED by '\t' STORED AS TEXTFILE;
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='CLusterA');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterA') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='ClusterB');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterB') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
{noformat}
the expect output of the hive query
{noformat}
SELECT ClusterName, t FROM T1 WHERE ProcessDate=’2013-08-05’ AND Hour=’00’;
{noformat}
should be
{noformat}
ClusterA2013-08-05 00:00:00
ClusterB2013-08-05 00:00:20
{noformat}
However it is
{noformat}
ClusterA2013-08-05 00:00:00
ClusterA2013-08-05 00:00:20
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738467#comment-13738467 ] Hudson commented on HIVE-2482: -- FAILURE: Integrated in Hive-trunk-hadoop2 #357 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/357/]) HIVE-2482 : Convenience UDFs for binary data type (Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513474) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFDecode.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBase64.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnbase64.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnhex.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDecode.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFEncode.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFBase64.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFHex.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnbase64.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFUnhex.java * /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q * /hive/trunk/ql/src/test/queries/clientpositive/udf_sentences.q * /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out * /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_hex.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_sentences.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_unhex.q.out Convenience UDFs for binary data type - Key: HIVE-2482 URL: https://issues.apache.org/jira/browse/HIVE-2482 Project: Hive Issue Type: New Feature Reporter: Ashutosh Chauhan Assignee: Mark Wagner Fix For: 0.12.0 Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch, HIVE-2482.3.patch, HIVE-2482.4.patch HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4885) Alternative object serialization for execution plan in hive testing
[
https://issues.apache.org/jira/browse/HIVE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738468#comment-13738468
]
Hudson commented on HIVE-4885:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #357 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/357/])
HIVE-4885 : Alternative object serialization for execution plan in hive testing
(Xuefu Zhang via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513465)
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
Alternative object serialization for execution plan in hive testing
Key: HIVE-4885
URL: https://issues.apache.org/jira/browse/HIVE-4885
Project: Hive
Issue Type: Improvement
Components: CLI
Affects Versions: 0.10.0, 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.12.0
Attachments: HIVE-4885.patch
Currently there are a lot of test cases involving in comparing execution
plan, such as those in TestParse suite. XmlEncoder is used to serialize the
generated plan by hive, and store it in the file for file diff comparison.
However, XmlEncoder is tied with Java compiler, whose implementation may
change from version to version. Thus, upgrade the compiler can generate a lot
of fake test failures. The following is an example of diff generated when
running hive with JDK7:
{code}
Begin query: case_sensitivity.q
diff -a
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.out
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/parse/case_sensitivity.q.out
diff -a -b
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/build/ql/test/logs/positive/case_sensitivity.q.xml
/data/4/hive-local/a2307.halxg.cloudera.com-hiveptest-2/cdh-source/ql/src/test/results/compiler/plan/case_sensitivity.q.xml
3c3
object class=org.apache.hadoop.hive.ql.exec.MapRedTask id=MapRedTask0
---
object id=MapRedTask0
class=org.apache.hadoop.hive.ql.exec.MapRedTask
12c12
object class=java.util.ArrayList id=ArrayList0
---
object id=ArrayList0 class=java.util.ArrayList
14c14
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask0
---
object id=MoveTask0
class=org.apache.hadoop.hive.ql.exec.MoveTask
18c18
object class=org.apache.hadoop.hive.ql.exec.MoveTask
id=MoveTask1
---
object id=MoveTask1
class=org.apache.hadoop.hive.ql.exec.MoveTask
22c22
object class=org.apache.hadoop.hive.ql.exec.StatsTask
id=StatsTask0
---
object id=StatsTask0
class=org.apache.hadoop.hive.ql.exec.StatsTask
60c60
object class=org.apache.hadoop.hive.ql.exec.MapRedTask
id=MapRedTask1
---
object id=MapRedTask1
class=org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
As it can be seen, the only difference is the order of the attributes in the
serialized XML doc, yet it brings 50+ test failures in Hive.
We need to have a better plan comparison, or object serialization to improve
the situation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5063) Fix some non-deterministic or not-updated tests
[ https://issues.apache.org/jira/browse/HIVE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738466#comment-13738466 ] Hudson commented on HIVE-5063: -- FAILURE: Integrated in Hive-trunk-hadoop2 #357 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/357/]) HIVE-5063: Fix some non-deterministic or not-updated tests (Navis via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513495) * /hive/trunk/ql/src/test/queries/clientpositive/nonblock_op_deduplicate.q * /hive/trunk/ql/src/test/queries/clientpositive/partition_date.q * /hive/trunk/ql/src/test/queries/clientpositive/partition_date2.q * /hive/trunk/ql/src/test/queries/clientpositive/ppd_vc.q * /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out * /hive/trunk/ql/src/test/results/clientpositive/input12.q.out * /hive/trunk/ql/src/test/results/clientpositive/join14.q.out * /hive/trunk/ql/src/test/results/clientpositive/nonblock_op_deduplicate.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_date.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_date2.q.out * /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out * /hive/trunk/ql/src/test/results/clientpositive/union_remove_19.q.out Fix some non-deterministic or not-updated tests --- Key: HIVE-5063 URL: https://issues.apache.org/jira/browse/HIVE-5063 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5063.D12177.1.patch update result auto_join14.q,input12.q,join14.q,union_remove_19.q fix non-determinisitcs partition_date.q,partition_date2.q,ppd_vc.q,nonblock_op_deduplicate.q -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Proposing a 0.11.1
All, I'd like to create an 0.11.1 with some fixes in it. I plan to put together a release candidate over the next week. I'm in the process of putting together the list of bugs that I want to include, but I wanted to solicit the jiras that others though would be important for an 0.11.1. Thanks, Owen
Re: Proposing a 0.11.1
Hi. Can I suggest to include HIVE-3772 (performance fix; committed to trunk) HIVE-4583 (OpenJDK7 support; all subtasks seem to be done) Thanks, Cos On Tue, Aug 13, 2013 at 10:02AM, Owen O'Malley wrote: All, I'd like to create an 0.11.1 with some fixes in it. I plan to put together a release candidate over the next week. I'm in the process of putting together the list of bugs that I want to include, but I wanted to solicit the jiras that others though would be important for an 0.11.1. Thanks, Owen
[jira] [Commented] (HIVE-5068) Some queries fail due to XMLEncoder error on JDK7
[
https://issues.apache.org/jira/browse/HIVE-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738506#comment-13738506
]
Xuefu Zhang commented on HIVE-5068:
---
[~brocknoland] Which tests (queries) are generating these errors?
Some queries fail due to XMLEncoder error on JDK7
-
Key: HIVE-5068
URL: https://issues.apache.org/jira/browse/HIVE-5068
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Looks like something snuck in that breaks the JDK 7 build:
{noformat}
Caused by: java.lang.Exception: XMLEncoder: discarding statement
ArrayList.add(ASTNode);
... 106 more
Caused by: java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at
java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:238)
at
java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:400)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:118)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeObject1(Encoder.java:258)
at java.beans.Encoder.cloneStatement(Encoder.java:271)
at java.beans.Encoder.writeStatement(Encoder.java:301)
at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:400)
... 105 more
Caused by: java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at java.beans.Encoder.getValue(Encoder.java:108)
at java.beans.Encoder.get(Encoder.java:252)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:112)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:232)
... 118 more
Caused by: java.lang.InstantiationException: org.antlr.runtime.CommonToken
at java.lang.Class.newInstance(Class.java:359)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at java.beans.Statement.invokeInternal(Statement.java:292)
at java.beans.Statement.access$000(Statement.java:58)
at java.beans.Statement$2.run(Statement.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at java.beans.Statement.invoke(Statement.java:182)
at java.beans.Expression.getValue(Expression.java:153)
at java.beans.Encoder.getValue(Encoder.java:105)
... 130 more
{noformat}
and
{noformat}
java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:426)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:330)
at
org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:611)
at org.apache.hadoop.hive.ql.plan.MapredWork.toXML(MapredWork.java:88)
at
org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:505)
at
[jira] [Updated] (HIVE-5047) Hive client filters partitions incorrectly via pushdown in certain cases involving or
[ https://issues.apache.org/jira/browse/HIVE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-5047: --- Status: Patch Available (was: Open) Hive client filters partitions incorrectly via pushdown in certain cases involving or --- Key: HIVE-5047 URL: https://issues.apache.org/jira/browse/HIVE-5047 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-5047.D12141.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5047) Hive client filters partitions incorrectly via pushdown in certain cases involving or
[ https://issues.apache.org/jira/browse/HIVE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738520#comment-13738520 ] Sergey Shelukhin commented on HIVE-5047: I didn't press submit patch. Pressing now... Hive client filters partitions incorrectly via pushdown in certain cases involving or --- Key: HIVE-5047 URL: https://issues.apache.org/jira/browse/HIVE-5047 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-5047.D12141.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[ https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738529#comment-13738529 ] Sergey Shelukhin commented on HIVE-4985: Hive QA just ran. I ran all the tests some days ago, recently I ran the tests that failed on Hive QA refactor/clean up partition name pruning to be usable inside metastore server -- Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch, HIVE-4985.D11961.3.patch, HIVE-4985.D11961.4.patch, HIVE-4985.D11961.5.patch Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5023) Hive get wrong result when partition has the same path but different schema or authority
[
https://issues.apache.org/jira/browse/HIVE-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738537#comment-13738537
]
Shuaishuai Nie commented on HIVE-5023:
--
Thanks [~ashutoshc] [~vikram.dixit] [~sushanth]
Hive get wrong result when partition has the same path but different schema
or authority
Key: HIVE-5023
URL: https://issues.apache.org/jira/browse/HIVE-5023
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Fix For: 0.12.0
Attachments: HIVE-5023.1.patch, HIVE-5023.2.patch
Hive does not differentiate scheme and authority in file uris which cause
wrong result when partition has the same path but different schema or
authority. Here is a simple repro
partition file path:
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text1.txt
with content 2013-08-05 00:00:00
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text2.txt
with content 2013-08-05 00:00:20
{noformat}
CREATE EXTERNAL TABLE IF NOT EXISTS T1 (t STRING) PARTITIONED BY (ProcessDate
STRING, Hour STRING, ClusterName STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED by '\t' STORED AS TEXTFILE;
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='CLusterA');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterA') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='ClusterB');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterB') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
{noformat}
the expect output of the hive query
{noformat}
SELECT ClusterName, t FROM T1 WHERE ProcessDate=’2013-08-05’ AND Hour=’00’;
{noformat}
should be
{noformat}
ClusterA2013-08-05 00:00:00
ClusterB2013-08-05 00:00:20
{noformat}
However it is
{noformat}
ClusterA2013-08-05 00:00:00
ClusterA2013-08-05 00:00:20
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5068) Some queries fail due to XMLEncoder error on JDK7
[
https://issues.apache.org/jira/browse/HIVE-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738543#comment-13738543
]
Brock Noland commented on HIVE-5068:
The example I am workign with is auto_join25.q
Some queries fail due to XMLEncoder error on JDK7
-
Key: HIVE-5068
URL: https://issues.apache.org/jira/browse/HIVE-5068
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Looks like something snuck in that breaks the JDK 7 build:
{noformat}
Caused by: java.lang.Exception: XMLEncoder: discarding statement
ArrayList.add(ASTNode);
... 106 more
Caused by: java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at
java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:238)
at
java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:400)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:118)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeObject1(Encoder.java:258)
at java.beans.Encoder.cloneStatement(Encoder.java:271)
at java.beans.Encoder.writeStatement(Encoder.java:301)
at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:400)
... 105 more
Caused by: java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at java.beans.Encoder.getValue(Encoder.java:108)
at java.beans.Encoder.get(Encoder.java:252)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:112)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115)
at java.beans.Encoder.writeObject(Encoder.java:74)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327)
at java.beans.Encoder.writeExpression(Encoder.java:330)
at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454)
at
java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:232)
... 118 more
Caused by: java.lang.InstantiationException: org.antlr.runtime.CommonToken
at java.lang.Class.newInstance(Class.java:359)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at java.beans.Statement.invokeInternal(Statement.java:292)
at java.beans.Statement.access$000(Statement.java:58)
at java.beans.Statement$2.run(Statement.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at java.beans.Statement.invoke(Statement.java:182)
at java.beans.Expression.getValue(Expression.java:153)
at java.beans.Encoder.getValue(Encoder.java:105)
... 130 more
{noformat}
and
{noformat}
java.lang.RuntimeException: Cannot serialize object
at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:598)
at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:426)
at java.beans.XMLEncoder.writeObject(XMLEncoder.java:330)
at
org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:611)
at org.apache.hadoop.hive.ql.plan.MapredWork.toXML(MapredWork.java:88)
at
org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:505)
at
[jira] [Updated] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[ https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4985: --- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Sergey! refactor/clean up partition name pruning to be usable inside metastore server -- Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.12.0 Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch, HIVE-4985.D11961.3.patch, HIVE-4985.D11961.4.patch, HIVE-4985.D11961.5.patch Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Proposing a 0.11.1
+ d...@bigtop.apache.org Hi Owen, I work on Apache Bigtop http://bigtop.apache.org and we were recently discussinghttp://search-hadoop.com/m/A8Jne2SAnHq1/bigtop+0.7+bomsubj=Re+DISCUSS+BOM+for+release+0+7+0+of+Bigtopinclusion of Hive 0.11 in the next release of Bigtop - Bigtop 0.7. However, we learned during that discussion that there is a request to remove the NPath UDF from Hive code (ASF Board minutes available herehttp://www.apache.org/foundation/records/minutes/2013/board_minutes_2013_06_19.txt ). I was wondering if Hive 0.11.1 release would address that? Or, if a later Hive release would or if the Hive PMC has determined that there is no issue to be addressed afterall? Thanks! Mark On Tue, Aug 13, 2013 at 10:12 AM, Konstantin Boudnik c...@apache.org wrote: Hi. Can I suggest to include HIVE-3772 (performance fix; committed to trunk) HIVE-4583 (OpenJDK7 support; all subtasks seem to be done) Thanks, Cos On Tue, Aug 13, 2013 at 10:02AM, Owen O'Malley wrote: All, I'd like to create an 0.11.1 with some fixes in it. I plan to put together a release candidate over the next week. I'm in the process of putting together the list of bugs that I want to include, but I wanted to solicit the jiras that others though would be important for an 0.11.1. Thanks, Owen
[jira] [Commented] (HIVE-4705) PreExecutePrinter, EnforceReadOnlyTables, PostExecutePrinter should be included in ql
[ https://issues.apache.org/jira/browse/HIVE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738568#comment-13738568 ] Ashutosh Chauhan commented on HIVE-4705: Was there any specific issue you run into because of this? PreExecutePrinter, EnforceReadOnlyTables, PostExecutePrinter should be included in ql - Key: HIVE-4705 URL: https://issues.apache.org/jira/browse/HIVE-4705 Project: Hive Issue Type: Test Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-4705.D11205.1.patch Currently included in ql-test but is referenced from tests in other modules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5075) bug in ExprProcFactory.genPruner
Sergey Shelukhin created HIVE-5075: -- Summary: bug in ExprProcFactory.genPruner Key: HIVE-5075 URL: https://issues.apache.org/jira/browse/HIVE-5075 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin The implications are not clear to me yet, I am just familiarizing myself with code around these parts, but ExprProcFactory.genPruner intends to return a value (indicating whether there are non-partition columns) via a primitive boolean parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-5075) bug in ExprProcFactory.genPruner
[ https://issues.apache.org/jira/browse/HIVE-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-5075: -- Assignee: Sergey Shelukhin bug in ExprProcFactory.genPruner Key: HIVE-5075 URL: https://issues.apache.org/jira/browse/HIVE-5075 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin The implications are not clear to me yet, I am just familiarizing myself with code around these parts, but ExprProcFactory.genPruner intends to return a value (indicating whether there are non-partition columns) via a primitive boolean parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
