[jira] [Commented] (HIVE-10180) Loop optimization for SIMD in ColumnArithmeticColumn.txt
[
https://issues.apache.org/jira/browse/HIVE-10180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491819#comment-14491819
]
Chengxiang Li commented on HIVE-10180:
--
Committed to trunk, thanks Gopal for review.
Loop optimization for SIMD in ColumnArithmeticColumn.txt
Key: HIVE-10180
URL: https://issues.apache.org/jira/browse/HIVE-10180
Project: Hive
Issue Type: Sub-task
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Priority: Minor
Attachments: HIVE-10180.1.patch, HIVE-10180.2.patch
JVM is quite strict on the code schema which may executed with SIMD
instructions, take a loop in DoubleColAddDoubleColumn.java for example,
{code:java}
for (int i = 0; i != n; i++) {
outputVector[i] = vector1[0] + vector2[i];
}
{code}
The vector1[0] reference would prevent JVM to execute this part of code
with vectorized instructions, we need to assign the vector1[0] to a
variable outside of loop, and use that variable in loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10148) update of bucking column should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491845#comment-14491845 ] Lefty Leverenz commented on HIVE-10148: --- Doc note: This is already documented in the Update section of the DML wiki, but perhaps it should also be mentioned in Hive Transactions. * [DML -- Update -- Synopsis | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Synopsis.4] * [Hive Transactions -- Limitations | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations] And I recommend fixing the typo in the Summary (bucking column) to make JIRA searches easier. update of bucking column should not be allowed -- Key: HIVE-10148 URL: https://issues.apache.org/jira/browse/HIVE-10148 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.1.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 1.2.0 Attachments: HIVE-10148.2.patch, HIVE-10148.3.patch, HIVE-10148.4.patch, HIVE-10148.5.patch, HIVE-10148.6.patch, HIVE-10148.patch update tbl set a = 5; should raise an error if 'a' is a bucketing column. Such operation is not supported but currently not checked for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10062) HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
[
https://issues.apache.org/jira/browse/HIVE-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491817#comment-14491817
]
Hive QA commented on HIVE-10062:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724846/HIVE-10062.04.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8674 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3401/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3401/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3401/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724846 - PreCommit-HIVE-TRUNK-Build
HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
-
Key: HIVE-10062
URL: https://issues.apache.org/jira/browse/HIVE-10062
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Priority: Critical
Attachments: HIVE-10062.01.patch, HIVE-10062.02.patch,
HIVE-10062.03.patch, HIVE-10062.04.patch
In q.test environment with src table, execute the following query:
{code}
CREATE TABLE DEST1(key STRING, value STRING) STORED AS TEXTFILE;
CREATE TABLE DEST2(key STRING, val1 STRING, val2 STRING) STORED AS TEXTFILE;
FROM (select 'tst1' as key, cast(count(1) as string) as value from src s1
UNION all
select s2.key as key, s2.value as value from src s2) unionsrc
INSERT OVERWRITE TABLE DEST1 SELECT unionsrc.key, COUNT(DISTINCT
SUBSTR(unionsrc.value,5)) GROUP BY unionsrc.key
INSERT OVERWRITE TABLE DEST2 SELECT unionsrc.key, unionsrc.value,
COUNT(DISTINCT SUBSTR(unionsrc.value,5))
GROUP BY unionsrc.key, unionsrc.value;
select * from DEST1;
select * from DEST2;
{code}
DEST1 and DEST2 should both have 310 rows. However, DEST2 only has 1 row
tst1500 1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3635) allow 't', 'T', '1', 'f', 'F', and '0' to be allowable true/false values for the boolean hive type
[ https://issues.apache.org/jira/browse/HIVE-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491890#comment-14491890 ] Lefty Leverenz commented on HIVE-3635: -- Doc note: *hive.lazysimple.extended_boolean_literal* is documented in the wiki in two places. * [Developer Guide -- Hive SerDe | https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-HiveSerDe] * [Configuration Properties -- hive.lazysimple.extended_boolean_literal | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.lazysimple.extended_boolean_literal] If the doc looks okay, we can remove the TODOC14 label from this jira. By the way, one typo in the parameter description was already fixed (Siimple) but two others remain (extened and camel caps for SerDe in LazySimpleSerde). They're fixed in the doc. allow 't', 'T', '1', 'f', 'F', and '0' to be allowable true/false values for the boolean hive type --- Key: HIVE-3635 URL: https://issues.apache.org/jira/browse/HIVE-3635 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 0.9.0 Reporter: Alexander Alten-Lorenz Assignee: Xuefu Zhang Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-3635.1.patch, HIVE-3635.2.patch, HIVE-3635.patch interpret t as true and f as false for boolean types. PostgreSQL exports represent it that way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10235) Loop optimization for SIMD in ColumnDivideColumn.txt
[
https://issues.apache.org/jira/browse/HIVE-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxiang Li updated HIVE-10235:
-
Attachment: HIVE-10235.1.patch
Test with Jmh VectorizationBench by the following command:
{code:actionscript}
java -jar hive-jmh/target/benchmarks.jar
org.apache.hive.benchmark.vectorization VectorizationBench -wi 3 -i 5 -f 1 -bm
avgt -tu ms
{code}
The performance result looks like
||Expressions||/w patch(ms)||/w/o patch(ms)||
|DoubleColDivideDoubleColumn|4033|6654|
|DoubleColDivideRepeatingDoubleColumn|1563|3048|
|LongColDivideLongColumn|7354|7561|
|LongColDivideRepeatingColumn|3161|3163|
For for double array division in loop, the packed instruction vdivpd is used
instead of vdivsd with patch applied, while there is no such instruction for
long division, so there is no improvement for long array division in loop.
Loop optimization for SIMD in ColumnDivideColumn.txt
Key: HIVE-10235
URL: https://issues.apache.org/jira/browse/HIVE-10235
Project: Hive
Issue Type: Sub-task
Components: Vectorization
Affects Versions: 1.1.0
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Priority: Minor
Attachments: HIVE-10235.1.patch
Found two loop which could be optimized for packed instruction set during
execution.
1. hasDivBy0 depends on the result of last loop, which prevent the loop be
executed vectorized.
{code:java}
for(int i = 0; i != n; i++) {
OperandType2 denom = vector2[i];
outputVector[i] = vector1[0] OperatorSymbol denom;
hasDivBy0 = hasDivBy0 || (denom == 0);
}
{code}
2. same as HIVE-10180, vector2\[0\] reference provent JVM optimizing loop
into packed instruction set.
{code:java}
for(int i = 0; i != n; i++) {
outputVector[i] = vector1[i] OperatorSymbol vector2[0];
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10304) Add deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10304: - Attachment: (was: HIVE-10304.3.patch) Add deprecation message to HiveCLI -- Key: HIVE-10304 URL: https://issues.apache.org/jira/browse/HIVE-10304 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 1.1.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10304.2.patch, HIVE-10304.patch As Beeline is now the recommended command line tool to Hive, we should add a message to HiveCLI to indicate that it is deprecated and redirect them to Beeline. This is not suggesting to remove HiveCLI for now, but just a helpful direction for user to know the direction to focus attention in Beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9408) Add hook interface so queries can be redacted before being placed in job.xml
[ https://issues.apache.org/jira/browse/HIVE-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287310#comment-14287310 ] Lefty Leverenz edited comment on HIVE-9408 at 4/13/15 5:14 AM: --- Doc note: This adds *hive.exec.query.redactor.hooks* to HiveConf.java, so it needs to be documented in the wiki. It could be put with other hook parameters or at the end of Query and DDL Execution (before SerDes and I/O). * [either put it after hive.exec.failure.hooks | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.exec.failure.hooks] * [... or just before SerDes and I/O | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-SerDesandI/O] Does this also need general documentation? If so, where? (And how about a release note?) was (Author: le...@hortonworks.com): Doc note: This adds *hive.exec.query.redactor.hooks* to HiveConf.java, so it needs to be documented in the wiki. It could be put with other hook parameters or at the end of Query and DDL Execution (before File Formats and I/O). * [either put it after hive.exec.failure.hooks | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.exec.failure.hooks] * [... or just before File Formats and I/O | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-FileFormatsandI/O] Does this also need general documentation? If so, where? (And how about a release note?) Add hook interface so queries can be redacted before being placed in job.xml Key: HIVE-9408 URL: https://issues.apache.org/jira/browse/HIVE-9408 Project: Hive Issue Type: Task Components: Query Processor Affects Versions: 0.15.0 Reporter: Brock Noland Assignee: Brock Noland Labels: TODOC15 Fix For: 1.1.0 Attachments: HIVE-9408.1.patch, HIVE-9408.2.patch, HIVE-9408.3.patch Today we take a query and place it in the job.xml file which is pushed to all nodes the query runs on. However it's possible the query contains sensitive information and should not directly be shown to users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10310) Support GROUPING() and GROUP_ID() in HIVE
[ https://issues.apache.org/jira/browse/HIVE-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanjiv singh updated HIVE-10310: Issue Type: New Feature (was: Wish) Support GROUPING() and GROUP_ID() in HIVE - Key: HIVE-10310 URL: https://issues.apache.org/jira/browse/HIVE-10310 Project: Hive Issue Type: New Feature Components: Parser, SQL Reporter: sanjiv singh Priority: Minor I have lots of queries using GROUPING() function. failing on hive , just because GROUPING() not supported in hive. See the Query below; SELECT fact_1_id, fact_2_id, GROUPING(fact_1_id) AS f1g, GROUPING(fact_2_id) AS f2g FROM dimension_tab GROUP BY CUBE (fact_1_id, fact_2_id) ORDER BY fact_1_id, fact_2_id; In order to run in HIVE all such queries, It need to be transformed to HIVE syntax. See below transformed query, compatible to hive. Equivalent have been derived using Case statement . SELECT fact_1_id, fact_2_id, (case when (GROUPING__ID 1) = 0 then 1 else 0 end) as f1g, (case when (GROUPING__ID 2) = 0 then 1 else 0 end) as f2g FROM dimension_tab GROUP BY fact_1_id, fact_2_id WITH CUBE ORDER BY fact_1_id, fact_2_id; It would be great if GROUPING() implemented in hive. I see two ways to do it 1) Handle it at parser level. 2) GROUPING() aggregate function to hive(recommended) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10269) HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491808#comment-14491808 ] Ferdinand Xu commented on HIVE-10269: - Seems the jenkins report is incorrect.There is no failed case in fact. HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor - Key: HIVE-10269 URL: https://issues.apache.org/jira/browse/HIVE-10269 Project: Hive Issue Type: Bug Components: Metastore Reporter: Gabor Liptak Assignee: Ferdinand Xu Attachments: HIVE-10269.patch Compiling trunk fails when building based on instructions in https://cwiki.apache.org/confluence/display/Hive/HowToContribute $ git status On branch trunk Your branch is up-to-date with 'origin/trunk'. nothing to commit, working directory clean $ mvn clean install -DskipTests -Phadoop-1 ...[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-metastore: Compilation failure: Compilation failure: [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6089,29] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] /tmp/hive/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[6090,35] cannot find symbol [ERROR] symbol: class JvmPauseMonitor [ERROR] location: package org.apache.hadoop.util [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hive-metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491811#comment-14491811 ] Ferdinand Xu commented on HIVE-10189: - Failed cases are irrelevant and the jenkins report is incorrect. Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.2.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. Current score is as follows: BenchmarkMode Samples Score Error Units o.a.h.b.v.VectorizationBench.DoubleAddDoubleExpr.bench avgt2 20719.882 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleAddLongExpr.bench avgt2 22216.747 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideDoubleExpr.benchavgt2 54319.682 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideLongExpr.bench avgt2 34774.870 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddDoubleExpr.bench avgt2 47144.954 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddLongExpr.bench avgt2 21483.787 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideDoubleExpr.bench avgt2 49765.990 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideLongExpr.benchavgt2 34117.538 ? NaN ns/op -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10304) Add deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10304: - Attachment: HIVE-10304.3.patch Add deprecation message to HiveCLI -- Key: HIVE-10304 URL: https://issues.apache.org/jira/browse/HIVE-10304 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 1.1.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10304.2.patch, HIVE-10304.3.patch, HIVE-10304.patch As Beeline is now the recommended command line tool to Hive, we should add a message to HiveCLI to indicate that it is deprecated and redirect them to Beeline. This is not suggesting to remove HiveCLI for now, but just a helpful direction for user to know the direction to focus attention in Beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9915) Allow specifying file format for managed tables
[ https://issues.apache.org/jira/browse/HIVE-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359998#comment-14359998 ] Lefty Leverenz edited comment on HIVE-9915 at 4/13/15 5:13 AM: --- Good idea, [~gopalv]. Thanks. The quadrant can be added to Configuration Properties and to the File Formats overview page (which is just a stub with links currently): * [Configuration Properties -- File Formats | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-FileFormats] * [File Formats | https://cwiki.apache.org/confluence/display/Hive/FileFormats] was (Author: leftylev): Good idea, [~gopalv]. Thanks. The quadrant can be added to Configuration Properties and to the File Formats overview page (which is just a stub with links currently): * [Configuration Properties -- File Formats and I/O | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-FileFormatsandI/O] * [File Formats | https://cwiki.apache.org/confluence/display/Hive/FileFormats] Allow specifying file format for managed tables --- Key: HIVE-9915 URL: https://issues.apache.org/jira/browse/HIVE-9915 Project: Hive Issue Type: Bug Components: File Formats Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Labels: TODOC1.2 Fix For: 1.2.0 Attachments: HIVE-9915.1.patch, HIVE-9915.2.patch We already allow setting a system wide default format. In some cases it's useful though to specify this only for managed tables, or distinguish external and managed via two variables. You might want to set a more efficient (than text) format for managed tables, but leave external to text (as they often are log files etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10189) Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization
[ https://issues.apache.org/jira/browse/HIVE-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491822#comment-14491822 ] Chengxiang Li commented on HIVE-10189: -- Committed to the trunk, thanks Ferdinad for this contribution. Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization Key: HIVE-10189 URL: https://issues.apache.org/jira/browse/HIVE-10189 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-10189.1.patch, HIVE-10189.2.patch, HIVE-10189.patch, avx-64.docx We should show the performance gain from SIMD optimization. Current score is as follows: BenchmarkMode Samples Score Error Units o.a.h.b.v.VectorizationBench.DoubleAddDoubleExpr.bench avgt2 20719.882 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleAddLongExpr.bench avgt2 22216.747 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideDoubleExpr.benchavgt2 54319.682 ? NaN ns/op o.a.h.b.v.VectorizationBench.DoubleDivideLongExpr.bench avgt2 34774.870 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddDoubleExpr.bench avgt2 47144.954 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongAddLongExpr.bench avgt2 21483.787 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideDoubleExpr.bench avgt2 49765.990 ? NaN ns/op o.a.h.b.v.VectorizationBench.LongDivideLongExpr.benchavgt2 34117.538 ? NaN ns/op -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10235) Loop optimization for SIMD in ColumnDivideColumn.txt
[
https://issues.apache.org/jira/browse/HIVE-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491944#comment-14491944
]
Hive QA commented on HIVE-10235:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724862/HIVE-10235.1.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8673 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3402/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3402/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724862 - PreCommit-HIVE-TRUNK-Build
Loop optimization for SIMD in ColumnDivideColumn.txt
Key: HIVE-10235
URL: https://issues.apache.org/jira/browse/HIVE-10235
Project: Hive
Issue Type: Sub-task
Components: Vectorization
Affects Versions: 1.1.0
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Priority: Minor
Attachments: HIVE-10235.1.patch
Found two loop which could be optimized for packed instruction set during
execution.
1. hasDivBy0 depends on the result of last loop, which prevent the loop be
executed vectorized.
{code:java}
for(int i = 0; i != n; i++) {
OperandType2 denom = vector2[i];
outputVector[i] = vector1[0] OperatorSymbol denom;
hasDivBy0 = hasDivBy0 || (denom == 0);
}
{code}
2. same as HIVE-10180, vector2\[0\] reference provent JVM optimizing loop
into packed instruction set.
{code:java}
for(int i = 0; i != n; i++) {
outputVector[i] = vector1[i] OperatorSymbol vector2[0];
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10304) Add deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10304: - Attachment: HIVE-10304.3.patch Add deprecation message to HiveCLI -- Key: HIVE-10304 URL: https://issues.apache.org/jira/browse/HIVE-10304 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 1.1.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-10304.2.patch, HIVE-10304.3.patch, HIVE-10304.patch As Beeline is now the recommended command line tool to Hive, we should add a message to HiveCLI to indicate that it is deprecated and redirect them to Beeline. This is not suggesting to remove HiveCLI for now, but just a helpful direction for user to know the direction to focus attention in Beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10148) update of bucking column should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491725#comment-14491725 ] Eugene Koifman commented on HIVE-10148: --- the set of failures is almost identical to HIVE-10242 but the patches are not related update of bucking column should not be allowed -- Key: HIVE-10148 URL: https://issues.apache.org/jira/browse/HIVE-10148 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.1.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HIVE-10148.2.patch, HIVE-10148.3.patch, HIVE-10148.4.patch, HIVE-10148.5.patch, HIVE-10148.6.patch, HIVE-10148.patch update tbl set a = 5; should raise an error if 'a' is a bucketing column. Such operation is not supported but currently not checked for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10242) ACID: insert overwrite prevents create table command
[ https://issues.apache.org/jira/browse/HIVE-10242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491723#comment-14491723 ] Eugene Koifman commented on HIVE-10242: --- the set of failures is almost identical to HIVE-10148 but the patches are not related [~alangates], could you review please? ACID: insert overwrite prevents create table command Key: HIVE-10242 URL: https://issues.apache.org/jira/browse/HIVE-10242 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HIVE-10242.2.patch, HIVE-10242.3.patch, HIVE-10242.patch 1. insert overwirte table DB.T1 select ... from T2: this takes X lock on DB.T1 and S lock on T2. X lock makes sense because we don't want anyone reading T1 while it's overwritten. S lock on T2 prevents if from being dropped while the query is in progress. 2. create table DB.T3: takes S lock on DB. This S lock gets blocked by X lock on T1. S lock prevents the DB from being dropped while create table is executed. If the insert statement is long running, this blocks DDL ops on the same database. This is a usability issue. There is no good reason why X lock on a table within a DB and S lock on DB should be in conflict. (this is different from a situation where X lock is on a partition and S lock is on the table to which this partition belongs. Here it makes sense. Basically there is no SQL way to address all tables in a DB but you can easily refer to all partitions of a table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10062) HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
[
https://issues.apache.org/jira/browse/HIVE-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-10062:
---
Attachment: HIVE-10062.04.patch
address test failures
HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
-
Key: HIVE-10062
URL: https://issues.apache.org/jira/browse/HIVE-10062
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Priority: Critical
Attachments: HIVE-10062.01.patch, HIVE-10062.02.patch,
HIVE-10062.03.patch, HIVE-10062.04.patch
In q.test environment with src table, execute the following query:
{code}
CREATE TABLE DEST1(key STRING, value STRING) STORED AS TEXTFILE;
CREATE TABLE DEST2(key STRING, val1 STRING, val2 STRING) STORED AS TEXTFILE;
FROM (select 'tst1' as key, cast(count(1) as string) as value from src s1
UNION all
select s2.key as key, s2.value as value from src s2) unionsrc
INSERT OVERWRITE TABLE DEST1 SELECT unionsrc.key, COUNT(DISTINCT
SUBSTR(unionsrc.value,5)) GROUP BY unionsrc.key
INSERT OVERWRITE TABLE DEST2 SELECT unionsrc.key, unionsrc.value,
COUNT(DISTINCT SUBSTR(unionsrc.value,5))
GROUP BY unionsrc.key, unionsrc.value;
select * from DEST1;
select * from DEST2;
{code}
DEST1 and DEST2 should both have 310 rows. However, DEST2 only has 1 row
tst1500 1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10012) LLAP: Hive sessions run before Slider registers to YARN registry fail to launch
[
https://issues.apache.org/jira/browse/HIVE-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-10012:
---
Attachment: HIVE-10220.2.patch
LLAP: Hive sessions run before Slider registers to YARN registry fail to
launch
---
Key: HIVE-10012
URL: https://issues.apache.org/jira/browse/HIVE-10012
Project: Hive
Issue Type: Sub-task
Affects Versions: llap
Reporter: Gopal V
Assignee: Gopal V
Fix For: llap
Attachments: HIVE-10012.1.patch, HIVE-10012.wip1.patch
The LLAP YARN registry only registers entries after at least one daemon is up.
Any Tez session starting before that will end up with an error listing
zookeeper directories.
{code}
2015-03-18 16:54:21,392 FATAL [main] app.DAGAppMaster: Error starting
DAGAppMaster
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.fs.PathNotFoundException:
`/users/sershe/services/org-apache-hive/llap0/components/workers':
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10012) LLAP: Hive sessions run before Slider registers to YARN registry fail to launch
[
https://issues.apache.org/jira/browse/HIVE-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-10012:
---
Attachment: (was: HIVE-10220.2.patch)
LLAP: Hive sessions run before Slider registers to YARN registry fail to
launch
---
Key: HIVE-10012
URL: https://issues.apache.org/jira/browse/HIVE-10012
Project: Hive
Issue Type: Sub-task
Affects Versions: llap
Reporter: Gopal V
Assignee: Gopal V
Fix For: llap
Attachments: HIVE-10012.1.patch, HIVE-10012.wip1.patch
The LLAP YARN registry only registers entries after at least one daemon is up.
Any Tez session starting before that will end up with an error listing
zookeeper directories.
{code}
2015-03-18 16:54:21,392 FATAL [main] app.DAGAppMaster: Error starting
DAGAppMaster
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.fs.PathNotFoundException:
`/users/sershe/services/org-apache-hive/llap0/components/workers':
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10310) Support GROUPING() in HIVE
[ https://issues.apache.org/jira/browse/HIVE-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanjiv singh updated HIVE-10310: Description: I have lots of queries using GROUPING() function. failing on hive , just because GROUPING() not supported in hive. See the Query below; SELECT fact_1_id, fact_2_id, GROUPING(fact_1_id) AS f1g, GROUPING(fact_2_id) AS f2g FROM dimension_tab GROUP BY CUBE (fact_1_id, fact_2_id) ORDER BY fact_1_id, fact_2_id; In order to run in HIVE all such queries, It need to be transformed to HIVE syntax. See below transformed query, compatible to hive. Equivalent have been derived using Case statement . SELECT fact_1_id, fact_2_id, (case when (GROUPING__ID 1) = 0 then 1 else 0 end) as f1g, (case when (GROUPING__ID 2) = 0 then 1 else 0 end) as f2g FROM dimension_tab GROUP BY fact_1_id, fact_2_id WITH CUBE ORDER BY fact_1_id, fact_2_id; It would be great if GROUPING() implemented in hive. I see two ways to do it 1) Handle it at parser level. 2) GROUPING() aggregate function to hive(recommended) was: I have lots of queries using GROUPING() function. failing on hive , just because GROUPING() not supported in hive. See the Query below; SELECT fact_1_id, fact_2_id, GROUPING(fact_1_id) AS f1g, GROUPING(fact_2_id) AS f2g FROM dimension_tab GROUP BY CUBE (fact_1_id, fact_2_id) ORDER BY fact_1_id, fact_2_id; In order to run in HIVE all such queries, It need to be transformed to HIVE syntax. See below transformed query, compatible to hive. Equivalent have been derived using Case statement . SELECT fact_1_id, fact_2_id, (case when (GROUPING__ID 1) = 0 then 1 else 0 end) as f1g, (case when (GROUPING__ID 2) = 0 then 1 else 0 end) as f2g FROM dimension_tab GROUP BY fact_1_id, fact_2_id WITH CUBE ORDER BY fact_1_id, fact_2_id; It would be great if GROUPING() implemented in hive. I see two ways to do it 1) Handle it parser level. 2) GROUPING() aggregate function to hive(recommended) Support GROUPING() in HIVE -- Key: HIVE-10310 URL: https://issues.apache.org/jira/browse/HIVE-10310 Project: Hive Issue Type: Wish Components: Parser, SQL Reporter: sanjiv singh Priority: Minor I have lots of queries using GROUPING() function. failing on hive , just because GROUPING() not supported in hive. See the Query below; SELECT fact_1_id, fact_2_id, GROUPING(fact_1_id) AS f1g, GROUPING(fact_2_id) AS f2g FROM dimension_tab GROUP BY CUBE (fact_1_id, fact_2_id) ORDER BY fact_1_id, fact_2_id; In order to run in HIVE all such queries, It need to be transformed to HIVE syntax. See below transformed query, compatible to hive. Equivalent have been derived using Case statement . SELECT fact_1_id, fact_2_id, (case when (GROUPING__ID 1) = 0 then 1 else 0 end) as f1g, (case when (GROUPING__ID 2) = 0 then 1 else 0 end) as f2g FROM dimension_tab GROUP BY fact_1_id, fact_2_id WITH CUBE ORDER BY fact_1_id, fact_2_id; It would be great if GROUPING() implemented in hive. I see two ways to do it 1) Handle it at parser level. 2) GROUPING() aggregate function to hive(recommended) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10310) Support GROUPING() and GROUP_ID() in HIVE
[ https://issues.apache.org/jira/browse/HIVE-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanjiv singh updated HIVE-10310: Summary: Support GROUPING() and GROUP_ID() in HIVE (was: Support GROUPING() in HIVE) Support GROUPING() and GROUP_ID() in HIVE - Key: HIVE-10310 URL: https://issues.apache.org/jira/browse/HIVE-10310 Project: Hive Issue Type: Wish Components: Parser, SQL Reporter: sanjiv singh Priority: Minor I have lots of queries using GROUPING() function. failing on hive , just because GROUPING() not supported in hive. See the Query below; SELECT fact_1_id, fact_2_id, GROUPING(fact_1_id) AS f1g, GROUPING(fact_2_id) AS f2g FROM dimension_tab GROUP BY CUBE (fact_1_id, fact_2_id) ORDER BY fact_1_id, fact_2_id; In order to run in HIVE all such queries, It need to be transformed to HIVE syntax. See below transformed query, compatible to hive. Equivalent have been derived using Case statement . SELECT fact_1_id, fact_2_id, (case when (GROUPING__ID 1) = 0 then 1 else 0 end) as f1g, (case when (GROUPING__ID 2) = 0 then 1 else 0 end) as f2g FROM dimension_tab GROUP BY fact_1_id, fact_2_id WITH CUBE ORDER BY fact_1_id, fact_2_id; It would be great if GROUPING() implemented in hive. I see two ways to do it 1) Handle it at parser level. 2) GROUPING() aggregate function to hive(recommended) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491374#comment-14491374
]
Hive QA commented on HIVE-9645:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724693/HIVE-9645.7.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8672 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3391/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3391/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3391/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724693 - PreCommit-HIVE-TRUNK-Build
Constant folding case NULL equality
---
Key: HIVE-9645
URL: https://issues.apache.org/jira/browse/HIVE-9645
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Gopal V
Assignee: Ashutosh Chauhan
Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.3.patch,
HIVE-9645.4.patch, HIVE-9645.5.patch, HIVE-9645.6.patch, HIVE-9645.7.patch,
HIVE-9645.patch
Hive logical optimizer does not follow the Null scan codepath when
encountering a NULL = 1;
NULL = 1 is not evaluated as false in the constant propogation implementation.
{code}
hive explain select count(1) from store_sales where null=1;
...
TableScan
alias: store_sales
filterExpr: (null = 1) (type: boolean)
Statistics: Num rows: 550076554 Data size: 49570324480
Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (null = 1) (type: boolean)
Statistics: Num rows: 275038277 Data size: 0 Basic stats:
PARTIAL Column stats: COMPLETE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9937) LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join
[ https://issues.apache.org/jira/browse/HIVE-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491441#comment-14491441 ] Matt McCline commented on HIVE-9937: The did not produce a TEST-*.xml stuff for TestMinimrCliDriver is occuring in other submits, so it is unrelated to this patch. LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join -- Key: HIVE-9937 URL: https://issues.apache.org/jira/browse/HIVE-9937 Project: Hive Issue Type: Sub-task Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-9937.01.patch, HIVE-9937.02.patch, HIVE-9937.03.patch, HIVE-9937.04.patch, HIVE-9937.05.patch, HIVE-9937.06.patch, HIVE-9937.07.patch, HIVE-9937.08.patch, HIVE-9937.09.patch, HIVE-9937.91.patch, HIVE-9937.92.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9937) LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join
[
https://issues.apache.org/jira/browse/HIVE-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491429#comment-14491429
]
Hive QA commented on HIVE-9937:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724707/HIVE-9937.92.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8686 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3393/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3393/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724707 - PreCommit-HIVE-TRUNK-Build
LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new
Vectorized Map Join
--
Key: HIVE-9937
URL: https://issues.apache.org/jira/browse/HIVE-9937
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-9937.01.patch, HIVE-9937.02.patch,
HIVE-9937.03.patch, HIVE-9937.04.patch, HIVE-9937.05.patch,
HIVE-9937.06.patch, HIVE-9937.07.patch, HIVE-9937.08.patch,
HIVE-9937.09.patch, HIVE-9937.91.patch, HIVE-9937.92.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9557) create UDF to measure strings similarity using Cosine Similarity algo
[
https://issues.apache.org/jira/browse/HIVE-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491457#comment-14491457
]
Bruno P. Kinoshita commented on HIVE-9557:
--
Hi,
We are implementing several string similarity algorithms in the sandbox
component [Commons-Text|http://commons.apache.org/sandbox/commons-text/]. It
would be great if this UDF could be written using that component.
create UDF to measure strings similarity using Cosine Similarity algo
-
Key: HIVE-9557
URL: https://issues.apache.org/jira/browse/HIVE-9557
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
algo description http://en.wikipedia.org/wiki/Cosine_similarity
{code}
--one word different, total 2 words
str_sim_cosine('Test String1', 'Test String2') = (2 - 1) / 2 = 0.5f
{code}
reference implementation:
https://github.com/Simmetrics/simmetrics/blob/master/src/uk/ac/shef/wit/simmetrics/similaritymetrics/CosineSimilarity.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10036) Writing ORC format big table causes OOM - too many fixed sized stream buffers
[
https://issues.apache.org/jira/browse/HIVE-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491458#comment-14491458
]
Hive QA commented on HIVE-10036:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724721/HIVE-10036.5.patch
{color:red}ERROR:{color} -1 due to 316 failed/errored test(s), 8672 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
[jira] [Commented] (HIVE-10220) Disable all non-concurrent access to BytesBytesHashMap
[
https://issues.apache.org/jira/browse/HIVE-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491555#comment-14491555
]
Hive QA commented on HIVE-10220:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724745/HIVE-10220.2.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8664 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
TestSparkClient - did not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3396/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3396/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724745 - PreCommit-HIVE-TRUNK-Build
Disable all non-concurrent access to BytesBytesHashMap
--
Key: HIVE-10220
URL: https://issues.apache.org/jira/browse/HIVE-10220
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-10220.1.patch, HIVE-10220.2.patch
HybridHashTableContainer can end up being cached if it does not spill - that
needs to follow HIVE-10128 thread safety patterns for the partitioned hash
maps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491566#comment-14491566
]
Ashutosh Chauhan commented on HIVE-9645:
Failures are unrelated. Patch is good to go.
Constant folding case NULL equality
---
Key: HIVE-9645
URL: https://issues.apache.org/jira/browse/HIVE-9645
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Gopal V
Assignee: Ashutosh Chauhan
Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.3.patch,
HIVE-9645.4.patch, HIVE-9645.5.patch, HIVE-9645.6.patch, HIVE-9645.7.patch,
HIVE-9645.patch
Hive logical optimizer does not follow the Null scan codepath when
encountering a NULL = 1;
NULL = 1 is not evaluated as false in the constant propogation implementation.
{code}
hive explain select count(1) from store_sales where null=1;
...
TableScan
alias: store_sales
filterExpr: (null = 1) (type: boolean)
Statistics: Num rows: 550076554 Data size: 49570324480
Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (null = 1) (type: boolean)
Statistics: Num rows: 275038277 Data size: 0 Basic stats:
PARTIAL Column stats: COMPLETE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10285) Incorrect endFunction call in HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491579#comment-14491579
]
Ashutosh Chauhan commented on HIVE-10285:
-
+1
Incorrect endFunction call in HiveMetaStore
---
Key: HIVE-10285
URL: https://issues.apache.org/jira/browse/HIVE-10285
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0
Reporter: Nezih Yigitbasi
Priority: Minor
Attachments: HIVE-10285.patch
The HiveMetaStore.get_function() method ends with an incorrect call to the
endFunction() method. Instead of:
{code}
endFunction(get_database, func != null, ex);
{code}
It should call:
{code}
endFunction(get_function, func != null, ex);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10309) TestJdbcWithMiniHS2.java broken because of the removal of hive.server2.thrift.http.max.worker.threads
[
https://issues.apache.org/jira/browse/HIVE-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491610#comment-14491610
]
Hive QA commented on HIVE-10309:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724749/HIVE-10309.1.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8672 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3397/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3397/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724749 - PreCommit-HIVE-TRUNK-Build
TestJdbcWithMiniHS2.java broken because of the removal of
hive.server2.thrift.http.max.worker.threads
--
Key: HIVE-10309
URL: https://issues.apache.org/jira/browse/HIVE-10309
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-10309.1.patch
HIVE-10271 removed hive.server2.thrift.http.min/max.worker.threads
properties, however these properties are used in a few more places in hive
code. For example, TestJdbcWithMiniHS2.java . We need to fix these as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[ https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mubashir Kazia updated HIVE-10312: -- Attachment: HIVE-10312.1.patch SASL.QOP in JDBC URL is ignored for Delegation token Authentication --- Key: HIVE-10312 URL: https://issues.apache.org/jira/browse/HIVE-10312 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 1.2.0 Reporter: Mubashir Kazia Fix For: 1.2.0 Attachments: HIVE-10312.1.patch When HS2 is configured for QOP other than auth (auth-int or auth-conf), Kerberos client connection works fine when the JDBC URL specifies the matching QOP, however when this HS2 is accessed through Oozie (Delegation token / Digest authentication), connections fails because the JDBC driver ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be valid for DIGEST Auth mech. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10242) ACID: insert overwrite prevents create table command
[
https://issues.apache.org/jira/browse/HIVE-10242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491670#comment-14491670
]
Hive QA commented on HIVE-10242:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724771/HIVE-10242.3.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8678 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3398/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3398/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3398/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724771 - PreCommit-HIVE-TRUNK-Build
ACID: insert overwrite prevents create table command
Key: HIVE-10242
URL: https://issues.apache.org/jira/browse/HIVE-10242
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-10242.2.patch, HIVE-10242.3.patch, HIVE-10242.patch
1. insert overwirte table DB.T1 select ... from T2: this takes X lock on
DB.T1 and S lock on T2.
X lock makes sense because we don't want anyone reading T1 while it's
overwritten. S lock on T2 prevents if from being dropped while the query is
in progress.
2. create table DB.T3: takes S lock on DB.
This S lock gets blocked by X lock on T1. S lock prevents the DB from being
dropped while create table is executed.
If the insert statement is long running, this blocks DDL ops on the same
database. This is a usability issue.
There is no good reason why X lock on a table within a DB and S lock on DB
should be in conflict.
(this is different from a situation where X lock is on a partition and S lock
is on the table to which this partition belongs. Here it makes sense.
Basically there is no SQL way to address all tables in a DB but you can
easily refer to all partitions of a table)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10148) update of bucking column should not be allowed
[
https://issues.apache.org/jira/browse/HIVE-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491717#comment-14491717
]
Hive QA commented on HIVE-10148:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724779/HIVE-10148.6.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8673 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3399/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3399/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3399/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724779 - PreCommit-HIVE-TRUNK-Build
update of bucking column should not be allowed
--
Key: HIVE-10148
URL: https://issues.apache.org/jira/browse/HIVE-10148
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 1.1.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-10148.2.patch, HIVE-10148.3.patch,
HIVE-10148.4.patch, HIVE-10148.5.patch, HIVE-10148.6.patch, HIVE-10148.patch
update tbl set a = 5;
should raise an error if 'a' is a bucketing column.
Such operation is not supported but currently not checked for.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10312) SASL.QOP in JDBC URL is ignored for Delegation token Authentication
[
https://issues.apache.org/jira/browse/HIVE-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491751#comment-14491751
]
Hive QA commented on HIVE-10312:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12724832/HIVE-10312.1.patch
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 8672 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3400/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3400/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3400/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12724832 - PreCommit-HIVE-TRUNK-Build
SASL.QOP in JDBC URL is ignored for Delegation token Authentication
---
Key: HIVE-10312
URL: https://issues.apache.org/jira/browse/HIVE-10312
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 1.2.0
Reporter: Mubashir Kazia
Fix For: 1.2.0
Attachments: HIVE-10312.1.patch
When HS2 is configured for QOP other than auth (auth-int or auth-conf),
Kerberos client connection works fine when the JDBC URL specifies the
matching QOP, however when this HS2 is accessed through Oozie (Delegation
token / Digest authentication), connections fails because the JDBC driver
ignores the SASL.QOP parameters in the JDBC URL. SASL.QOP setting should be
valid for DIGEST Auth mech.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
