[jira] [Updated] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[ https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13196: --- Priority: Minor (was: Major) > UDFLike: reduce Regex NFA sizes > --- > > Key: HIVE-13196 > URL: https://issues.apache.org/jira/browse/HIVE-13196 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-13196.1.patch > > > The NFAs built from complex regexes in UDFLike are extremely complex and > spend a lot of time doing simple expression matching with no backtracking. > Prevent NFA -> DFA explosion by using reluctant regex matches instead of > greedy matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[
https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175218#comment-15175218
]
Gopal V commented on HIVE-13196:
Wrote a JMH bench, which explains this change -
https://github.com/t3rmin4t0r/regexbench
{code}
# Run complete. Total time: 00:00:41
Benchmark Mode CntScoreError Units
RegexBench.testGreedyRegexHit avgt5 340.991 ± 7.929 ns/op
RegexBench.testGreedyRegexMiss avgt5 466.184 ± 21.349 ns/op
RegexBench.testLazyRegexHit avgt5 72.456 ± 16.156 ns/op
RegexBench.testLazyRegexMissavgt5 366.955 ± 49.159 ns/op
{code}
> UDFLike: reduce Regex NFA sizes
> ---
>
> Key: HIVE-13196
> URL: https://issues.apache.org/jira/browse/HIVE-13196
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-13196.1.patch
>
>
> The NFAs built from complex regexes in UDFLike are extremely complex and
> spend a lot of time doing simple expression matching with no backtracking.
> Prevent NFA -> DFA explosion by using reluctant regex matches instead of
> greedy matches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[ https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13196: --- Status: Patch Available (was: Open) > UDFLike: reduce Regex NFA sizes > --- > > Key: HIVE-13196 > URL: https://issues.apache.org/jira/browse/HIVE-13196 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 2.0.0, 1.2.1, 1.3.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13196.1.patch > > > The NFAs built from complex regexes in UDFLike are extremely complex and > spend a lot of time doing simple expression matching with no backtracking. > Prevent NFA -> DFA explosion by using reluctant regex matches instead of > greedy matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[ https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13196: --- Attachment: HIVE-13196.1.patch > UDFLike: reduce Regex NFA sizes > --- > > Key: HIVE-13196 > URL: https://issues.apache.org/jira/browse/HIVE-13196 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13196.1.patch > > > The NFAs built from complex regexes in UDFLike are extremely complex and > spend a lot of time doing simple expression matching with no backtracking. > Prevent NFA -> DFA explosion by using reluctant regex matches instead of > greedy matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175202#comment-15175202
]
Hive QA commented on HIVE-11483:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790685/HIVE-11483.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9765 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_mapjoin1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7141/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7141/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7141/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790685 - PreCommit-HIVE-TRUNK-Build
> Add encoding and decoding for query string config
> -
>
> Key: HIVE-11483
> URL: https://issues.apache.org/jira/browse/HIVE-11483
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch,
> HIVE-11483.03.patch
>
>
> We have seen some queries in production where some of the literals passed in
> the query have control characters, which result in exception when query
> string is set in the job xml.
> Proposing a solution to encode the query string in configuration and provide
> getters decoded string.
> Here is a commit in a forked repo :
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> Suggestions are welcome on the solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13084) Vectorization throws exception where there is case statement in group by
[
https://issues.apache.org/jira/browse/HIVE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175191#comment-15175191
]
Rajesh Balamohan commented on HIVE-13084:
-
[~mmccline] - This works for simple if conditions but not for complex (e.g If
(condition, if(condition2 and condition3 or condition4)...)
> Vectorization throws exception where there is case statement in group by
>
>
> Key: HIVE-13084
> URL: https://issues.apache.org/jira/browse/HIVE-13084
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Rajesh Balamohan
>Assignee: Matt McCline
> Attachments: vector_between_date.q
>
>
> When there is case statement in group by, hive throws unable to vectorize
> exception.
> e.g query just to demonstrate the problem
> {noformat}
> explain select l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END as wk from lineitem_test_l_shipdate_ts
> group by l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END;
> org.apache.hadoop.hive.ql.metadata.HiveException: Could not vectorize
> expression: org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2
> File Output Operator [FS_7]
> Group By Operator [GBY_5] (rows=888777234 width=108)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=1777554469 width=108)
> Output:["_col0","_col1"],keys:_col0, _col1
> Select Operator [SEL_1] (rows=1777554469 width=108)
> Output:["_col0","_col1"]
> TableScan [TS_0] (rows=1777554469 width=108)
>
> rajesh@lineitem_test_l_shipdate_ts,lineitem_test_l_shipdate_ts,Tbl:COMPLETE,Col:NONE,Output:["l_partkey","l_commitdate"]
> {noformat}
> \cc [~mmccline], [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175117#comment-15175117
]
Alejandro Fernandez commented on HIVE-13063:
[~jdere], I updated the patch with those 2 tests.
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Status: Open (was: Patch Available)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: HIVE-13063.patch
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: (was: HIVE-13063.patch)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: Screen Shot 2016-02-17 at 7.20.57 PM.png, Screen Shot
> 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13188: Status: Patch Available (was: In Progress) > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13188: Attachment: HIVE-13188.02.patch > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175102#comment-15175102 ] Rajat Khandelwal commented on HIVE-13188: - Taking patch from reviewboard and attaching > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-11483: Attachment: HIVE-11483.03.patch > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175099#comment-15175099 ] Rajat Khandelwal commented on HIVE-11483: - Taking patch from reviewboard and attaching > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[
https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175015#comment-15175015
]
Hive QA commented on HIVE-13169:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790664/HIVE-13169.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9748 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-cte_4.q-orc_merge5.q-vectorization_limit.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7140/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7140/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7140/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790664 - PreCommit-HIVE-TRUNK-Build
> HiveServer2: Support delegation token based connection when using http
> transport
>
>
> Key: HIVE-13169
> URL: https://issues.apache.org/jira/browse/HIVE-13169
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Thejas M Nair
> Attachments: HIVE-13169.1.patch, HIVE-13169.2.patch,
> HIVE-13169.3.patch, HIVE-13169.3.patch, HIVE-13169.4.patch, HIVE-13169.5.patch
>
>
> HIVE-5155 introduced support for delegation token based connection. However,
> it was intended for tcp transport mode. We need to have similar mechanisms
> for http transport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13195) TPCDS Q75 doesnt generate optimal plan
[ https://issues.apache.org/jira/browse/HIVE-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174864#comment-15174864 ] Gopal V commented on HIVE-13195: [~ashutoshc]: can you upload the original .json as well, so that I can fix lipwig? > TPCDS Q75 doesnt generate optimal plan > -- > > Key: HIVE-13195 > URL: https://issues.apache.org/jira/browse/HIVE-13195 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan > Attachments: query75.master.svg, query75.sql.0.14.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13195) TPCDS Q75 doesnt generate optimal plan
[ https://issues.apache.org/jira/browse/HIVE-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13195: Attachment: query75.master.svg query75.sql.0.14.svg We were generating better plan on 0.14 Plans on 0.14 & master is attached. > TPCDS Q75 doesnt generate optimal plan > -- > > Key: HIVE-13195 > URL: https://issues.apache.org/jira/browse/HIVE-13195 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan > Attachments: query75.master.svg, query75.sql.0.14.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174841#comment-15174841 ] Ashutosh Chauhan commented on HIVE-13096: - This heuristic change should have supposedly impacted only selection of table chosen for streaming, and not change shape of tez dag. Is that expected ? > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.03.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13194: Description: Session has session Hive object stored in a field. Given that Hive object is not thread safe and is also taken from a threadlocal to start with, that is not a good idea. (was: Session has session Hive object stored in the field. Given that Hive object is not thread safe and is also taken from a threadlocal to start with, it's not a good idea.) > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13002:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master. Thanks!
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13002:
Fix Version/s: 2.1.0
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[ https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12995: Status: Patch Available (was: Open) Need to run it on the cluster just in case, but for now I am doing something else. > LLAP: Synthetic file ids need collision checks > -- > > Key: HIVE-12995 > URL: https://issues.apache.org/jira/browse/HIVE-12995 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12995.patch > > > LLAP synthetic file ids do not have any way of checking whether a collision > occurs other than a data-error. > Synthetic file-ids have only been used with unit tests so far - but they will > be needed to add cache mechanisms to non-HDFS filesystems. > In case of Synthetic file-ids, it is recommended that we track the full-tuple > (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id > can be compared against the parameters & only accepted if those match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[ https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12995: Attachment: HIVE-12995.patch This patch basically changes long to object everywhere. I also stumbled upon some high-level cache related classes and remove high-level cache support rather than changing them. > LLAP: Synthetic file ids need collision checks > -- > > Key: HIVE-12995 > URL: https://issues.apache.org/jira/browse/HIVE-12995 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12995.patch > > > LLAP synthetic file ids do not have any way of checking whether a collision > occurs other than a data-error. > Synthetic file-ids have only been used with unit tests so far - but they will > be needed to add cache mechanisms to non-HDFS filesystems. > In case of Synthetic file-ids, it is recommended that we track the full-tuple > (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id > can be compared against the parameters & only accepted if those match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174819#comment-15174819
]
Jason Dere commented on HIVE-13132:
---
It is true that HIVE-2573 still loads all permanent functions from all
databases during startup. One thing that might improve things is HIVE-10319,
which rather than calling once for every database, only makes a single call to
the metastore to retrieve all permanent functions. Can you check if this helps
at all?
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174799#comment-15174799
]
Ashutosh Chauhan commented on HIVE-13002:
-
Whether patch has fixed root cause, thats not clear but it certainly is an
improvement from state of art.
+1
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174800#comment-15174800 ] Ashutosh Chauhan commented on HIVE-13040: - I will take a look. > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[
https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174797#comment-15174797
]
Hive QA commented on HIVE-11675:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790631/HIVE-11675.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9765 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCachePpd
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7139/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7139/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7139/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790631 - PreCommit-HIVE-TRUNK-Build
> make use of file footer PPD API in ETL strategy or separate strategy
>
>
> Key: HIVE-11675
> URL: https://issues.apache.org/jira/browse/HIVE-11675
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch,
> HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch,
> HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch,
> HIVE-11675.patch
>
>
> Need to take a look at the best flow. It won't be much different if we do
> filtering metastore call for each partition. So perhaps we'd need the custom
> sync point/batching after all.
> Or we can make it opportunistic and not fetch any footers unless it can be
> pushed down to metastore or fetched from local cache, that way the only slow
> threaded op is directory listings
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[
https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174765#comment-15174765
]
Sergey Shelukhin commented on HIVE-13040:
-
cp of the comment in the other jira
It looks like spark tests get stuck because in some cases, 0 splits are
generated
{noformat}
2016-03-01T09:24:19,709 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO SparkPlan:
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - !!! Spark
Plan
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) -
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - MapTran 1 <-- ( MapInput 2 (cache off) )
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) -
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - !!! Spark
Plan
2016-03-01T09:24:19,711 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO Utilities: PLAN PATH =
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/550ff857-09c8-4e04-b4be-cb6084110e22/map.xml
2016-03-01T09:24:19,712 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
SerializationUtilities: Deserializing MapWork using kryo
2016-03-01T09:24:19,714 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check
non-combinable ones.
2016-03-01T09:24:19,717 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: CombineHiveInputSplit creating pool for
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/0;
using filter path
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/0
2016-03-01T09:24:19,722 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO FileInputFormat: Total
input paths to process : 1
2016-03-01T09:24:19,723 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: number of splits 0
2016-03-01T09:24:19,723 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: Number of all splits 0
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.KryoMessageCodec
(KryoMessageCodec.java:decode(98)) - Decoded message of type
org.apache.hive.spark.client.rpc.Rpc$MessageHeader (6 bytes)
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.KryoMessageCodec
(KryoMessageCodec.java:decode(98)) - Decoded message of type
org.apache.hive.spark.client.BaseProtocol$JobSubmitted (95 bytes)
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.RpcDispatcher
(RpcDispatcher.java:channelRead0(74)) - [ClientProtocol] Received RPC message:
type=CALL id=189 payload=org.apache.hive.spark.client.BaseProtocol$JobSubmitted
2016-03-01T09:24:19,725 INFO [RPC-Handler-3[]]: client.SparkClientImpl
(SparkClientImpl.java:handle(571)) - Received spark job ID: 36 for
5ec4a71b-de5c-4f01-bf41-dc67c4875e2e
{noformat}
Then it hangs forever reporting that the job is started, until it times out.
There's no logging at all after 09:24:15 in Spark logs, so I am assuming it's
some issue with Hive-on-Spark/HoS test when no splits are generated.
> Handle empty bucket creations more efficiently
> ---
>
> Key: HIVE-13040
> URL: https://issues.apache.org/jira/browse/HIVE-13040
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.1.0
>
> Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch,
> HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch,
> HIVE-13040.7.patch, HIVE-13040.patch
>
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174764#comment-15174764
]
Sergey Shelukhin commented on HIVE-13002:
-
Probably caused by HIVE-13040
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174758#comment-15174758 ] Sergey Shelukhin edited comment on HIVE-13040 at 3/2/16 1:11 AM: - https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20text%20~%20TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more%20%20ORDER%20BY%20updated%20DESC ... clicking thru the jiras shows bunch of unrelated (to spark) patches with multiple spark timeouts, the issue that first started happening here long before this was committed and any other JIRAs had it. I may be wrong but I think this might be related... cc [~ashutoshc] [~xuefuz] was (Author: sershe): https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20text%20~%20TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more%20%20ORDER%20BY%20updated%20DESC ... clicking thru the jiras shows bunch of unrelated (to spark) patches with the issue that first started happening here long before this was committed and any other JIRAs had it. I may be wrong but I think this might be related... cc [~ashutoshc] [~xuefuz] > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174758#comment-15174758 ] Sergey Shelukhin commented on HIVE-13040: - https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20text%20~%20TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more%20%20ORDER%20BY%20updated%20DESC ... clicking thru the jiras shows bunch of unrelated (to spark) patches with the issue that first started happening here long before this was committed and any other JIRAs had it. I may be wrong but I think this might be related... cc [~ashutoshc] [~xuefuz] > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174752#comment-15174752 ] Sergey Shelukhin commented on HIVE-13040: - I think spark failures might be related. This is the earliest committed JIRA after which these tests started timing out all together in all the JIRAs. The cause seems to be having 0 splits (see HIVE-13002 comment). Would this patch cause an issue like that? > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174745#comment-15174745
]
Sergey Shelukhin commented on HIVE-13002:
-
Nm, looks like it also happens in other JIRAs e.g. HIVE-13186, HIVE-13174 etc.
It also happens for me on master with join_empty test.
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13186) ALTER TABLE RENAME should lowercase table name and hdfs location
[ https://issues.apache.org/jira/browse/HIVE-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174736#comment-15174736 ] Jason Dere commented on HIVE-13186: --- +1 > ALTER TABLE RENAME should lowercase table name and hdfs location > > > Key: HIVE-13186 > URL: https://issues.apache.org/jira/browse/HIVE-13186 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13186.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174721#comment-15174721
]
Sergey Shelukhin commented on HIVE-13002:
-
It looks like spark tests get stuck because in some cases, 0 splits are
generated
{noformat}
2016-03-01T09:24:19,709 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO SparkPlan:
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - !!! Spark
Plan
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) -
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - MapTran 1 <-- ( MapInput 2 (cache off) )
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) -
2016-03-01T09:24:19,710 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - !!! Spark
Plan
2016-03-01T09:24:19,711 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO Utilities: PLAN PATH =
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/550ff857-09c8-4e04-b4be-cb6084110e22/map.xml
2016-03-01T09:24:19,712 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
SerializationUtilities: Deserializing MapWork using kryo
2016-03-01T09:24:19,714 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check
non-combinable ones.
2016-03-01T09:24:19,717 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: CombineHiveInputSplit creating pool for
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/0;
using filter path
file:/home/hiveptest/54.158.228.44-hiveptest-0/apache-github-source-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/894f65b8-0cb1-495f-9fa9-4948b8e482e1/hive_2016-03-01_09-24-18_701_9015622114754920872-1/-mr-10005/0
2016-03-01T09:24:19,722 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO FileInputFormat: Total
input paths to process : 1
2016-03-01T09:24:19,723 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: number of splits 0
2016-03-01T09:24:19,723 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(593)) - 16/03/01 09:24:19 INFO
CombineHiveInputFormat: Number of all splits 0
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.KryoMessageCodec
(KryoMessageCodec.java:decode(98)) - Decoded message of type
org.apache.hive.spark.client.rpc.Rpc$MessageHeader (6 bytes)
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.KryoMessageCodec
(KryoMessageCodec.java:decode(98)) - Decoded message of type
org.apache.hive.spark.client.BaseProtocol$JobSubmitted (95 bytes)
2016-03-01T09:24:19,725 DEBUG [RPC-Handler-3[]]: rpc.RpcDispatcher
(RpcDispatcher.java:channelRead0(74)) - [ClientProtocol] Received RPC message:
type=CALL id=189 payload=org.apache.hive.spark.client.BaseProtocol$JobSubmitted
2016-03-01T09:24:19,725 INFO [RPC-Handler-3[]]: client.SparkClientImpl
(SparkClientImpl.java:handle(571)) - Received spark job ID: 36 for
5ec4a71b-de5c-4f01-bf41-dc67c4875e2e
{noformat}
Then it hangs forever reporting that the job is started, until it times out.
There's no logging at all after 09:24:15 in Spark logs, so I am assuming it's
some issue with Hive-on-Spark/HoS test when no splits are generated.
Looking at why this could be the case.
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
[jira] [Commented] (HIVE-13186) ALTER TABLE RENAME should lowercase table name and hdfs location
[ https://issues.apache.org/jira/browse/HIVE-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174718#comment-15174718 ] Wei Zheng commented on HIVE-13186: -- Test failures don't fail locally on my laptop, and they don't seem related to the change. [~jdere] Can you take a look? > ALTER TABLE RENAME should lowercase table name and hdfs location > > > Key: HIVE-13186 > URL: https://issues.apache.org/jira/browse/HIVE-13186 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13186.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174592#comment-15174592 ] Ashutosh Chauhan commented on HIVE-12994: - +1 > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Fix Version/s: 2.1.0 Status: Patch Available (was: Open) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Attachment: HIVE-5370.2.patch Updated patch to master and verified all affected tests > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Target Version/s: 2.1.0 Status: Open (was: Patch Available) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.patch, > HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13174) Remove Vectorizer noise in logs
[
https://issues.apache.org/jira/browse/HIVE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-13174:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.0
1.3.0
Status: Resolved (was: Patch Available)
Patch pushed to master and branch-1.
> Remove Vectorizer noise in logs
> ---
>
> Key: HIVE-13174
> URL: https://issues.apache.org/jira/browse/HIVE-13174
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13174.patch
>
>
> If you have a table with a bin column you're hs2/client logs are full of the
> stack traces below. These should either be made debug or we just log the
> message not the trace.
> {code}
> 2015-10-12 12:34:23,922 INFO [main]: physical.Vectorizer
> (Vectorizer.java:validateExprNodeDesc(1249)) - Failed to vectorize
> org.apache.hadoop.hive.ql.metadata.HiveException: No vector argument type for
> type name binary
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getConstantVectorExpression(VectorizationContext.java:872)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:443)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1243)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1234)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateSelectOperator(Vectorizer.java:1100)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateMapWorkOperator(Vectorizer.java:911)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkValidationNodeProcessor.process(Vectorizer.java:581)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateMapWork(Vectorizer.java:412)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:355)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:330)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:890)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:469)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10188)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12558: - Status: Patch Available (was: Open) > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.wip.patch, sample-output.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174537#comment-15174537 ] Ashutosh Chauhan commented on HIVE-13069: - +1 > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13186) ALTER TABLE RENAME should lowercase table name and hdfs location
[
https://issues.apache.org/jira/browse/HIVE-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174529#comment-15174529
]
Hive QA commented on HIVE-13186:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790628/HIVE-13186.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9780 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7138/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7138/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7138/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790628 - PreCommit-HIVE-TRUNK-Build
> ALTER TABLE RENAME should lowercase table name and hdfs location
>
>
> Key: HIVE-13186
> URL: https://issues.apache.org/jira/browse/HIVE-13186
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13186.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174508#comment-15174508 ] Sergey Shelukhin commented on HIVE-12558: - To us, yes, but not to anyone who is not a Hive/LLAP developer. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.wip.patch, sample-output.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174481#comment-15174481 ] Pengcheng Xiong commented on HIVE-13095: [~leftylev], if we treat an incomplete feature as a bug, then it is both a bug and a new feature. :) > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
[
https://issues.apache.org/jira/browse/HIVE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13192:
-
Attachment: HIVE-13192.1.patch
[~jdere] / [~wzheng] can you please look at the change and see if it makes
sense.
Thanks
Hari
> OOM error when MapJoinBytesTableContainer memory threshold is too low
> --
>
> Key: HIVE-13192
> URL: https://issues.apache.org/jira/browse/HIVE-13192
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13192.1.patch
>
>
> HIVE-11449 covered the scenario with HybridHashTableContainer, but we can
> still run into the error as shown below:
> {code}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
> diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
> at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:114)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:190)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:216)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:316)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
[
https://issues.apache.org/jira/browse/HIVE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13192:
-
Status: Patch Available (was: Open)
> OOM error when MapJoinBytesTableContainer memory threshold is too low
> --
>
> Key: HIVE-13192
> URL: https://issues.apache.org/jira/browse/HIVE-13192
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13192.1.patch
>
>
> HIVE-11449 covered the scenario with HybridHashTableContainer, but we can
> still run into the error as shown below:
> {code}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
> diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
> at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:114)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:190)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:216)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:316)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
[
https://issues.apache.org/jira/browse/HIVE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13192:
-
Description:
HIVE-11449 covered the scenario with HybridHashTableContainer, but we can still
run into the error as shown below:
{code}
Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
at org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
at
org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
at
org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
at
org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:114)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:190)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:216)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:316)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
... 13 more
{code}
was:
HIVE-11449 covered the scenario with HybridHashTableContainer, but we can still
the error as shown below:
{code}
Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174452#comment-15174452
]
Aihua Xu commented on HIVE-13149:
-
Attached the patch-2: to fix the issue observed from the unit tests. We need to
pass in a copy of conf to the Hive instance since otherwise it's sharing with
the session one.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Attachment: HIVE-13149.2.patch
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13146) OrcFile table property values are case sensitive
[ https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174420#comment-15174420 ] Andrew Sears commented on HIVE-13146: - Just a bug Lefty, case closed > OrcFile table property values are case sensitive > > > Key: HIVE-13146 > URL: https://issues.apache.org/jira/browse/HIVE-13146 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Andrew Sears >Assignee: Yongzhi Chen >Priority: Minor > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch, > HIVE-13146.3.patch > > > In Hive v1.2.1.2.3, with Tez , create an external table with compression > SNAPPY value marked as lower case. Table is created successfully. Insert > data into table fails with no enum constant error. > CREATE EXTERNAL TABLE mydb.mytable > (id int) > PARTITIONED BY (business_date date) > STORED AS ORC > LOCATION > '/data/mydb/mytable' > TBLPROPERTIES ( > 'orc.compress'='snappy'); > set hive.exec.dynamic.partition=true; > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE mydb.mytable PARTITION (business_date) > SELECT * from mydb.sourcetable; > Caused by: java.lang.IllegalArgumentException: No enum constant > org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy > at java.lang.Enum.valueOf(Enum.java:238) > at > org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25) > Constant SNAPPY needs to be uppercase in definition to fix. Case should be > agnostic or throw error on creation of table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13146) OrcFile table property values are case sensitive
[ https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174406#comment-15174406 ] Lefty Leverenz commented on HIVE-13146: --- Should this be documented in the wiki, in which case* it needs a TODOC1.3 label, or is it just a bug fix? *pun intended > OrcFile table property values are case sensitive > > > Key: HIVE-13146 > URL: https://issues.apache.org/jira/browse/HIVE-13146 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Andrew Sears >Assignee: Yongzhi Chen >Priority: Minor > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch, > HIVE-13146.3.patch > > > In Hive v1.2.1.2.3, with Tez , create an external table with compression > SNAPPY value marked as lower case. Table is created successfully. Insert > data into table fails with no enum constant error. > CREATE EXTERNAL TABLE mydb.mytable > (id int) > PARTITIONED BY (business_date date) > STORED AS ORC > LOCATION > '/data/mydb/mytable' > TBLPROPERTIES ( > 'orc.compress'='snappy'); > set hive.exec.dynamic.partition=true; > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE mydb.mytable PARTITION (business_date) > SELECT * from mydb.sourcetable; > Caused by: java.lang.IllegalArgumentException: No enum constant > org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy > at java.lang.Enum.valueOf(Enum.java:238) > at > org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25) > Constant SNAPPY needs to be uppercase in definition to fix. Case should be > agnostic or throw error on creation of table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13151) Clean up UGI objects in FileSystem cache for transactions
[ https://issues.apache.org/jira/browse/HIVE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13151: - Status: Patch Available (was: Open) > Clean up UGI objects in FileSystem cache for transactions > - > > Key: HIVE-13151 > URL: https://issues.apache.org/jira/browse/HIVE-13151 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13151.1.patch, HIVE-13151.2.patch, > HIVE-13151.3..patch > > > One issue with FileSystem.CACHE is that it does not clean itself. The key in > that cache includes UGI object. When new UGI objects are created and used > with the FileSystem api, new entries get added to the cache. > We need to manually clean up those UGI objects once they are no longer in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13151) Clean up UGI objects in FileSystem cache for transactions
[ https://issues.apache.org/jira/browse/HIVE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13151: - Status: Open (was: Patch Available) > Clean up UGI objects in FileSystem cache for transactions > - > > Key: HIVE-13151 > URL: https://issues.apache.org/jira/browse/HIVE-13151 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13151.1.patch, HIVE-13151.2.patch, > HIVE-13151.3..patch > > > One issue with FileSystem.CACHE is that it does not clean itself. The key in > that cache includes UGI object. When new UGI objects are created and used > with the FileSystem api, new entries get added to the cache. > We need to manually clean up those UGI objects once they are no longer in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12757) Fix TestCodahaleMetrics#testFileReporting
[ https://issues.apache.org/jira/browse/HIVE-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12757: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Test failures cant be related (as this is a test fix). Committed to master, thanks a lot Aihua for review. > Fix TestCodahaleMetrics#testFileReporting > - > > Key: HIVE-12757 > URL: https://issues.apache.org/jira/browse/HIVE-12757 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.1.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-12757.2.patch, HIVE-12757.patch > > > Codahale Metrics file reporter is time based, hence test is as well. On slow > machines, sometimes the file is not written fast enough to be read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-10632: - Attachment: HIVE-10632.6.patch patch 6 for test > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, HIVE-10632.6.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13069: --- Attachment: HIVE-13069.01.patch > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13069: --- Status: Patch Available (was: In Progress) > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13069: --- Status: Open (was: Patch Available) > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13069 started by Jesus Camacho Rodriguez. -- > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174270#comment-15174270
]
Gopal V commented on HIVE-13191:
With real tables, this isn't a problem.
{code}
hive> create temporary table x as select 11 as key, 1 confuse_you, 1 one, 0
zero;
hive> select a.key, a.one, b.one, a.zero, b.zero from x a left join x b on
a.key = b.key;
11 1 1 0 0
Time taken: 0.922 seconds, Fetched: 1 row(s)
{code}
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174243#comment-15174243
]
Hive QA commented on HIVE-13002:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790621/HIVE-13002.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9778 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7137/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7137/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7137/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790621 - PreCommit-HIVE-TRUNK-Build
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.03.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13175: - Attachment: HIVE-13175.2.patch > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13190) Vectorization: Dummy table row-limits get multiplied 100x accidentally
[
https://issues.apache.org/jira/browse/HIVE-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13190:
---
Description:
100x more rows are produced from dummy tables.
{code}
hive> select count(1) from (select * from (Select 1 a) x order by x.a) y;
100
Time taken: 0.913 seconds, Fetched: 1 row(s)
hive>
{code}
simpler example.
{code}
hive> create temporary table dual as select 1;
Table default.dual stats: [numFiles=1, numRows=100, totalSize=200,
rawDataSize=100]
OK
Time taken: 1.482 seconds
{code}
was:
100x more rows are produced from dummy tables.
{code}
hive> select count(1) from (select * from (Select 1 a) x order by x.a) y;
100
Time taken: 0.913 seconds, Fetched: 1 row(s)
hive>
{code}
> Vectorization: Dummy table row-limits get multiplied 100x accidentally
> --
>
> Key: HIVE-13190
> URL: https://issues.apache.org/jira/browse/HIVE-13190
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
>
> 100x more rows are produced from dummy tables.
> {code}
> hive> select count(1) from (select * from (Select 1 a) x order by x.a) y;
> 100
> Time taken: 0.913 seconds, Fetched: 1 row(s)
> hive>
> {code}
> simpler example.
> {code}
> hive> create temporary table dual as select 1;
> Table default.dual stats: [numFiles=1, numRows=100, totalSize=200,
> rawDataSize=100]
> OK
> Time taken: 1.482 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13145) Separate the output path of metrics file of HS2 and HMS
[
https://issues.apache.org/jira/browse/HIVE-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174148#comment-15174148
]
Szehon Ho commented on HIVE-13145:
--
Yea I was optimizing for simplicity by reducing number of flags in Hive. I
just am not sure if its worth to have another flag for this case unless you
feel strongly. Another concern is user might think that in embedded mode, HS2
and HMS metrics will go to two files which is not true?
> Separate the output path of metrics file of HS2 and HMS
> ---
>
> Key: HIVE-13145
> URL: https://issues.apache.org/jira/browse/HIVE-13145
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Metastore
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: HIVE-13145.1.patch, HIVE-13145.2.patch
>
>
> The output path of metrics file of HS2 and HMS can define by
> {{hive.service.metrics.file.location}} property at present.
> When it starts HS2 and HMS by the same server, both metrics is written in the
> same file. And when confirming this file, it is difficult to judge which
> metrics it is.
> Therefore the output path of metrics file of HS2 and HMS is separated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173975#comment-15173975
]
Gopal V commented on HIVE-13189:
I suspect another added fix would be to fold the args using preferred types.
{{date_add('2000-01-01', day_key);}} should not parse the date for every
iteration (the impl can compute that from the Const object inspectors during
initialize).
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173947#comment-15173947
]
Rajesh Balamohan commented on HIVE-13189:
-
JMH comparison of SimpleDateFormat vs Joda DateTimeFormatter
{noformat}
# JMH 1.11.2 (released 124 days ago, please consider updating!)
# VM version: JDK 1.8.0_05, VM 25.5-b02
# VM invoker:
/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/bin/java
# VM options:
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: org.apache.jmh.TestJodaVsSimpleDateFormat.testWithJodaTime
# Run progress: 0.00% complete, ETA 00:03:20
# Fork: 1 of 1
# Warmup Iteration 1: 395.761 ns/op
# Warmup Iteration 2: 396.304 ns/op
# Warmup Iteration 3: 388.342 ns/op
# Warmup Iteration 4: 407.058 ns/op
# Warmup Iteration 5: 392.305 ns/op
Iteration 1: 387.758 ns/op
Iteration 2: 419.816 ns/op
Iteration 3: 444.825 ns/op
Iteration 4: 435.538 ns/op
Iteration 5: 431.213 ns/op
Result "testWithJodaTime":
423.830 ±(99.9%) 85.014 ns/op [Average]
(min, avg, max) = (387.758, 423.830, 444.825), stdev = 22.078
CI (99.9%): [338.817, 508.844] (assumes normal distribution)
# JMH 1.11.2 (released 124 days ago, please consider updating!)
# VM version: JDK 1.8.0_05, VM 25.5-b02
# VM invoker:
/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/bin/java
# VM options:
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: org.apache.jmh.TestJodaVsSimpleDateFormat.testWithSimpleDateFormat
# Run progress: 50.00% complete, ETA 00:01:40
# Fork: 1 of 1
# Warmup Iteration 1: 847.271 ns/op
# Warmup Iteration 2: 839.440 ns/op
# Warmup Iteration 3: 840.931 ns/op
# Warmup Iteration 4: 819.619 ns/op
# Warmup Iteration 5: 838.692 ns/op
Iteration 1: 845.421 ns/op
Iteration 2: 857.534 ns/op
Iteration 3: 857.405 ns/op
Iteration 4: 810.189 ns/op
Iteration 5: 808.703 ns/op
Result "testWithSimpleDateFormat":
835.850 ±(99.9%) 94.750 ns/op [Average]
(min, avg, max) = (808.703, 835.850, 857.534), stdev = 24.606
CI (99.9%): [741.101, 930.600] (assumes normal distribution)
# Run complete. Total time: 00:03:20
BenchmarkMode CntScore
Error Units
TestJodaVsSimpleDateFormat.testWithJodaTime avgt5 423.830 ±
85.014 ns/op
TestJodaVsSimpleDateFormat.testWithSimpleDateFormat avgt5 835.850 ±
94.750 ns/op
{noformat}
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
[jira] [Updated] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-13189:
Issue Type: Improvement (was: Bug)
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13146) OrcFile table property values are case sensitive
[ https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13146: Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed into master and branch-1 > OrcFile table property values are case sensitive > > > Key: HIVE-13146 > URL: https://issues.apache.org/jira/browse/HIVE-13146 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Andrew Sears >Assignee: Yongzhi Chen >Priority: Minor > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch, > HIVE-13146.3.patch > > > In Hive v1.2.1.2.3, with Tez , create an external table with compression > SNAPPY value marked as lower case. Table is created successfully. Insert > data into table fails with no enum constant error. > CREATE EXTERNAL TABLE mydb.mytable > (id int) > PARTITIONED BY (business_date date) > STORED AS ORC > LOCATION > '/data/mydb/mytable' > TBLPROPERTIES ( > 'orc.compress'='snappy'); > set hive.exec.dynamic.partition=true; > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE mydb.mytable PARTITION (business_date) > SELECT * from mydb.sourcetable; > Caused by: java.lang.IllegalArgumentException: No enum constant > org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy > at java.lang.Enum.valueOf(Enum.java:238) > at > org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25) > Constant SNAPPY needs to be uppercase in definition to fix. Case should be > agnostic or throw error on creation of table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks to Yongzhi and Sergey for reviewing.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13145) Separate the output path of metrics file of HS2 and HMS
[
https://issues.apache.org/jira/browse/HIVE-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173796#comment-15173796
]
Shinichi Yamashita commented on HIVE-13145:
---
Thank you for your comment.
I know that there are some solutions to solve this topic. But I think the
user's work increases by those ways. And I think it's better to settle it by
default setting, not workaround.
And I think it's easy to understand for the user to separate property name.
> Separate the output path of metrics file of HS2 and HMS
> ---
>
> Key: HIVE-13145
> URL: https://issues.apache.org/jira/browse/HIVE-13145
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Metastore
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: HIVE-13145.1.patch, HIVE-13145.2.patch
>
>
> The output path of metrics file of HS2 and HMS can define by
> {{hive.service.metrics.file.location}} property at present.
> When it starts HS2 and HMS by the same server, both metrics is written in the
> same file. And when confirming this file, it is difficult to judge which
> metrics it is.
> Therefore the output path of metrics file of HS2 and HMS is separated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[
https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173793#comment-15173793
]
Hive QA commented on HIVE-12994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790722/HIVE-12994.12.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/125/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/125/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-125/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790722 - PreCommit-HIVE-METASTORE-Test
> Implement support for NULLS FIRST/NULLS LAST
>
>
> Key: HIVE-12994
> URL: https://issues.apache.org/jira/browse/HIVE-12994
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Parser, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch,
> HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch,
> HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch,
> HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch,
> HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch
>
>
> From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to
> determine whether nulls appear before or after non-null data values when the
> ORDER BY clause is used.
> SQL standard does not specify the behavior by default. Currently in Hive,
> null values sort as if lower than any non-null value; that is, NULLS FIRST is
> the default for ASC order, and NULLS LAST for DESC order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173782#comment-15173782 ] Jesus Camacho Rodriguez commented on HIVE-12994: [~ashutoshc], I added the keyword_3.q test file to cover this case. > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Attachment: HIVE-12994.12.patch > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Status: Patch Available (was: In Progress) > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Status: Open (was: Patch Available) > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-12994 started by Jesus Camacho Rodriguez. -- > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13146) OrcFile table property values are case sensitive
[ https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173771#comment-15173771 ] Yongzhi Chen commented on HIVE-13146: - The failures are not related. > OrcFile table property values are case sensitive > > > Key: HIVE-13146 > URL: https://issues.apache.org/jira/browse/HIVE-13146 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Andrew Sears >Assignee: Yongzhi Chen >Priority: Minor > Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch, > HIVE-13146.3.patch > > > In Hive v1.2.1.2.3, with Tez , create an external table with compression > SNAPPY value marked as lower case. Table is created successfully. Insert > data into table fails with no enum constant error. > CREATE EXTERNAL TABLE mydb.mytable > (id int) > PARTITIONED BY (business_date date) > STORED AS ORC > LOCATION > '/data/mydb/mytable' > TBLPROPERTIES ( > 'orc.compress'='snappy'); > set hive.exec.dynamic.partition=true; > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE mydb.mytable PARTITION (business_date) > SELECT * from mydb.sourcetable; > Caused by: java.lang.IllegalArgumentException: No enum constant > org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy > at java.lang.Enum.valueOf(Enum.java:238) > at > org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25) > Constant SNAPPY needs to be uppercase in definition to fix. Case should be > agnostic or throw error on creation of table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[
https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173725#comment-15173725
]
Hive QA commented on HIVE-13175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790575/HIVE-13175.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 9764 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_partition_diff_num_cols.q-vector_decimal_aggregate.q-vector_date_1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTransactionalValidation
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTransactionalValidation
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTransactionalValidation
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTransactionalValidation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7136/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7136/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7136/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790575 - PreCommit-HIVE-TRUNK-Build
> Disallow making external tables transactional
> -
>
> Key: HIVE-13175
> URL: https://issues.apache.org/jira/browse/HIVE-13175
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13175.1.patch
>
>
> The fact that compactor rewrites contents of ACID tables is in conflict with
> what is expected of external tables.
> Conversely, end user can write to External table which certainly not what is
> expected of ACID table.
> So we should explicitly disallow making an external table ACID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173707#comment-15173707 ] Rajat Khandelwal commented on HIVE-4629: As far as I understand, the logs are removed once the operation completes. So if my operation finished with an error and I want to see the logs after it fails, how do I proceed? > HS2 should support an API to retrieve query logs > > > Key: HIVE-4629 > URL: https://issues.apache.org/jira/browse/HIVE-4629 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Shreepadma Venugopalan >Assignee: Dong Chen > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, > HIVE-4629.2.patch, HIVE-4629.3.patch.txt, HIVE-4629.4.patch, > HIVE-4629.5.patch, HIVE-4629.6.patch, HIVE-4629.7.patch, HIVE-4629.8.patch, > HIVE-4629.9.patch > > > HiveServer2 should support an API to retrieve query logs. This is > particularly relevant because HiveServer2 supports async execution but > doesn't provide a way to report progress. Providing an API to retrieve query > logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12757) Fix TestCodahaleMetrics#testFileReporting
[
https://issues.apache.org/jira/browse/HIVE-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173531#comment-15173531
]
Hive QA commented on HIVE-12757:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790534/HIVE-12757.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9748 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_outer_join5.q-custom_input_output_format.q-vector_groupby_reduce.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7135/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7135/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7135/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790534 - PreCommit-HIVE-TRUNK-Build
> Fix TestCodahaleMetrics#testFileReporting
> -
>
> Key: HIVE-12757
> URL: https://issues.apache.org/jira/browse/HIVE-12757
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.1.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-12757.2.patch, HIVE-12757.patch
>
>
> Codahale Metrics file reporter is time based, hence test is as well. On slow
> machines, sometimes the file is not written fast enough to be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173465#comment-15173465 ] Austin Lee commented on HIVE-12679: --- These failures are not related to my change. > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.1.patch, HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173433#comment-15173433 ] Lefty Leverenz commented on HIVE-13095: --- Is this a new feature that needs to be documented in the wiki (TODOC2.1 label), or is it just a bug fix? > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
