[jira] [Updated] (HIVE-12301) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test failure for udf_percentile.q
[ https://issues.apache.org/jira/browse/HIVE-12301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12301: --- Attachment: HIVE-12301.01.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test > failure for udf_percentile.q > --- > > Key: HIVE-12301 > URL: https://issues.apache.org/jira/browse/HIVE-12301 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12301.01.patch > > > The position in argList is mapped to a wrong column from RS operator -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12305) CBO: Calcite Operator To Hive Operator (Calcite Return Path): UDAF can not pull up constant expressions
[ https://issues.apache.org/jira/browse/HIVE-12305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12305: --- Attachment: HIVE-12305.01.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): UDAF can not > pull up constant expressions > --- > > Key: HIVE-12305 > URL: https://issues.apache.org/jira/browse/HIVE-12305 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12305.01.patch > > > to repro, run annotate_stats_groupby.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12323) Change default value of hive.mapred.mode to strict
[ https://issues.apache.org/jira/browse/HIVE-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986518#comment-14986518 ] Pengcheng Xiong commented on HIVE-12323: Advanced users = VIP? :) > Change default value of hive.mapred.mode to strict > --- > > Key: HIVE-12323 > URL: https://issues.apache.org/jira/browse/HIVE-12323 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan > > Its better to be conservative and strict so that users are saved from > avoidable mistakes. Advanced users can chose to go to nonstrict mode when > they know what that entails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12210) Fix a few failing tests
[
https://issues.apache.org/jira/browse/HIVE-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962643#comment-14962643
]
Pengcheng Xiong commented on HIVE-12210:
+1 pending QA test run.
> Fix a few failing tests
> ---
>
> Key: HIVE-12210
> URL: https://issues.apache.org/jira/browse/HIVE-12210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aleksei S
>Assignee: Aleksei S
> Attachments: HIVE-12210.patch
>
>
> The following tests fail after HIVE-11785 because of missing
> "serialization.escape.crlf true" property in the output.
> {code}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[ https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11927: --- Attachment: HIVE-11927.05.patch > Implement/Enable constant related optimization rules in Calcite: enable > HiveReduceExpressionsRule to fold constants > --- > > Key: HIVE-11927 > URL: https://issues.apache.org/jira/browse/HIVE-11927 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch, > HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12157) select-clause doesn't support unicode alias
[ https://issues.apache.org/jira/browse/HIVE-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957676#comment-14957676 ] Pengcheng Xiong commented on HIVE-12157: [~richarddu], thanks a lot for your prompt patch and investigation. The lexer change sounds good to me. However, could you add more test cases? (1) Because you now extend the lexer, I guess other identifiers, e.g., column name itself can also use the unicode. For example, can u test "select 行1 from test limit 10;" (2) Other than junit test, could you add more q tests? For example, "select a as 行1 from test a join test b where a.行1= b.a", etc. Because this is a significant change to Hive, we need much more tests. Thanks. > select-clause doesn't support unicode alias > --- > > Key: HIVE-12157 > URL: https://issues.apache.org/jira/browse/HIVE-12157 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 1.2.1 >Reporter: richard du >Assignee: richard du >Priority: Minor > Attachments: HIVE-12157.patch, TestUnicodeAliasParse.java > > > Parser will throw exception when I use alias: > hive> desc test; > OK > a int > b string > Time taken: 0.135 seconds, Fetched: 2 row(s) > hive> select a as 行1 from test limit 10; > NoViableAltException(302@[134:7: ( ( ( KW_AS )? identifier ) | ( KW_AS LPAREN > identifier ( COMMA identifier )* RPAREN ) )?]) > at org.antlr.runtime.DFA.noViableAlt(DFA.java:158) > at org.antlr.runtime.DFA.predict(DFA.java:116) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2915) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > FAILED: ParseException line 1:13 cannot recognize input near 'as' '1' 'from' > in selection target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-12156: -- Assignee: Pengcheng Xiong > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Pengcheng Xiong > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12156: --- Attachment: HIVE-12156.01.patch [~jpullokkaran], could you take a look? Thanks. > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Pengcheng Xiong > Attachments: HIVE-12156.01.patch > > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957956#comment-14957956 ] Pengcheng Xiong commented on HIVE-12156: [~busyjay], sorry, I misunderstood your problem. I will submit a patch soon and you can apply and test if it works. Thanks. > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Pengcheng Xiong > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11894: --- Attachment: HIVE-11894.05.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch, HIVE-11894.05.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11895: --- Attachment: HIVE-11895.02.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11895.01.patch, HIVE-11895.02.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12210) Fix a few failing tests: testCliDriver_udf_explode and
[
https://issues.apache.org/jira/browse/HIVE-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12210:
---
Summary: Fix a few failing tests: testCliDriver_udf_explode and (was: Fix
a few failing tests)
> Fix a few failing tests: testCliDriver_udf_explode and
> ---
>
> Key: HIVE-12210
> URL: https://issues.apache.org/jira/browse/HIVE-12210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aleksei S
>Assignee: Aleksei S
> Attachments: HIVE-12210.patch
>
>
> The following tests fail after HIVE-11785 because of missing
> "serialization.escape.crlf true" property in the output.
> {code}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12210) Fix a few failing tests: testCliDriver_udf_explode and testCliDriver_udtf_explode
[
https://issues.apache.org/jira/browse/HIVE-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12210:
---
Summary: Fix a few failing tests: testCliDriver_udf_explode and
testCliDriver_udtf_explode (was: Fix a few failing tests:
testCliDriver_udf_explode and )
> Fix a few failing tests: testCliDriver_udf_explode and
> testCliDriver_udtf_explode
> -
>
> Key: HIVE-12210
> URL: https://issues.apache.org/jira/browse/HIVE-12210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aleksei S
>Assignee: Aleksei S
> Attachments: HIVE-12210.patch
>
>
> The following tests fail after HIVE-11785 because of missing
> "serialization.escape.crlf true" property in the output.
> {code}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12210) Fix a few failing tests: testCliDriver_udf_explode and testCliDriver_udtf_explode
[
https://issues.apache.org/jira/browse/HIVE-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964334#comment-14964334
]
Pengcheng Xiong commented on HIVE-12210:
pushed to master. Thanks [~Aleksei]
> Fix a few failing tests: testCliDriver_udf_explode and
> testCliDriver_udtf_explode
> -
>
> Key: HIVE-12210
> URL: https://issues.apache.org/jira/browse/HIVE-12210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aleksei S
>Assignee: Aleksei S
> Fix For: 2.0.0
>
> Attachments: HIVE-12210.patch
>
>
> The following tests fail after HIVE-11785 because of missing
> "serialization.escape.crlf true" property in the output.
> {code}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12210) Fix a few failing tests: testCliDriver_udf_explode and testCliDriver_udtf_explode
[
https://issues.apache.org/jira/browse/HIVE-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12210:
---
Fix Version/s: 2.0.0
> Fix a few failing tests: testCliDriver_udf_explode and
> testCliDriver_udtf_explode
> -
>
> Key: HIVE-12210
> URL: https://issues.apache.org/jira/browse/HIVE-12210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aleksei S
>Assignee: Aleksei S
> Fix For: 2.0.0
>
> Attachments: HIVE-12210.patch
>
>
> The following tests fail after HIVE-11785 because of missing
> "serialization.escape.crlf true" property in the output.
> {code}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-7693:
--
Attachment: HIVE-7693.04.patch
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Fix For: 2.0.0
>
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-7693:
--
Fix Version/s: 2.0.0
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Fix For: 2.0.0
>
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12157) select-clause doesn't support unicode alias
[ https://issues.apache.org/jira/browse/HIVE-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961058#comment-14961058 ] Pengcheng Xiong commented on HIVE-12157: [~richarddu], yes, you need to change that to support unicode. I have done a similar change before. Could you please refer to HIVE-11699, take a look and do some investigation and estimate how much change you may need to make? Thanks. > select-clause doesn't support unicode alias > --- > > Key: HIVE-12157 > URL: https://issues.apache.org/jira/browse/HIVE-12157 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 1.2.1 >Reporter: richard du >Assignee: richard du >Priority: Minor > Attachments: HIVE-12157.patch, TestUnicodeAliasParse.java > > > Parser will throw exception when I use alias: > hive> desc test; > OK > a int > b string > Time taken: 0.135 seconds, Fetched: 2 row(s) > hive> select a as 行1 from test limit 10; > NoViableAltException(302@[134:7: ( ( ( KW_AS )? identifier ) | ( KW_AS LPAREN > identifier ( COMMA identifier )* RPAREN ) )?]) > at org.antlr.runtime.DFA.noViableAlt(DFA.java:158) > at org.antlr.runtime.DFA.predict(DFA.java:116) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2915) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > FAILED: ParseException line 1:13 cannot recognize input near 'as' '1' 'from' > in selection target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-7693:
--
Attachment: HIVE-7693.03.patch
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962012#comment-14962012 ] Pengcheng Xiong commented on HIVE-11895: The failed tests are unrelated. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11895.01.patch, HIVE-11895.02.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962013#comment-14962013 ] Pengcheng Xiong commented on HIVE-11895: [~ashutoshc], could you please take a look? Thanks! > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11895.01.patch, HIVE-11895.02.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11894: --- Attachment: HIVE-11894.07.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch, HIVE-11894.05.patch, > HIVE-11894.06.patch, HIVE-11894.07.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11894: --- Attachment: HIVE-11894.08.patch all the failed tests passed on my laptop. resubmit. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch, HIVE-11894.05.patch, > HIVE-11894.06.patch, HIVE-11894.07.patch, HIVE-11894.08.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong reassigned HIVE-7693:
-
Assignee: Pengcheng Xiong
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12094) nDV of aggregate columns tend to be log scale - not unique
[
https://issues.apache.org/jira/browse/HIVE-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953387#comment-14953387
]
Pengcheng Xiong commented on HIVE-12094:
[~gopalv], yes, the stats estimation for aggregation is very rough right now.
> nDV of aggregate columns tend to be log scale - not unique
> --
>
> Key: HIVE-12094
> URL: https://issues.apache.org/jira/browse/HIVE-12094
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>
> Stats for aggregate columns do not process properly if declared as a simple
> nDV
> {code}
> select count(distinct l_suppkey) from lineitem group by l_orderkey having
> count(distinct l_suppkey) = 1
> {code}
> will mis-estimate the cardinality of the output by a significant margin.
> The log-scale of the nDV in general skews towards a very low number, which is
> not accounted for in the StatsRulesProcFactory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953428#comment-14953428 ] Pengcheng Xiong commented on HIVE-11894: The failed test case is unrelated. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch, HIVE-11894.05.patch, > HIVE-11894.06.patch, HIVE-11894.07.patch, HIVE-11894.08.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12065) FS stats collection may generate incorrect stats for multi-insert query
[ https://issues.apache.org/jira/browse/HIVE-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953817#comment-14953817 ] Pengcheng Xiong commented on HIVE-12065: I checked the source code change, confirmed test cases change(all of them involve multi-insert). All of them look good to me. +1. This should fix lots of incorrect multi-insert status. > FS stats collection may generate incorrect stats for multi-insert query > --- > > Key: HIVE-12065 > URL: https://issues.apache.org/jira/browse/HIVE-12065 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12065.2.patch, HIVE-12065.3.patch, > HIVE-12065.4.patch, HIVE-12065.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11301) thrift metastore issue when getting stats results in disconnect
[
https://issues.apache.org/jira/browse/HIVE-11301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11301:
---
Fix Version/s: 1.0.2
> thrift metastore issue when getting stats results in disconnect
> ---
>
> Key: HIVE-11301
> URL: https://issues.apache.org/jira/browse/HIVE-11301
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Sergey Shelukhin
>Assignee: Pengcheng Xiong
> Fix For: 1.3.0, 2.0.0, 1.0.2, 1.2.2
>
> Attachments: HIVE-11301.01.patch, HIVE-11301.02.patch
>
>
> On metastore side it looks like this:
> {noformat}
> 2015-07-17 20:32:27,795 ERROR [pool-3-thread-150]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing
> of message.
> org.apache.thrift.protocol.TProtocolException: Required field 'colStats' is
> unset! Struct:AggrStats(colStats:null, partsFound:0)
> at
> org.apache.hadoop.hive.metastore.api.AggrStats.validate(AggrStats.java:389)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.validate(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.write(ThriftHiveMetastore.java)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> and then
> {noformat}
> 2015-07-17 20:32:27,796 WARN [pool-3-thread-150]:
> transport.TIOStreamTransport (TIOStreamTransport.java:close(112)) - Error
> closing output stream.
> java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> at
> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
> at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.close(TFilterTransport.java:52)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Which on client manifests as
> {noformat}
> 2015-07-17 20:32:27,796 WARN [main()]: metastore.RetryingMetaStoreClient
> (RetryingMetaStoreClient.java:invoke(187)) - MetaStoreClient lost connection.
> Attempting to reconnect.
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
>
[jira] [Updated] (HIVE-12011) unable to create temporary table using CTAS if regular table with that name already exists
[
https://issues.apache.org/jira/browse/HIVE-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12011:
---
Affects Version/s: 1.2.1
> unable to create temporary table using CTAS if regular table with that name
> already exists
> --
>
> Key: HIVE-12011
> URL: https://issues.apache.org/jira/browse/HIVE-12011
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.0.0
>
> Attachments: HIVE-12011.01.patch
>
>
> CTAS temporary table query fails if regular table with the same name already
> exists.
> Steps to reproduce the issue:
> {noformat}
> hive> use dbtemptable;
> OK
> Time taken: 0.273 seconds
> hive> create table a(i int);
> OK
> Time taken: 0.297 seconds
> hive> create temporary table a(i int);
> OK
> Time taken: 0.165 seconds
> hive> create table b(i int);
> OK
> Time taken: 0.212 seconds
> hive> create temporary table b as select * from a;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> hive> create table c(i int);
> OK
> Time taken: 0.264 seconds
> hive> create temporary table b as select * from c;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12011) unable to create temporary table using CTAS if regular table with that name already exists
[
https://issues.apache.org/jira/browse/HIVE-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945829#comment-14945829
]
Pengcheng Xiong commented on HIVE-12011:
pushed to master. Thanks [~jpullokkaran] for the review.
> unable to create temporary table using CTAS if regular table with that name
> already exists
> --
>
> Key: HIVE-12011
> URL: https://issues.apache.org/jira/browse/HIVE-12011
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12011.01.patch
>
>
> CTAS temporary table query fails if regular table with the same name already
> exists.
> Steps to reproduce the issue:
> {noformat}
> hive> use dbtemptable;
> OK
> Time taken: 0.273 seconds
> hive> create table a(i int);
> OK
> Time taken: 0.297 seconds
> hive> create temporary table a(i int);
> OK
> Time taken: 0.165 seconds
> hive> create table b(i int);
> OK
> Time taken: 0.212 seconds
> hive> create temporary table b as select * from a;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> hive> create table c(i int);
> OK
> Time taken: 0.264 seconds
> hive> create temporary table b as select * from c;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11894: --- Attachment: HIVE-11894.04.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12011) unable to create temporary table using CTAS if regular table with that name already exists
[
https://issues.apache.org/jira/browse/HIVE-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12011:
---
Fix Version/s: 2.0.0
> unable to create temporary table using CTAS if regular table with that name
> already exists
> --
>
> Key: HIVE-12011
> URL: https://issues.apache.org/jira/browse/HIVE-12011
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.0.0
>
> Attachments: HIVE-12011.01.patch
>
>
> CTAS temporary table query fails if regular table with the same name already
> exists.
> Steps to reproduce the issue:
> {noformat}
> hive> use dbtemptable;
> OK
> Time taken: 0.273 seconds
> hive> create table a(i int);
> OK
> Time taken: 0.297 seconds
> hive> create temporary table a(i int);
> OK
> Time taken: 0.165 seconds
> hive> create table b(i int);
> OK
> Time taken: 0.212 seconds
> hive> create temporary table b as select * from a;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> hive> create table c(i int);
> OK
> Time taken: 0.264 seconds
> hive> create temporary table b as select * from c;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
> Table already exists: dbtemptable.b
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945918#comment-14945918 ] Pengcheng Xiong commented on HIVE-11894: [~ashutoshc], could you please take a look? Thanks. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8765) TPC-DS Q21 : Incorrect join order makes query run slower (Not scaling selectivity by NDV)
[
https://issues.apache.org/jira/browse/HIVE-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949780#comment-14949780
]
Pengcheng Xiong commented on HIVE-8765:
---
Close the issue because (1) The correlation between the join column and the
predicate column is unknown after discussing with [~rhbutani] and (2) the
current plan to join data_dim will generate 405000 rows as intermediate result
and the other plan to join item first will generate 726850 rows. The
accumulated cost is even higher.
> TPC-DS Q21 : Incorrect join order makes query run slower (Not scaling
> selectivity by NDV)
> --
>
> Key: HIVE-8765
> URL: https://issues.apache.org/jira/browse/HIVE-8765
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Laljo John Pullokkaran
>
> CBO joins with date_dim first instead of item where item is the more
> selective join.
> Query
> {code}
> select *
> from(select w_warehouse_name
> ,i_item_id
> ,sum(case when (cast(d_date as date) < cast ('1998-04-08' as
> date))
> then inv_quantity_on_hand
> else 0 end) as inv_before
> ,sum(case when (cast(d_date as date) >= cast ('1998-04-08' as
> date))
> then inv_quantity_on_hand
> else 0 end) as inv_after
>from inventory
>,warehouse
>,item
>,date_dim
>where i_current_price between 0.99 and 1.49
> and item.i_item_sk = inventory.inv_item_sk
> and inventory.inv_warehouse_sk = warehouse.w_warehouse_sk
> and inventory.inv_date_sk= date_dim.d_date_sk
> and d_date between '1998-03-09' and '1998-05-07'
>group by w_warehouse_name, i_item_id) x
> where (case when inv_before > 0
> then inv_after / inv_before
> else null
> end) between 2.0/3.0 and 3.0/2.0
> order by w_warehouse_name
> ,i_item_id
> limit 100
> {code}
> Logical Plan
> {code}
> 2014-11-06 16:58:32,041 DEBUG [main]: parse.SemanticAnalyzer
> (SemanticAnalyzer.java:apply(12631)) - Plan After Join Reordering:
> HiveSortRel(fetch=[100]): rowcount = 1.0, cumulative cost =
> {1.627879384609158E9 rows, 2.0 cpu, 0.0 io}, id = 12521
> HiveSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC]): rowcount =
> 1.0, cumulative cost = {1.627879368609158E9 rows, 1.0 cpu, 0.0 io}, id = 12519
> HiveProjectRel(w_warehouse_name=[$0], i_item_id=[$1], inv_before=[$2],
> inv_after=[$3]): rowcount = 1.0, cumulative cost = {1.627879352609158E9 rows,
> 0.0 cpu, 0.0 io}, id = 12517
> HiveFilterRel(condition=[between(false, when(>($2, 0),
> /(CAST($3):DOUBLE, CAST($2):DOUBLE), null), /(2E0, 3E0), /(3E0, 2E0))]):
> rowcount = 1.0, cumulative cost = {1.627879352609158E9 rows, 0.0 cpu, 0.0
> io}, id = 12515
> HiveAggregateRel(group=[{0, 1}], agg#0=[sum($2)], agg#1=[sum($3)]):
> rowcount = 1.7688372892644288, cumulative cost = {1.627879352609158E9 rows,
> 0.0 cpu, 0.0 io}, id = 12513
> HiveProjectRel($f0=[$5], $f1=[$7], $f2=[when(<(CAST($10):DATE,
> CAST('1998-04-08'):DATE), $2, 0)], $f3=[when(>=(CAST($10):DATE,
> CAST('1998-04-08'):DATE), $2, 0)]): rowcount = 1.8477987480495097, cumulative
> cost = {1.627879352609158E9 rows, 0.0 cpu, 0.0 io}, id = 12511
> HiveProjectRel(inv_item_sk=[$2], inv_warehouse_sk=[$3],
> inv_quantity_on_hand=[$4], inv_date_sk=[$5], w_warehouse_sk=[$0],
> w_warehouse_name=[$1], i_item_sk=[$8], i_item_id=[$9], i_current_price=[$10],
> d_date_sk=[$6], d_date=[$7]): rowcount = 1.8477987480495097, cumulative cost
> = {1.627879352609158E9 rows, 0.0 cpu, 0.0 io}, id = 12577
> HiveJoinRel(condition=[=($3, $0)], joinType=[inner]): rowcount
> = 1.8477987480495097, cumulative cost = {1.627879352609158E9 rows, 0.0 cpu,
> 0.0 io}, id = 12575
> HiveProjectRel(w_warehouse_sk=[$0], w_warehouse_name=[$2]):
> rowcount = 27.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 12463
>
> HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_3.warehouse]]):
> rowcount = 27.0, cumulative cost = {0}, id = 12287
> HiveJoinRel(condition=[=($6, $0)], joinType=[inner]):
> rowcount = 1.8477987480495097, cumulative cost = {1.6278793237613592E9 rows,
> 0.0 cpu, 0.0 io}, id = 12573
> HiveJoinRel(condition=[=($3, $4)], joinType=[inner]):
> rowcount = 22284.45290147709, cumulative cost = {1.627857001E9 rows, 0.0 cpu,
> 0.0 io}, id = 12534
> HiveProjectRel(inv_item_sk=[$0], inv_warehouse_sk=[$1],
> inv_quantity_on_hand=[$2], inv_date_sk=[$3]): rowcount = 1.627857E9,
> cumulative cost = {0.0 rows, 0.0 cpu, 0.0
[jira] [Resolved] (HIVE-8765) TPC-DS Q21 : Incorrect join order makes query run slower (Not scaling selectivity by NDV)
[
https://issues.apache.org/jira/browse/HIVE-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong resolved HIVE-8765.
---
Resolution: Won't Fix
Assignee: Pengcheng Xiong (was: Laljo John Pullokkaran)
> TPC-DS Q21 : Incorrect join order makes query run slower (Not scaling
> selectivity by NDV)
> --
>
> Key: HIVE-8765
> URL: https://issues.apache.org/jira/browse/HIVE-8765
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Pengcheng Xiong
>
> CBO joins with date_dim first instead of item where item is the more
> selective join.
> Query
> {code}
> select *
> from(select w_warehouse_name
> ,i_item_id
> ,sum(case when (cast(d_date as date) < cast ('1998-04-08' as
> date))
> then inv_quantity_on_hand
> else 0 end) as inv_before
> ,sum(case when (cast(d_date as date) >= cast ('1998-04-08' as
> date))
> then inv_quantity_on_hand
> else 0 end) as inv_after
>from inventory
>,warehouse
>,item
>,date_dim
>where i_current_price between 0.99 and 1.49
> and item.i_item_sk = inventory.inv_item_sk
> and inventory.inv_warehouse_sk = warehouse.w_warehouse_sk
> and inventory.inv_date_sk= date_dim.d_date_sk
> and d_date between '1998-03-09' and '1998-05-07'
>group by w_warehouse_name, i_item_id) x
> where (case when inv_before > 0
> then inv_after / inv_before
> else null
> end) between 2.0/3.0 and 3.0/2.0
> order by w_warehouse_name
> ,i_item_id
> limit 100
> {code}
> Logical Plan
> {code}
> 2014-11-06 16:58:32,041 DEBUG [main]: parse.SemanticAnalyzer
> (SemanticAnalyzer.java:apply(12631)) - Plan After Join Reordering:
> HiveSortRel(fetch=[100]): rowcount = 1.0, cumulative cost =
> {1.627879384609158E9 rows, 2.0 cpu, 0.0 io}, id = 12521
> HiveSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC]): rowcount =
> 1.0, cumulative cost = {1.627879368609158E9 rows, 1.0 cpu, 0.0 io}, id = 12519
> HiveProjectRel(w_warehouse_name=[$0], i_item_id=[$1], inv_before=[$2],
> inv_after=[$3]): rowcount = 1.0, cumulative cost = {1.627879352609158E9 rows,
> 0.0 cpu, 0.0 io}, id = 12517
> HiveFilterRel(condition=[between(false, when(>($2, 0),
> /(CAST($3):DOUBLE, CAST($2):DOUBLE), null), /(2E0, 3E0), /(3E0, 2E0))]):
> rowcount = 1.0, cumulative cost = {1.627879352609158E9 rows, 0.0 cpu, 0.0
> io}, id = 12515
> HiveAggregateRel(group=[{0, 1}], agg#0=[sum($2)], agg#1=[sum($3)]):
> rowcount = 1.7688372892644288, cumulative cost = {1.627879352609158E9 rows,
> 0.0 cpu, 0.0 io}, id = 12513
> HiveProjectRel($f0=[$5], $f1=[$7], $f2=[when(<(CAST($10):DATE,
> CAST('1998-04-08'):DATE), $2, 0)], $f3=[when(>=(CAST($10):DATE,
> CAST('1998-04-08'):DATE), $2, 0)]): rowcount = 1.8477987480495097, cumulative
> cost = {1.627879352609158E9 rows, 0.0 cpu, 0.0 io}, id = 12511
> HiveProjectRel(inv_item_sk=[$2], inv_warehouse_sk=[$3],
> inv_quantity_on_hand=[$4], inv_date_sk=[$5], w_warehouse_sk=[$0],
> w_warehouse_name=[$1], i_item_sk=[$8], i_item_id=[$9], i_current_price=[$10],
> d_date_sk=[$6], d_date=[$7]): rowcount = 1.8477987480495097, cumulative cost
> = {1.627879352609158E9 rows, 0.0 cpu, 0.0 io}, id = 12577
> HiveJoinRel(condition=[=($3, $0)], joinType=[inner]): rowcount
> = 1.8477987480495097, cumulative cost = {1.627879352609158E9 rows, 0.0 cpu,
> 0.0 io}, id = 12575
> HiveProjectRel(w_warehouse_sk=[$0], w_warehouse_name=[$2]):
> rowcount = 27.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 12463
>
> HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_3.warehouse]]):
> rowcount = 27.0, cumulative cost = {0}, id = 12287
> HiveJoinRel(condition=[=($6, $0)], joinType=[inner]):
> rowcount = 1.8477987480495097, cumulative cost = {1.6278793237613592E9 rows,
> 0.0 cpu, 0.0 io}, id = 12573
> HiveJoinRel(condition=[=($3, $4)], joinType=[inner]):
> rowcount = 22284.45290147709, cumulative cost = {1.627857001E9 rows, 0.0 cpu,
> 0.0 io}, id = 12534
> HiveProjectRel(inv_item_sk=[$0], inv_warehouse_sk=[$1],
> inv_quantity_on_hand=[$2], inv_date_sk=[$3]): rowcount = 1.627857E9,
> cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 12460
>
> HiveTableScanRel(table=[[tpcds_bin_partitioned_orc_3.inventory]]):
> rowcount = 1.627857E9, cumulative cost = {0}, id = 12284
> HiveProjectRel(d_date_sk=[$0], d_date=[$2]): rowcount =
> 1.0, cumulative cost = {0.0 rows, 0.0
[jira] [Updated] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11894: --- Attachment: HIVE-11894.06.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch, > HIVE-11894.03.patch, HIVE-11894.04.patch, HIVE-11894.05.patch, > HIVE-11894.06.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-7693:
--
Attachment: HIVE-7693.01.patch
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12010) Tests should use FileSystem based stats collection mechanism
[ https://issues.apache.org/jira/browse/HIVE-12010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12010: --- Description: Although fs based collection mechanism is default for last few releases, tests still use jdbc for stats collection. The main advantage of fs based collection over jdbc based one is the scalability. In jdbc case, a single database (normally co-located with the metastore relational database) is used to handle all the stats collected by all the tasks. This single database is responsible to maintain the consistency for the stats, which will become a bottleneck and face scalability issue when the number of tasks is huge. In fs case, each task is writing stats into hdfs which does not have scalability issue. (was: Although fs based collection mechanism is default for last few releases, tests still use jdbc for stats collection.) > Tests should use FileSystem based stats collection mechanism > > > Key: HIVE-12010 > URL: https://issues.apache.org/jira/browse/HIVE-12010 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12010.1.patch, HIVE-12010.2.patch, > HIVE-12010.3.patch, HIVE-12010.4.patch, HIVE-12010.patch > > > Although fs based collection mechanism is default for last few releases, > tests still use jdbc for stats collection. The main advantage of fs based > collection over jdbc based one is the scalability. In jdbc case, a single > database (normally co-located with the metastore relational database) is used > to handle all the stats collected by all the tasks. This single database is > responsible to maintain the consistency for the stats, which will become a > bottleneck and face scalability issue when the number of tasks is huge. In fs > case, each task is writing stats into hdfs which does not have scalability > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12156) expanding view doesn't quote reserved keyword
[
https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955635#comment-14955635
]
Pengcheng Xiong commented on HIVE-12156:
[~busyjay], this is current limitation of Hive. Hive parser does NOT support 2
or more levels of "dot". For example,
{code}
select default.src.key from src;
{code}
will fail. This has nothing to do with reserved keywords. If you are willing to
work on the new feature to support 2 or more levels of dot, I would be happy to
review it. Thanks.
> expanding view doesn't quote reserved keyword
> -
>
> Key: HIVE-12156
> URL: https://issues.apache.org/jira/browse/HIVE-12156
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.2.1
> Environment: hadoop 2.7
> hive 1.2.1
>Reporter: Jay Lee
>
> hive> create table testreserved (data struct<`end`:string, id: string>);
> OK
> Time taken: 0.274 seconds
> hive> create view testreservedview as select data.`end` as data_end, data.id
> as data_id from testreserved;
> OK
> Time taken: 0.769 seconds
> hive> select data.`end` from testreserved;
> OK
> Time taken: 1.852 seconds
> hive> select data_id from testreservedview;
> NoViableAltException(98@[])
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> ...
> FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as'
> 'data_end' in expression specification in definition of VIEW testreservedview
> [
> select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as
> `data_id` from `test`.`testreserved`
> ] used as testreservedview at Line 1:20
> When view is expanded, field should be quote with backquote.
--
This
[jira] [Commented] (HIVE-12157) select-clause doesn't support unicode alias
[ https://issues.apache.org/jira/browse/HIVE-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955630#comment-14955630 ] Pengcheng Xiong commented on HIVE-12157: [~richarddu], this is current limitation of Hive. Hive does NOT support unicode. If you are willing to work on this new feature, I would be happy to review it. Thanks. > select-clause doesn't support unicode alias > --- > > Key: HIVE-12157 > URL: https://issues.apache.org/jira/browse/HIVE-12157 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 1.2.1 >Reporter: richard du >Priority: Minor > > Parser will throw exception when I use alias: > hive> desc test; > OK > a int > b string > Time taken: 0.135 seconds, Fetched: 2 row(s) > hive> select a as 行1 from test limit 10; > NoViableAltException(302@[134:7: ( ( ( KW_AS )? identifier ) | ( KW_AS LPAREN > identifier ( COMMA identifier )* RPAREN ) )?]) > at org.antlr.runtime.DFA.noViableAlt(DFA.java:158) > at org.antlr.runtime.DFA.predict(DFA.java:116) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2915) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > FAILED: ParseException line 1:13 cannot recognize input near 'as' '1' 'from' > in selection target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11195) Make auto_sortmerge_join_16.q result sequence more stable
[ https://issues.apache.org/jira/browse/HIVE-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-11195. Resolution: Fixed Make auto_sortmerge_join_16.q result sequence more stable - Key: HIVE-11195 URL: https://issues.apache.org/jira/browse/HIVE-11195 Project: Hive Issue Type: Improvement Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Trivial adding -- SORT_QUERY_RESULTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11211) Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory
[ https://issues.apache.org/jira/browse/HIVE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11211: --- Attachment: (was: HIVE-11211.01.patch) Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory -- Key: HIVE-11211 URL: https://issues.apache.org/jira/browse/HIVE-11211 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11211) Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory
[ https://issues.apache.org/jira/browse/HIVE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11211: --- Attachment: HIVE-11211.02.patch Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory -- Key: HIVE-11211 URL: https://issues.apache.org/jira/browse/HIVE-11211 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11211.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11211) Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory
[ https://issues.apache.org/jira/browse/HIVE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11211: --- Attachment: HIVE-11211.01.patch [~mmokhtar], could you please review it? Thanks. Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory -- Key: HIVE-11211 URL: https://issues.apache.org/jira/browse/HIVE-11211 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11211.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11211) Reset the fields in JoinStatsRule in StatsRulesProcFactory
[ https://issues.apache.org/jira/browse/HIVE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11211: --- Attachment: HIVE-11211.03.patch .03 patch changes the name to getCardinality following [~jpullokkaran]'s suggestion. Reset the fields in JoinStatsRule in StatsRulesProcFactory -- Key: HIVE-11211 URL: https://issues.apache.org/jira/browse/HIVE-11211 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11211.02.patch, HIVE-11211.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Collect column stats when set hive.stats.autogather=true
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Attachment: (was: Design doc for auto column stats gathering.docx) Collect column stats when set hive.stats.autogather=true Key: HIVE-11160 URL: https://issues.apache.org/jira/browse/HIVE-11160 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11160.01.patch Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using Analyze command. In this patch, the column stats will also be collected automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Summary: Auto-gather column stats (was: Collect column stats when set hive.stats.autogather=true) Auto-gather column stats Key: HIVE-11160 URL: https://issues.apache.org/jira/browse/HIVE-11160 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using Analyze command. In this patch, the column stats will also be collected automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11224) AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
[
https://issues.apache.org/jira/browse/HIVE-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621721#comment-14621721
]
Pengcheng Xiong commented on HIVE-11224:
As per [~jpullokkaran]'s request, could [~thejas] and [~vgumashta] take a look?
It is related to the HIVE-HIVE-10382, AggregateStatsCache work. Thanks.
AggregateStatsCache triggers java.util.ConcurrentModificationException under
some conditions
Key: HIVE-11224
URL: https://issues.apache.org/jira/browse/HIVE-11224
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11224.01.patch
Stack
{code}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$EntryIterator.next(HashMap.java:962)
at java.util.HashMap$EntryIterator.next(HashMap.java:960)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.findBestMatch(AggregateStatsCache.java:244)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.get(AggregateStatsCache.java:186)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1131)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6174)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6170)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2405)
at
org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6170)
at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5707)
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy8.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2067)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11224) AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
[
https://issues.apache.org/jira/browse/HIVE-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11224:
---
Attachment: HIVE-11224.01.patch
AggregateStatsCache triggers java.util.ConcurrentModificationException under
some conditions
Key: HIVE-11224
URL: https://issues.apache.org/jira/browse/HIVE-11224
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11224.01.patch
Stack
{code}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$EntryIterator.next(HashMap.java:962)
at java.util.HashMap$EntryIterator.next(HashMap.java:960)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.findBestMatch(AggregateStatsCache.java:244)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.get(AggregateStatsCache.java:186)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1131)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6174)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6170)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2405)
at
org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6170)
at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5707)
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy8.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2067)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Collect column stats when set hive.stats.autogather=true
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Attachment: HIVE-11160.02.patch Collect column stats when set hive.stats.autogather=true Key: HIVE-11160 URL: https://issues.apache.org/jira/browse/HIVE-11160 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using Analyze command. In this patch, the column stats will also be collected automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11161) create patch for branch-1 for HiveOnTez: make explain user level = true as default
[ https://issues.apache.org/jira/browse/HIVE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-11161. Resolution: Fixed create patch for branch-1 for HiveOnTez: make explain user level = true as default -- Key: HIVE-11161 URL: https://issues.apache.org/jira/browse/HIVE-11161 Project: Hive Issue Type: Sub-task Components: Diagnosability Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Fix For: 2.0.0 Attachments: HIVE-11161.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11211) Reset the fields in JoinStatsRule in StatsRulesProcFactory
[ https://issues.apache.org/jira/browse/HIVE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11211: --- Summary: Reset the fields in JoinStatsRule in StatsRulesProcFactory (was: Reset the static fields in static class JoinStatsRule in StatsRulesProcFactory) Reset the fields in JoinStatsRule in StatsRulesProcFactory -- Key: HIVE-11211 URL: https://issues.apache.org/jira/browse/HIVE-11211 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11211.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11231) CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output of ba_table_union.q more stable
[ https://issues.apache.org/jira/browse/HIVE-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11231: --- Attachment: HIVE-11231.01.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output of ba_table_union.q more stable - Key: HIVE-11231 URL: https://issues.apache.org/jira/browse/HIVE-11231 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Minor Attachments: HIVE-11231.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11232) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the output of select_same_col.q
[ https://issues.apache.org/jira/browse/HIVE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11232: --- Attachment: HIVE-11232.01.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the output of select_same_col.q - Key: HIVE-11232 URL: https://issues.apache.org/jira/browse/HIVE-11232 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11232.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11165) Calcite planner might have a thread-safety issue compiling in parallel
[
https://issues.apache.org/jira/browse/HIVE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621286#comment-14621286
]
Pengcheng Xiong commented on HIVE-11165:
I attached query 13 here. I did not know the root cause yet but I saw lots of
predicates. I suspect that this is related to the recent optimization on PPD?
[~jpullokkaran]?
{code}
select avg(ss_quantity)
,avg(ss_ext_sales_price)
,avg(ss_ext_wholesale_cost)
,sum(ss_ext_wholesale_cost)
from store_sales
,store
,customer_demographics
,household_demographics
,customer_address
,date_dim
where store.s_store_sk = store_sales.ss_store_sk
and store_sales.ss_sold_date_sk = date_dim.d_date_sk and date_dim.d_year =
2001
and ss_sold_date between '2001-01-01' and '2001-12-31'
and((store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
and customer_demographics.cd_marital_status = 'M'
and customer_demographics.cd_education_status = '4 yr Degree'
and store_sales.ss_sales_price between 100.00 and 150.00
and household_demographics.hd_dep_count = 3
)or
(store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
and customer_demographics.cd_marital_status = 'D'
and customer_demographics.cd_education_status = 'Primary'
and store_sales.ss_sales_price between 50.00 and 100.00
and household_demographics.hd_dep_count = 1
) or
(store_sales.ss_hdemo_sk=household_demographics.hd_demo_sk
and customer_demographics.cd_demo_sk = ss_cdemo_sk
and customer_demographics.cd_marital_status = 'U'
and customer_demographics.cd_education_status = 'Advanced Degree'
and store_sales.ss_sales_price between 150.00 and 200.00
and household_demographics.hd_dep_count = 1
))
and((store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('KY', 'GA', 'NM')
and store_sales.ss_net_profit between 100 and 200
) or
(store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('MT', 'OR', 'IN')
and store_sales.ss_net_profit between 150 and 300
) or
(store_sales.ss_addr_sk = customer_address.ca_address_sk
and customer_address.ca_country = 'United States'
and customer_address.ca_state in ('WI', 'MO', 'WV')
and store_sales.ss_net_profit between 50 and 250
))
;
{code}
Calcite planner might have a thread-safety issue compiling in parallel
--
Key: HIVE-11165
URL: https://issues.apache.org/jira/browse/HIVE-11165
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 2.0.0
Reporter: Gopal V
Assignee: Laljo John Pullokkaran
Attachments: RunJar-2015-06-30.snapshot
After about 6 minutes trying to plan a query, the HiveServer2 was killed to
restore functionality to a test run.
The HEP planner is stuck on a TopologicalOrder traversal and there were no
queries being fed into the HiveServer2 after it got stuck.
TPC-DS query13 was the query in question, at 4 way parallel, which triggered
the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9780) Add another level of explain for RDBMS audience
[ https://issues.apache.org/jira/browse/HIVE-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008228#comment-15008228 ] Pengcheng Xiong commented on HIVE-9780: --- [~cartershanklin], thanks a lot for your comments. It is already on by default since 1.2.0. cc'ing [~jpullokkaran] > Add another level of explain for RDBMS audience > --- > > Key: HIVE-9780 > URL: https://issues.apache.org/jira/browse/HIVE-9780 > Project: Hive > Issue Type: New Feature > Components: Diagnosability >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Fix For: 1.2.0 > > Attachments: HIVE-9780.01.patch, HIVE-9780.02.patch, > HIVE-9780.03.patch, HIVE-9780.04.patch, HIVE-9780.05.patch, > HIVE-9780.06.patch, HIVE-9780.07.patch, HIVE-9780.08.patch, > HIVE-9780.09.patch, HIVE-9780.10.patch > > > Current Hive Explain (default) is targeted at MR Audience. We need a new > level of explain plan to be targeted at RDBMS audience. The explain requires > these: > 1) The focus needs to be on what part of the query is being executed rather > than internals of the engines > 2) There needs to be a clearly readable tree of operations > 3) Examples - Table scan should mention the table being scanned, the Sarg, > the size of table and expected cardinality after the Sarg'ed read. The join > should mention the table being joined with and the join condition. The > aggregate should mention the columns in the group-by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11629) CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join
[ https://issues.apache.org/jira/browse/HIVE-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715266#comment-14715266 ] Pengcheng Xiong commented on HIVE-11629: [~jcamachorodriguez], as per [~jpullokkaran]'s request, here is one more that needs your review. Thanks. CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join -- Key: HIVE-11629 URL: https://issues.apache.org/jira/browse/HIVE-11629 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11629.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11614: --- Attachment: HIVE-11614.02.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem - Key: HIVE-11614 URL: https://issues.apache.org/jira/browse/HIVE-11614 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11646) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator
[ https://issues.apache.org/jira/browse/HIVE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717418#comment-14717418 ] Pengcheng Xiong commented on HIVE-11646: [~jcamachorodriguez], as per [~jpullokkaran]'s request, could you also review this patch? Thanks. CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator --- Key: HIVE-11646 URL: https://issues.apache.org/jira/browse/HIVE-11646 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11646.01.patch Current return path only supports a single windowing spec. All the following window spec will overwrite the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11629) CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join
[ https://issues.apache.org/jira/browse/HIVE-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11629: --- Attachment: HIVE-11629.02.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join -- Key: HIVE-11629 URL: https://issues.apache.org/jira/browse/HIVE-11629 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11629.01.patch, HIVE-11629.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11646) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator
[
https://issues.apache.org/jira/browse/HIVE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721162#comment-14721162
]
Pengcheng Xiong commented on HIVE-11646:
[~jcamachorodriguez], thanks for your comments. The reality is that, it is
overwriting the previous window specs. For example {code}select p_mfgr,
p_retailprice, p_size,
round(sum(p_retailprice) over w1 , 2) = round(sum(lag(p_retailprice,1,0.0))
over w1 + last_value(p_retailprice) over w1 , 2),
max(p_retailprice) over w1 - min(p_retailprice) over w1 =
last_value(p_retailprice) over w1 - first_value(p_retailprice) over w1
from part
window w1 as (distribute by p_mfgr sort by p_retailprice){code}. And, that is
why I {code}getWindowColumnAlias() {code}. I will add this test case later.
CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple
window spec for PTF operator
---
Key: HIVE-11646
URL: https://issues.apache.org/jira/browse/HIVE-11646
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11646.01.patch
Current return path only supports a single windowing spec. All the following
window spec will overwrite the first one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9720) Metastore does not properly migrate column stats when renaming a table across databases.
[
https://issues.apache.org/jira/browse/HIVE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721406#comment-14721406
]
Pengcheng Xiong commented on HIVE-9720:
---
[~ctang.ma] and [~xuefuz], sorry to trouble you guys. [~ashutoshc] and I
recently met a similar problem and we tracked it here. Basically, we have a
partitioned table and then it is renamed. The column stats of the partitions
are all gone because of the renaming. Lines 223-224 of HiveAlterHandler.java
did this. This is good because it at least prevents the bug from happening. We
are wondering, when you guys did this patch, did you consider the option to
update the columns stats for the partitions (change the table name) rather than
deleting them? It seems a better option? Thanks!
Metastore does not properly migrate column stats when renaming a table across
databases.
Key: HIVE-9720
URL: https://issues.apache.org/jira/browse/HIVE-9720
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.13.1
Reporter: Alexander Behm
Assignee: Chaoyu Tang
Fix For: 1.2.0
Attachments: HIVE-9720.1.patch, HIVE-9720.1.patch, HIVE-9720.patch
It appears that the Hive Metastore does not properly migrate column
statistics when renaming a table across databases. While renaming across
databases is not supported in HiveQL, it can be done via the Metastore Thrift
API.
The problem is that such a newly renamed table cannot be dropped (unless
renamed back to its original database/name).
Here are steps for reproducing the issue.
1. From the Hive shell/beeline:
{code}
create database db1;
create database db2;
create table db1.mv (i int);
use db1;
analyze table mv compute statistics for columns i;
{code}
2. From a Java program:
{code}
public static void main(String[] args) throws Exception {
HiveConf conf = new HiveConf(MetaStoreClientPool.class);
HiveMetaStoreClient hiveClient = new HiveMetaStoreClient(conf);
Table t = hiveClient.getTable(db1, mv);
t.setDbName(db2);
t.setTableName(mv2);
hiveClient.alter_table(db1, mv, t);
}
{code}
3. From the Hive shell/beeline:
{code}
drop table db2.mv2;
{code}
Stack shown when running 3:
{code}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:javax.jdo.JDODataStoreException: Exception thrown
flushing changes to datastore
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at
org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
at
org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
[jira] [Commented] (HIVE-11629) CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join
[ https://issues.apache.org/jira/browse/HIVE-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721423#comment-14721423 ] Pengcheng Xiong commented on HIVE-11629: The test case failures are unrelated and it passed on my laptop. Pushed to master. Thanks [~jcamachorodriguez] for the review! CBO: Calcite Operator To Hive Operator (Calcite Return Path) : fix the filter expressions for full outer join and right outer join -- Key: HIVE-11629 URL: https://issues.apache.org/jira/browse/HIVE-11629 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11629.01.patch, HIVE-11629.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9720) Metastore does not properly migrate column stats when renaming a table across databases.
[
https://issues.apache.org/jira/browse/HIVE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-9720:
--
Attachment: HIVE-9720-branch-1.0.01.patch
> Metastore does not properly migrate column stats when renaming a table across
> databases.
>
>
> Key: HIVE-9720
> URL: https://issues.apache.org/jira/browse/HIVE-9720
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
> Fix For: 1.2.0
>
> Attachments: HIVE-9720-branch-1.0.01.patch, HIVE-9720.1.patch,
> HIVE-9720.1.patch, HIVE-9720.patch
>
>
> It appears that the Hive Metastore does not properly migrate column
> statistics when renaming a table across databases. While renaming across
> databases is not supported in HiveQL, it can be done via the Metastore Thrift
> API.
> The problem is that such a newly renamed table cannot be dropped (unless
> renamed back to its original database/name).
> Here are steps for reproducing the issue.
> 1. From the Hive shell/beeline:
> {code}
> create database db1;
> create database db2;
> create table db1.mv (i int);
> use db1;
> analyze table mv compute statistics for columns i;
> {code}
> 2. From a Java program:
> {code}
> public static void main(String[] args) throws Exception {
> HiveConf conf = new HiveConf(MetaStoreClientPool.class);
> HiveMetaStoreClient hiveClient = new HiveMetaStoreClient(conf);
> Table t = hiveClient.getTable("db1", "mv");
> t.setDbName("db2");
> t.setTableName("mv2");
> hiveClient.alter_table("db1", "mv", t);
> }
> {code}
> 3. From the Hive shell/beeline:
> {code}
> drop table db2.mv2;
> {code}
> Stack shown when running 3:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDODataStoreException: Exception thrown
> flushing changes to datastore
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
> at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> NestedThrowablesStackTrace:
> java.sql.BatchUpdateException: Batch entry 0 DELETE FROM "TBLS" WHERE
> "TBL_ID"='1621' was aborted. Call getNextException to see the cause.
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598)
> at
>
[jira] [Commented] (HIVE-9720) Metastore does not properly migrate column stats when renaming a table across databases.
[
https://issues.apache.org/jira/browse/HIVE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724101#comment-14724101
]
Pengcheng Xiong commented on HIVE-9720:
---
[~ctang.ma], thanks a lot for your reply. As you are the most experienced one
on this, we are looking forward to seeing if there is any better options. And
by the way, I submit a patch for branch-1.0 for maintenance purpose.
> Metastore does not properly migrate column stats when renaming a table across
> databases.
>
>
> Key: HIVE-9720
> URL: https://issues.apache.org/jira/browse/HIVE-9720
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
> Fix For: 1.2.0
>
> Attachments: HIVE-9720-branch-1.0.01.patch, HIVE-9720.1.patch,
> HIVE-9720.1.patch, HIVE-9720.patch
>
>
> It appears that the Hive Metastore does not properly migrate column
> statistics when renaming a table across databases. While renaming across
> databases is not supported in HiveQL, it can be done via the Metastore Thrift
> API.
> The problem is that such a newly renamed table cannot be dropped (unless
> renamed back to its original database/name).
> Here are steps for reproducing the issue.
> 1. From the Hive shell/beeline:
> {code}
> create database db1;
> create database db2;
> create table db1.mv (i int);
> use db1;
> analyze table mv compute statistics for columns i;
> {code}
> 2. From a Java program:
> {code}
> public static void main(String[] args) throws Exception {
> HiveConf conf = new HiveConf(MetaStoreClientPool.class);
> HiveMetaStoreClient hiveClient = new HiveMetaStoreClient(conf);
> Table t = hiveClient.getTable("db1", "mv");
> t.setDbName("db2");
> t.setTableName("mv2");
> hiveClient.alter_table("db1", "mv", t);
> }
> {code}
> 3. From the Hive shell/beeline:
> {code}
> drop table db2.mv2;
> {code}
> Stack shown when running 3:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDODataStoreException: Exception thrown
> flushing changes to datastore
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
> at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> NestedThrowablesStackTrace:
> java.sql.BatchUpdateException: Batch entry 0 DELETE FROM "TBLS" WHERE
> "TBL_ID"='1621' was aborted. Call
[jira] [Updated] (HIVE-11600) Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
[
https://issues.apache.org/jira/browse/HIVE-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11600:
---
Attachment: HIVE-11600.05.patch
> Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
>
>
> Key: HIVE-11600
> URL: https://issues.apache.org/jira/browse/HIVE-11600
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11600.01.patch, HIVE-11600.02.patch,
> HIVE-11600.03.patch, HIVE-11600.04.patch, HIVE-11600.05.patch
>
>
> Current hive only support single column in clause, e.g.,
> {code}select * from src where col0 in (v1,v2,v3);{code}
> We want it to support
> {code}select * from src where (col0,col1+3) in
> ((col0+v1,v2),(v3,v4-col1));{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11699) Support special characters in table names
[ https://issues.apache.org/jira/browse/HIVE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11699: --- Description: Right now table names can only be "[a-zA-z_0-9]+". This patch tries to investigate how much change there should be if we would like to support special characters, e.g., "/" in table names. (was: Right now table names can only be "[a-zA-z_0-9]+". This patch tries to investigate how much change there should be if we would like to support special characters, e.g., "\" in table names.) > Support special characters in table names > - > > Key: HIVE-11699 > URL: https://issues.apache.org/jira/browse/HIVE-11699 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > Right now table names can only be "[a-zA-z_0-9]+". This patch tries to > investigate how much change there should be if we would like to support > special characters, e.g., "/" in table names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11600) Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
[
https://issues.apache.org/jira/browse/HIVE-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725723#comment-14725723
]
Pengcheng Xiong commented on HIVE-11600:
The test case failures are unrelated and they passed on my laptop.
[~jpullokkaran], could you please take a look? Thanks.
> Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
>
>
> Key: HIVE-11600
> URL: https://issues.apache.org/jira/browse/HIVE-11600
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11600.01.patch, HIVE-11600.02.patch,
> HIVE-11600.03.patch, HIVE-11600.04.patch, HIVE-11600.05.patch
>
>
> Current hive only support single column in clause, e.g.,
> {code}select * from src where col0 in (v1,v2,v3);{code}
> We want it to support
> {code}select * from src where (col0,col1+3) in
> ((col0+v1,v2),(v3,v4-col1));{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11699) Support special characters in quoted table names
[ https://issues.apache.org/jira/browse/HIVE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11699: --- Attachment: HIVE-11699.01.patch initial patch without test cases. > Support special characters in quoted table names > > > Key: HIVE-11699 > URL: https://issues.apache.org/jira/browse/HIVE-11699 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11699.01.patch > > > Right now table names can only be "[a-zA-z_0-9]+". This patch tries to > investigate how much change there should be if we would like to support > special characters, e.g., "/" in table names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11699) Support special characters in quoted table names
[
https://issues.apache.org/jira/browse/HIVE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725846#comment-14725846
]
Pengcheng Xiong commented on HIVE-11699:
[~sershe], thanks a lot for the comments. I will address the character issues
later. And, I also thought about extending it to column names. The final
question is actually very critical and needs more work. Right now, my patch
only works one way, that is, we only do the encoding, but we do not do
decoding. So a user can use {code}create table `h/d` (col string);{code} and
also query with {code}select * from `h/d`{code}. However, {code}show
tables{code} will give you {code}h$47${code}.
> Support special characters in quoted table names
>
>
> Key: HIVE-11699
> URL: https://issues.apache.org/jira/browse/HIVE-11699
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11699.01.patch
>
>
> Right now table names can only be "[a-zA-z_0-9]+". This patch tries to
> investigate how much change there should be if we would like to support
> special characters, e.g., "/" in table names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11699) Support special characters in quoted table names
[ https://issues.apache.org/jira/browse/HIVE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11699: --- Summary: Support special characters in quoted table names (was: Support special characters in table names) > Support special characters in quoted table names > > > Key: HIVE-11699 > URL: https://issues.apache.org/jira/browse/HIVE-11699 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > Right now table names can only be "[a-zA-z_0-9]+". This patch tries to > investigate how much change there should be if we would like to support > special characters, e.g., "/" in table names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11699) Support special characters in quoted table names
[ https://issues.apache.org/jira/browse/HIVE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725781#comment-14725781 ] Pengcheng Xiong commented on HIVE-11699: [~sershe], could you point me to the places where i can find test cases for the unicode characters as table names? Thanks. > Support special characters in quoted table names > > > Key: HIVE-11699 > URL: https://issues.apache.org/jira/browse/HIVE-11699 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11699.01.patch > > > Right now table names can only be "[a-zA-z_0-9]+". This patch tries to > investigate how much change there should be if we would like to support > special characters, e.g., "/" in table names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11600) Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
[
https://issues.apache.org/jira/browse/HIVE-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731079#comment-14731079
]
Pengcheng Xiong commented on HIVE-11600:
Based on the review comments, pushed it to master. Thanks for [~jpullokkaran]'s
review.
> Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
>
>
> Key: HIVE-11600
> URL: https://issues.apache.org/jira/browse/HIVE-11600
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11600.01.patch, HIVE-11600.02.patch,
> HIVE-11600.03.patch, HIVE-11600.04.patch, HIVE-11600.05.patch
>
>
> Current hive only support single column in clause, e.g.,
> {code}select * from src where col0 in (v1,v2,v3);{code}
> We want it to support
> {code}select * from src where (col0,col1+3) in
> ((col0+v1,v2),(v3,v4-col1));{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-11703) Make RegExp and RLike reserved keywords
[ https://issues.apache.org/jira/browse/HIVE-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-11703. Resolution: Fixed > Make RegExp and RLike reserved keywords > --- > > Key: HIVE-11703 > URL: https://issues.apache.org/jira/browse/HIVE-11703 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > RegExp and RLike are treated as "precedenceEqualNegatableOperator" in Hive. > They actually come from MySQL. Both of them are not keywords in SQL2011. But > they are reserved keywords in MySQL. Making them reserved ones can eliminate > the current 14 ambiguities that we have in current Hive. If users still would > like to use them as identifiers/function names, users can "set > hive.support.sql11.reserved.keywords=false;" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11614: --- Attachment: HIVE-11614.04.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch, HIVE-11614.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11614: --- Attachment: (was: HIVE-11614.04.patch) > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch, HIVE-11614.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11614: --- Attachment: HIVE-11614.04.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch, HIVE-11614.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736106#comment-14736106 ] Pengcheng Xiong commented on HIVE-11614: [~jpullokkaran], could you please take a look? The test failure is unrelated and it also appeared in the other pre-commit runs. Thanks. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch, HIVE-11614.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11646) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator
[ https://issues.apache.org/jira/browse/HIVE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11646: --- Attachment: HIVE-11646.02.patch new patch with test cases [~jcamachorodriguez], could you take another look? Thanks. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple > window spec for PTF operator > --- > > Key: HIVE-11646 > URL: https://issues.apache.org/jira/browse/HIVE-11646 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11646.01.patch, HIVE-11646.02.patch > > > Current return path only supports a single windowing spec. All the following > window spec will overwrite the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731649#comment-14731649 ] Pengcheng Xiong commented on HIVE-11614: [~jpullokkaran], most of the test failures are due to different golden files (the new ones sounds better). The only thing that we need to worry about is the failures in TestJdbcDriver2. Could you take a look? Thanks. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11614) CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem
[ https://issues.apache.org/jira/browse/HIVE-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11614: --- Attachment: HIVE-11614.03.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after > order by has problem > - > > Key: HIVE-11614 > URL: https://issues.apache.org/jira/browse/HIVE-11614 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11614.01.patch, HIVE-11614.02.patch, > HIVE-11614.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11728) WITH clause uses regular table instead of intermidiate relation when regular table exits with same name as of intermidiate relation.
[ https://issues.apache.org/jira/browse/HIVE-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729913#comment-14729913 ] Pengcheng Xiong commented on HIVE-11728: [~chaitanyakul], this was fixed in https://issues.apache.org/jira/browse/HIVE-10698. Please apply the patch or upgrade your Hive. > WITH clause uses regular table instead of intermidiate relation when regular > table exits with same name as of intermidiate relation. > > > Key: HIVE-11728 > URL: https://issues.apache.org/jira/browse/HIVE-11728 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Parser >Affects Versions: 0.13.0, 0.14.0 > Environment: Linux, Hive 0.13 or 0.14, CDH or HDP cluster. >Reporter: Chaitanya Kulkarni >Assignee: Pengcheng Xiong > > If a table of name 'tab1' exists in a database, and within a SQL statement, a > CTE (with clause) is used with intermediate relation having same name as > 'tab1' then Hive uses 'tab1' regular table in query rather than using > intermediate relation. > Steps to recreate the issue: > 1. Create a table with name 'tab1'. > 2. Load some sample data in table 'tab1'. > 3. Write a hive query to use some CTE with intermediate relation name as > 'tab1', and execute the query. > e.g. with tab1 as (select * from orders), > select count(tab1.*) from tab1; > If we have a table with same name like 'tab1', and CTE is also using 'tab1' > as relation name, then query engine refers to 'tab1' which is a regular > table. This is not as per implementation in other database systems, like > PostgreSQL. > complete test script: > create table test > ( id int, > name varchar(100)); > > insert into test values (1, 'abc'); > insert into test values (2, 'xyz'); > > select * from test; > > with test as ( > select * from other_table) > select * from test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11728) WITH clause uses regular table instead of intermidiate relation when regular table exits with same name as of intermidiate relation.
[ https://issues.apache.org/jira/browse/HIVE-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-11728. Resolution: Duplicate > WITH clause uses regular table instead of intermidiate relation when regular > table exits with same name as of intermidiate relation. > > > Key: HIVE-11728 > URL: https://issues.apache.org/jira/browse/HIVE-11728 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Parser >Affects Versions: 0.13.0, 0.14.0 > Environment: Linux, Hive 0.13 or 0.14, CDH or HDP cluster. >Reporter: Chaitanya Kulkarni >Assignee: Pengcheng Xiong > > If a table of name 'tab1' exists in a database, and within a SQL statement, a > CTE (with clause) is used with intermediate relation having same name as > 'tab1' then Hive uses 'tab1' regular table in query rather than using > intermediate relation. > Steps to recreate the issue: > 1. Create a table with name 'tab1'. > 2. Load some sample data in table 'tab1'. > 3. Write a hive query to use some CTE with intermediate relation name as > 'tab1', and execute the query. > e.g. with tab1 as (select * from orders), > select count(tab1.*) from tab1; > If we have a table with same name like 'tab1', and CTE is also using 'tab1' > as relation name, then query engine refers to 'tab1' which is a regular > table. This is not as per implementation in other database systems, like > PostgreSQL. > complete test script: > create table test > ( id int, > name varchar(100)); > > insert into test values (1, 'abc'); > insert into test values (2, 'xyz'); > > select * from test; > > with test as ( > select * from other_table) > select * from test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11728) WITH clause uses regular table instead of intermidiate relation when regular table exits with same name as of intermidiate relation.
[ https://issues.apache.org/jira/browse/HIVE-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-11728: -- Assignee: Pengcheng Xiong > WITH clause uses regular table instead of intermidiate relation when regular > table exits with same name as of intermidiate relation. > > > Key: HIVE-11728 > URL: https://issues.apache.org/jira/browse/HIVE-11728 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Parser >Affects Versions: 0.13.0, 0.14.0 > Environment: Linux, Hive 0.13 or 0.14, CDH or HDP cluster. >Reporter: Chaitanya Kulkarni >Assignee: Pengcheng Xiong > > If a table of name 'tab1' exists in a database, and within a SQL statement, a > CTE (with clause) is used with intermediate relation having same name as > 'tab1' then Hive uses 'tab1' regular table in query rather than using > intermediate relation. > Steps to recreate the issue: > 1. Create a table with name 'tab1'. > 2. Load some sample data in table 'tab1'. > 3. Write a hive query to use some CTE with intermediate relation name as > 'tab1', and execute the query. > e.g. with tab1 as (select * from orders), > select count(tab1.*) from tab1; > If we have a table with same name like 'tab1', and CTE is also using 'tab1' > as relation name, then query engine refers to 'tab1' which is a regular > table. This is not as per implementation in other database systems, like > PostgreSQL. > complete test script: > create table test > ( id int, > name varchar(100)); > > insert into test values (1, 'abc'); > insert into test values (2, 'xyz'); > > select * from test; > > with test as ( > select * from other_table) > select * from test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11623) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the tableAlias for ReduceSink operator
[ https://issues.apache.org/jira/browse/HIVE-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11623: --- Attachment: HIVE-11623.03.patch address [~jcamachorodriguez]'s comments. [~jpullokkaran]. do you still want to take a look? Thanks. CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the tableAlias for ReduceSink operator Key: HIVE-11623 URL: https://issues.apache.org/jira/browse/HIVE-11623 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11623.01.patch, HIVE-11623.02.patch, HIVE-11623.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11646) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator
[ https://issues.apache.org/jira/browse/HIVE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11646: --- Attachment: HIVE-11646.01.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator --- Key: HIVE-11646 URL: https://issues.apache.org/jira/browse/HIVE-11646 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11646.01.patch Current return path only supports a single windowing spec. All the following window spec will overwrite the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11646) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator
[ https://issues.apache.org/jira/browse/HIVE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11646: --- Attachment: (was: HIVE-11646.01.patch) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix multiple window spec for PTF operator --- Key: HIVE-11646 URL: https://issues.apache.org/jira/browse/HIVE-11646 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11646.01.patch Current return path only supports a single windowing spec. All the following window spec will overwrite the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11703) Make RegExp and RLike reserved keywords
[ https://issues.apache.org/jira/browse/HIVE-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734032#comment-14734032 ] Pengcheng Xiong commented on HIVE-11703: [~leftylev], thanks for your comments. I have modified the wiki accordingly. > Make RegExp and RLike reserved keywords > --- > > Key: HIVE-11703 > URL: https://issues.apache.org/jira/browse/HIVE-11703 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > > RegExp and RLike are treated as "precedenceEqualNegatableOperator" in Hive. > They actually come from MySQL. Both of them are not keywords in SQL2011. But > they are reserved keywords in MySQL. Making them reserved ones can eliminate > the current 14 ambiguities that we have in current Hive. If users still would > like to use them as identifiers/function names, users can "set > hive.support.sql11.reserved.keywords=false;" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11301) thrift metastore issue when getting stats results in disconnect
[
https://issues.apache.org/jira/browse/HIVE-11301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11301:
---
Fix Version/s: (was: 1.2.0)
1.2.2
> thrift metastore issue when getting stats results in disconnect
> ---
>
> Key: HIVE-11301
> URL: https://issues.apache.org/jira/browse/HIVE-11301
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Sergey Shelukhin
>Assignee: Pengcheng Xiong
> Fix For: 1.3.0, 2.0.0, 1.2.2
>
> Attachments: HIVE-11301.01.patch, HIVE-11301.02.patch
>
>
> On metastore side it looks like this:
> {noformat}
> 2015-07-17 20:32:27,795 ERROR [pool-3-thread-150]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing
> of message.
> org.apache.thrift.protocol.TProtocolException: Required field 'colStats' is
> unset! Struct:AggrStats(colStats:null, partsFound:0)
> at
> org.apache.hadoop.hive.metastore.api.AggrStats.validate(AggrStats.java:389)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.validate(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.write(ThriftHiveMetastore.java)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> and then
> {noformat}
> 2015-07-17 20:32:27,796 WARN [pool-3-thread-150]:
> transport.TIOStreamTransport (TIOStreamTransport.java:close(112)) - Error
> closing output stream.
> java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> at
> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
> at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.close(TFilterTransport.java:52)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Which on client manifests as
> {noformat}
> 2015-07-17 20:32:27,796 WARN [main()]: metastore.RetryingMetaStoreClient
> (RetryingMetaStoreClient.java:invoke(187)) - MetaStoreClient lost connection.
> Attempting to reconnect.
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
>
[jira] [Commented] (HIVE-11786) Deprecate the use of redundant column in colunm stats related tables
[ https://issues.apache.org/jira/browse/HIVE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739130#comment-14739130 ] Pengcheng Xiong commented on HIVE-11786: I totally agree that these redundant columns violate database normalization rules and cause a lot of inconvenience. I also suffered a lot before. It seems to be a big change? also cc'ing [~ashutoshc] and [~alangates] to watch if it will affect HBase based metastore? Thanks. > Deprecate the use of redundant column in colunm stats related tables > > > Key: HIVE-11786 > URL: https://issues.apache.org/jira/browse/HIVE-11786 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > > The stats tables such as TAB_COL_STATS, PART_COL_STATS have redundant columns > such as DB_NAME, TABLE_NAME, PARTITION_NAME since these tables already have > foreign key like TBL_ID, or PART_ID referencing to TBLS or PARTITIONS. > These redundant columns violate database normalization rules and cause a lot > of inconvenience (sometimes difficult) in column stats related feature > implementation. For example, when renaming a table, we have to update > TABLE_NAME column in these tables as well which is unnecessary. > This JIRA is first to deprecate the use of these columns at HMS code level. A > followed JIRA is to be opened to focus on DB schema change and upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.01.patch initial patch, need to add more test cases. > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11792) User explain in tez does not preserve ordering
[
https://issues.apache.org/jira/browse/HIVE-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739830#comment-14739830
]
Pengcheng Xiong commented on HIVE-11792:
I just checked the output change of explain_user1,2,3, all of them looked good
to me. +1 pending QA tests.
> User explain in tez does not preserve ordering
> --
>
> Key: HIVE-11792
> URL: https://issues.apache.org/jira/browse/HIVE-11792
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-11792.1.patch
>
>
> HIVE-11037 introduced user level explain. The output of operator attributes
> are JSONObject which does not preserve ordering. This is causing diffs in
> test output when run under tez vs llap like below
> {code}
> -partition:{"ts":"2012-01-03+14:46:31","ds":"2012-01-03"}
> -
> table:{"serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part","input
> format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output
> format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat"}
> +partition:{"ds":"2012-01-03","ts":"2012-01-03+14:46:31"}
> +table:{"name:":"default.src_orc_merge_test_part","input
> format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output
> format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde"}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10328) Enable new return path for cbo
[ https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-10328: --- Attachment: HIVE-10328.10.patch > Enable new return path for cbo > -- > > Key: HIVE-10328 > URL: https://issues.apache.org/jira/browse/HIVE-10328 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch, > HIVE-10328.2.patch, HIVE-10328.3.patch, HIVE-10328.4.patch, > HIVE-10328.4.patch, HIVE-10328.5.patch, HIVE-10328.6.patch, > HIVE-10328.7.patch, HIVE-10328.8.patch, HIVE-10328.9.patch, HIVE-10328.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10328) Enable new return path for cbo
[ https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-10328: --- Attachment: HIVE-10328.11.patch > Enable new return path for cbo > -- > > Key: HIVE-10328 > URL: https://issues.apache.org/jira/browse/HIVE-10328 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch, > HIVE-10328.11.patch, HIVE-10328.2.patch, HIVE-10328.3.patch, > HIVE-10328.4.patch, HIVE-10328.4.patch, HIVE-10328.5.patch, > HIVE-10328.6.patch, HIVE-10328.7.patch, HIVE-10328.8.patch, > HIVE-10328.9.patch, HIVE-10328.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11815) Correct the column/table names in subquery expression when create a view
[
https://issues.apache.org/jira/browse/HIVE-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11815:
---
Attachment: HIVE-11815.01.patch
> Correct the column/table names in subquery expression when create a view
>
>
> Key: HIVE-11815
> URL: https://issues.apache.org/jira/browse/HIVE-11815
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11815.01.patch
>
>
> Right now Hive does not quote column/table names in subquery expression when
> create a view. For example
> {code}
> hive>
> > create table tc (`@d` int);
> OK
> Time taken: 0.119 seconds
> hive> create view tcv as select * from tc b where exists (select a.`@d` from
> tc a where b.`@d`=a.`@d`);
> OK
> Time taken: 0.075 seconds
> hive> describe extended tcv;
> OK
> @dint
> Detailed Table InformationTable(tableName:tcv, dbName:default,
> owner:pxiong, createTime:1442250005, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:@d, type:int, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{transient_lastDdlTime=1442250005},
> viewOriginalText:select * from tc b where exists (select a.@d from tc a where
> b.@d=a.@d), viewExpandedText:select `b`.`@d` from `default`.`tc` `b` where
> exists (select a.@d from tc a where b.@d=a.@d), tableType:VIRTUAL_VIEW)
> Time taken: 0.063 seconds, Fetched: 3 row(s)
> hive> select * from tcv;
> FAILED: SemanticException line 1:63 character '@' not supported here
> line 1:84 character '@' not supported here
> line 1:89 character '@' not supported here in definition of VIEW tcv [
> select `b`.`@d` from `default`.`tc` `b` where exists (select a.@d from tc a
> where b.@d=a.@d)
> ] used as tcv at Line 1:14
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11815) Correct the column/table names in subquery expression when creating a view
[
https://issues.apache.org/jira/browse/HIVE-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11815:
---
Summary: Correct the column/table names in subquery expression when
creating a view (was: Correct the column/table names in subquery expression
when create a view)
> Correct the column/table names in subquery expression when creating a view
> --
>
> Key: HIVE-11815
> URL: https://issues.apache.org/jira/browse/HIVE-11815
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11815.01.patch
>
>
> Right now Hive does not quote column/table names in subquery expression when
> create a view. For example
> {code}
> hive>
> > create table tc (`@d` int);
> OK
> Time taken: 0.119 seconds
> hive> create view tcv as select * from tc b where exists (select a.`@d` from
> tc a where b.`@d`=a.`@d`);
> OK
> Time taken: 0.075 seconds
> hive> describe extended tcv;
> OK
> @dint
> Detailed Table InformationTable(tableName:tcv, dbName:default,
> owner:pxiong, createTime:1442250005, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:@d, type:int, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{transient_lastDdlTime=1442250005},
> viewOriginalText:select * from tc b where exists (select a.@d from tc a where
> b.@d=a.@d), viewExpandedText:select `b`.`@d` from `default`.`tc` `b` where
> exists (select a.@d from tc a where b.@d=a.@d), tableType:VIRTUAL_VIEW)
> Time taken: 0.063 seconds, Fetched: 3 row(s)
> hive> select * from tcv;
> FAILED: SemanticException line 1:63 character '@' not supported here
> line 1:84 character '@' not supported here
> line 1:89 character '@' not supported here in definition of VIEW tcv [
> select `b`.`@d` from `default`.`tc` `b` where exists (select a.@d from tc a
> where b.@d=a.@d)
> ] used as tcv at Line 1:14
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11815) Correct the column/table names in subquery expression when create a view
[
https://issues.apache.org/jira/browse/HIVE-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11815:
---
Attachment: (was: HIVE-11815.01.patch)
> Correct the column/table names in subquery expression when create a view
>
>
> Key: HIVE-11815
> URL: https://issues.apache.org/jira/browse/HIVE-11815
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> Right now Hive does not quote column/table names in subquery expression when
> create a view. For example
> {code}
> hive>
> > create table tc (`@d` int);
> OK
> Time taken: 0.119 seconds
> hive> create view tcv as select * from tc b where exists (select a.`@d` from
> tc a where b.`@d`=a.`@d`);
> OK
> Time taken: 0.075 seconds
> hive> describe extended tcv;
> OK
> @dint
> Detailed Table InformationTable(tableName:tcv, dbName:default,
> owner:pxiong, createTime:1442250005, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:@d, type:int, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{transient_lastDdlTime=1442250005},
> viewOriginalText:select * from tc b where exists (select a.@d from tc a where
> b.@d=a.@d), viewExpandedText:select `b`.`@d` from `default`.`tc` `b` where
> exists (select a.@d from tc a where b.@d=a.@d), tableType:VIRTUAL_VIEW)
> Time taken: 0.063 seconds, Fetched: 3 row(s)
> hive> select * from tcv;
> FAILED: SemanticException line 1:63 character '@' not supported here
> line 1:84 character '@' not supported here
> line 1:89 character '@' not supported here in definition of VIEW tcv [
> select `b`.`@d` from `default`.`tc` `b` where exists (select a.@d from tc a
> where b.@d=a.@d)
> ] used as tcv at Line 1:14
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11815) Correct the column/table names in subquery expression when create a view
[
https://issues.apache.org/jira/browse/HIVE-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11815:
---
Attachment: HIVE-11815.01.patch
> Correct the column/table names in subquery expression when create a view
>
>
> Key: HIVE-11815
> URL: https://issues.apache.org/jira/browse/HIVE-11815
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> Right now Hive does not quote column/table names in subquery expression when
> create a view. For example
> {code}
> hive>
> > create table tc (`@d` int);
> OK
> Time taken: 0.119 seconds
> hive> create view tcv as select * from tc b where exists (select a.`@d` from
> tc a where b.`@d`=a.`@d`);
> OK
> Time taken: 0.075 seconds
> hive> describe extended tcv;
> OK
> @dint
> Detailed Table InformationTable(tableName:tcv, dbName:default,
> owner:pxiong, createTime:1442250005, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:@d, type:int, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{transient_lastDdlTime=1442250005},
> viewOriginalText:select * from tc b where exists (select a.@d from tc a where
> b.@d=a.@d), viewExpandedText:select `b`.`@d` from `default`.`tc` `b` where
> exists (select a.@d from tc a where b.@d=a.@d), tableType:VIRTUAL_VIEW)
> Time taken: 0.063 seconds, Fetched: 3 row(s)
> hive> select * from tcv;
> FAILED: SemanticException line 1:63 character '@' not supported here
> line 1:84 character '@' not supported here
> line 1:89 character '@' not supported here in definition of VIEW tcv [
> select `b`.`@d` from `default`.`tc` `b` where exists (select a.@d from tc a
> where b.@d=a.@d)
> ] used as tcv at Line 1:14
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11815) Correct the column/table names in subquery expression when creating a view
[
https://issues.apache.org/jira/browse/HIVE-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11815:
---
Attachment: HIVE-11815.02.patch
> Correct the column/table names in subquery expression when creating a view
> --
>
> Key: HIVE-11815
> URL: https://issues.apache.org/jira/browse/HIVE-11815
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11815.01.patch, HIVE-11815.02.patch
>
>
> Right now Hive does not quote column/table names in subquery expression when
> create a view. For example
> {code}
> hive>
> > create table tc (`@d` int);
> OK
> Time taken: 0.119 seconds
> hive> create view tcv as select * from tc b where exists (select a.`@d` from
> tc a where b.`@d`=a.`@d`);
> OK
> Time taken: 0.075 seconds
> hive> describe extended tcv;
> OK
> @dint
> Detailed Table InformationTable(tableName:tcv, dbName:default,
> owner:pxiong, createTime:1442250005, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:@d, type:int, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{transient_lastDdlTime=1442250005},
> viewOriginalText:select * from tc b where exists (select a.@d from tc a where
> b.@d=a.@d), viewExpandedText:select `b`.`@d` from `default`.`tc` `b` where
> exists (select a.@d from tc a where b.@d=a.@d), tableType:VIRTUAL_VIEW)
> Time taken: 0.063 seconds, Fetched: 3 row(s)
> hive> select * from tcv;
> FAILED: SemanticException line 1:63 character '@' not supported here
> line 1:84 character '@' not supported here
> line 1:89 character '@' not supported here in definition of VIEW tcv [
> select `b`.`@d` from `default`.`tc` `b` where exists (select a.@d from tc a
> where b.@d=a.@d)
> ] used as tcv at Line 1:14
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10982) Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver
[ https://issues.apache.org/jira/browse/HIVE-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740091#comment-14740091 ] Pengcheng Xiong commented on HIVE-10982: [~libing], thanks! cc'ing [~vgumashta] > Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver > -- > > Key: HIVE-10982 > URL: https://issues.apache.org/jira/browse/HIVE-10982 > Project: Hive > Issue Type: Improvement > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Bing Li >Assignee: Bing Li >Priority: Critical > > The current JDBC driver for Hive hard-code the value of setFetchSize to 50, > which will be a bottleneck for performance. > Pentaho filed this issue as http://jira.pentaho.com/browse/PDI-11511, whose > status is open. > Also it has discussion in > http://forums.pentaho.com/showthread.php?158381-Hive-JDBC-Query-too-slow-too-many-fetches-after-query-execution-Kettle-Xform > http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3ccacq46vevgrfqg5rwxnr1psgyz7dcf07mvlo8mm2qit3anm1...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
