[jira] [Updated] (HIVE-5530) null pointer exception when case returns null
[ https://issues.apache.org/jira/browse/HIVE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5530: --- Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) > null pointer exception when case returns null > - > > Key: HIVE-5530 > URL: https://issues.apache.org/jira/browse/HIVE-5530 > Project: Hive > Issue Type: Bug > Components: SQL >Affects Versions: 0.11.0 >Reporter: N Campbell >Assignee: Navis >Priority: Minor > Fix For: 1.2.0 > > Attachments: HIVE-5530.1.patch.txt, HIVE-5530.2.patch.txt > > > The following expression will cause an NPE > > select case when 1 = 1 then null end from t -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13138) Add client to communicate with interface, initial split setup
[ https://issues.apache.org/jira/browse/HIVE-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-13138. --- Resolution: Fixed Fix Version/s: llap Committed to branch llap > Add client to communicate with interface, initial split setup > - > > Key: HIVE-13138 > URL: https://issues.apache.org/jira/browse/HIVE-13138 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-13138.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13140) Wire the client to submit execution fragments
[ https://issues.apache.org/jira/browse/HIVE-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-13140. --- Resolution: Fixed Fix Version/s: llap Committed to branch llap > Wire the client to submit execution fragments > - > > Key: HIVE-13140 > URL: https://issues.apache.org/jira/browse/HIVE-13140 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth > Fix For: llap > > Attachments: HIVE-13140.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13141) Hive on Spark over HBase should accept parameters starting with "zookeeper.znode"
[ https://issues.apache.org/jira/browse/HIVE-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13141: - Attachment: HIVE-13141.patch > Hive on Spark over HBase should accept parameters starting with > "zookeeper.znode" > - > > Key: HIVE-13141 > URL: https://issues.apache.org/jira/browse/HIVE-13141 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Nemon Lou >Assignee: Nemon Lou >Priority: Minor > Attachments: HIVE-13141.patch > > > HBase related paramters has been added by HIVE-12708. > Following the same way,parameters starting with "zookeeper.znode" should be > add too,which are also HBase related paramters . > Refering to http://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/ > I have seen a failure with Hive on Spark over HBase due to customize > zookeeper.znode.parent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13141) Hive on Spark over HBase should accept parameters starting with "zookeeper.znode"
[ https://issues.apache.org/jira/browse/HIVE-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13141: - Status: Patch Available (was: Open) > Hive on Spark over HBase should accept parameters starting with > "zookeeper.znode" > - > > Key: HIVE-13141 > URL: https://issues.apache.org/jira/browse/HIVE-13141 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou >Priority: Minor > Attachments: HIVE-13141.patch > > > HBase related paramters has been added by HIVE-12708. > Following the same way,parameters starting with "zookeeper.znode" should be > add too,which are also HBase related paramters . > Refering to http://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/ > I have seen a failure with Hive on Spark over HBase due to customize > zookeeper.znode.parent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160338#comment-15160338 ] Ashutosh Chauhan commented on HIVE-13095: - +1 pending test run > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13140) Wire the client to submit execution fragments
[ https://issues.apache.org/jira/browse/HIVE-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13140: -- Attachment: HIVE-13140.01.patch > Wire the client to submit execution fragments > - > > Key: HIVE-13140 > URL: https://issues.apache.org/jira/browse/HIVE-13140 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth > Attachments: HIVE-13140.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13138) Add client to communicate with interface, initial split setup
[ https://issues.apache.org/jira/browse/HIVE-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13138: -- Attachment: HIVE-13138.01.patch > Add client to communicate with interface, initial split setup > - > > Key: HIVE-13138 > URL: https://issues.apache.org/jira/browse/HIVE-13138 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13138.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13139) Unfold TOK_ALLCOLREF of source table/view at QB stage
[ https://issues.apache.org/jira/browse/HIVE-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13139: --- Summary: Unfold TOK_ALLCOLREF of source table/view at QB stage (was: Unfold TOK_ALLCOLREF at QB stage) > Unfold TOK_ALLCOLREF of source table/view at QB stage > - > > Key: HIVE-13139 > URL: https://issues.apache.org/jira/browse/HIVE-13139 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > TOK_ALLCOLREF is now unfolded to real column names at genPlan stage. This > patch tries to unfold it at QB stage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13138) Add client to communicate with interface, initial split setup
[ https://issues.apache.org/jira/browse/HIVE-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13138: -- Summary: Add client to communicate with interface, initial split setup (was: Add an initial client to communicate with the daemons) > Add client to communicate with interface, initial split setup > - > > Key: HIVE-13138 > URL: https://issues.apache.org/jira/browse/HIVE-13138 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13095: --- Status: Open (was: Patch Available) > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13095: --- Status: Patch Available (was: Open) > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12808) Logical PPD: Push filter clauses through PTF(Windowing) into TS
[
https://issues.apache.org/jira/browse/HIVE-12808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160315#comment-15160315
]
Ashutosh Chauhan commented on HIVE-12808:
-
+1
> Logical PPD: Push filter clauses through PTF(Windowing) into TS
> ---
>
> Key: HIVE-12808
> URL: https://issues.apache.org/jira/browse/HIVE-12808
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Gopal V
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-12808.01.patch, HIVE-12808.02.patch,
> HIVE-12808.03.patch, HIVE-12808.04.patch, HIVE-12808.05.patch
>
>
> Simplified repro case of [HCC
> #8880|https://community.hortonworks.com/questions/8880/hive-on-tez-pushdown-predicate-doesnt-work-in-part.html],
> with the slow query showing the push-down miss.
> And the manually rewritten query to indicate the expected one.
> Part of the problem could be the window range not being split apart for PPD,
> but the FIL is not pushed down even if the rownum filter is removed.
> {code}
> create temporary table positions (regionid string, id bigint, deviceid
> string, ts string);
> insert into positions values('1d6a0be1-6366-4692-9597-ebd5cd0f01d1',
> 1422792010, '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-01'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-01'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-02'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-02');
> -- slow query
> explain
> WITH t1 AS
> (
> SELECT *,
> Row_number() over ( PARTITION BY regionid, id, deviceid
> ORDER BY ts DESC) AS rownos
> FROM positions ),
> latestposition as (
>SELECT *
>FROM t1
>WHERE rownos = 1)
> SELECT *
> FROM latestposition
> WHERE regionid='1d6a0be1-6366-4692-9597-ebd5cd0f01d1'
> ANDid=1422792010
> ANDdeviceid='6c5d1a30-2331-448b-a726-a380d6b3a432';
> -- fast query
> explain
> WITH t1 AS
> (
> SELECT *,
> Row_number() over ( PARTITION BY regionid, id, deviceid
> ORDER BY ts DESC) AS rownos
> FROM positions
> WHERE regionid='1d6a0be1-6366-4692-9597-ebd5cd0f01d1'
> ANDid=1422792010
> ANDdeviceid='6c5d1a30-2331-448b-a726-a380d6b3a432'
> ),latestposition as (
>SELECT *
>FROM t1
>WHERE rownos = 1)
> SELECT *
> FROM latestposition
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Status: In Progress (was: Patch Available) looking into the failed tests > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Attachment: HIVE-13096.02.patch > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Status: Patch Available (was: In Progress) > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13096 started by Jesus Camacho Rodriguez. -- > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Status: Open (was: Patch Available) > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[
https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160221#comment-15160221
]
Jesus Camacho Rodriguez commented on HIVE-13096:
[~ashutoshc], thanks for checking.
In fact, I had missed that the {{getCumulativeCost}} method was overriden for
Join operators (as it is part of _HiveRelMdDistinctRowCount_ ), thanks for
catching that. However, the default {{getCumulativeCost}} is still applied over
the rest of the operators (it is in _RelMdPercentageOriginalRows_).
Hence, to mimic CBO cumulative cardinality estimation, we should combine both.
I have uploaded a new patch with the updated method.
> Cost to choose side table in MapJoin conversion based on cumulative
> cardinality
> ---
>
> Key: HIVE-13096
> URL: https://issues.apache.org/jira/browse/HIVE-13096
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13096.01.patch, HIVE-13096.patch
>
>
> HIVE-11954 changed the logic to choose the side table in the MapJoin
> conversion algorithm. Initial heuristic for the cost was based on number of
> heavyweight operators.
> This extends that work so the heuristic is based on accumulate cardinality.
> In the future, we should choose the side based on total latency for the input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs
[ https://issues.apache.org/jira/browse/HIVE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13135: --- Attachment: HIVE-13135.2.patch > LLAP: HTTPS Webservices needs trusted keystore configs > -- > > Key: HIVE-13135 > URL: https://issues.apache.org/jira/browse/HIVE-13135 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13135.1.patch, HIVE-13135.2.patch > > > ssl-server.xml is not picked up internally by the hive-common HttpServer > impl, unlike the default configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-12092:
---
Status: Open (was: Patch Available)
Revisiting idea as a general UDF -> SARGs interface.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13115) MetaStore Direct SQL getPartitions call fail when the columns schemas for a partition are null
[
https://issues.apache.org/jira/browse/HIVE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-13115:
---
Attachment: HIVE-13115.reproduce.issue.patch
I've attached a patch which reproduce the issue. Though there are no test
failures, since the code falls back on ORM. But we can see the exception (I've
added in the description) in the logs.
We can apply the patch and run the following command.
{noformat}
mvn clean test -Dtest=TestEmbeddedHiveMetaStore -Phadoop-2
{noformat}
Then check the logs in {{itests/hive-unit/target/tmp/log/hive.log}}, we can see
the exception reproduced.
> MetaStore Direct SQL getPartitions call fail when the columns schemas for a
> partition are null
> --
>
> Key: HIVE-13115
> URL: https://issues.apache.org/jira/browse/HIVE-13115
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Attachments: HIVE-13115.reproduce.issue.patch
>
>
> We are seeing the following exception in our MetaStore logs
> {noformat}
> 2016-02-11 00:00:19,002 DEBUG metastore.MetaStoreDirectSql
> (MetaStoreDirectSql.java:timingTrace(602)) - Direct SQL query in 5.842372ms +
> 1.066728ms, the query is [select "PARTITIONS"."PART_ID" from "PARTITIONS"
> inner join "TBLS" on "PART
> ITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? order by
> "PART_NAME" asc]
> 2016-02-11 00:00:19,021 ERROR metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2243)) - Direct SQL failed, falling
> back to ORM
> MetaException(message:Unexpected null for one of the IDs, SD 6437, column
> null, serde 6437 for a non- view)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:360)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:224)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$1.getSqlResult(ObjectStore.java:1563)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$1.getSqlResult(ObjectStore.java:1559)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1570)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1553)
> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy5.getPartitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2526)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions.getResult(ThriftHiveMetastore.java:8747)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions.getResult(ThriftHiveMetastore.java:8731)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:617)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:613)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1591)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:613)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This direct SQL call fails for every {{getPartitions}} call and then falls
> back to ORM.
> The query which fails is
> {code}
> select
> PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID,
> SERDES.SERDE_ID, PARTITIONS.CREATE_TIME,
> PARTITIONS.LAST_ACCESS_TIME,
[jira] [Commented] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160171#comment-15160171
]
Gopal V commented on HIVE-13134:
[~vgumashta]: I have attached a patch which does what I need for now - the
change of location to jdbc/ folder is arbitrary at this point.
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-13134.1.patch
>
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> when hotfixes are deployed.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13134:
---
Attachment: HIVE-13134.1.patch
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-13134.1.patch
>
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> when hotfixes are deployed.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs
[ https://issues.apache.org/jira/browse/HIVE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160165#comment-15160165 ] Sergey Shelukhin commented on HIVE-13135: - +1. Btw, after HIVE-12942, I don't think LLAP_WEB_AUTO_AUTH serves any purpose. It used to override bunch of conf settings that the user could have set manually to custom values instead of relying on the values that would work by default; now it just calls some methods that should probably always be called, if it's off, there's no alternative way for the user to set this up as far as I see. Can be removed on commit or in a separate JIRA. > LLAP: HTTPS Webservices needs trusted keystore configs > -- > > Key: HIVE-13135 > URL: https://issues.apache.org/jira/browse/HIVE-13135 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13135.1.patch > > > ssl-server.xml is not picked up internally by the hive-common HttpServer > impl, unlike the default configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13133) Create initial InputFormat + record readers/writers
[ https://issues.apache.org/jira/browse/HIVE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13133: Fix Version/s: llap > Create initial InputFormat + record readers/writers > --- > > Key: HIVE-13133 > URL: https://issues.apache.org/jira/browse/HIVE-13133 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: llap > > Attachments: HIVE-13133.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13133) Create initial InputFormat + record readers/writers
[ https://issues.apache.org/jira/browse/HIVE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160160#comment-15160160 ] Sergey Shelukhin commented on HIVE-13133: - Please specify the fix version for these so there's no confusion... > Create initial InputFormat + record readers/writers > --- > > Key: HIVE-13133 > URL: https://issues.apache.org/jira/browse/HIVE-13133 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: llap > > Attachments: HIVE-13133.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs
[ https://issues.apache.org/jira/browse/HIVE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13135: --- Attachment: HIVE-13135.1.patch > LLAP: HTTPS Webservices needs trusted keystore configs > -- > > Key: HIVE-13135 > URL: https://issues.apache.org/jira/browse/HIVE-13135 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13135.1.patch > > > ssl-server.xml is not picked up internally by the hive-common HttpServer > impl, unlike the default configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs
[ https://issues.apache.org/jira/browse/HIVE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160135#comment-15160135 ] Gopal V commented on HIVE-13135: Probably was a bad idea to always enable SSL too - adding a config to disable/enable that (& using the magic word "SPNEGO" in the HiveConf) > LLAP: HTTPS Webservices needs trusted keystore configs > -- > > Key: HIVE-13135 > URL: https://issues.apache.org/jira/browse/HIVE-13135 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > > ssl-server.xml is not picked up internally by the hive-common HttpServer > impl, unlike the default configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13134:
---
Description:
JDBC standalone contains many shaded jars, which tends to diverge in version
without being aware of the issue on the cluster.
{code}
$ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
1047 Tue Feb 23 17:21:04 PST 2016
org/slf4j/helpers/SubstituteLoggerFactory.class
933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
{code}
Still need to retain those in the shaded version, but the jar has to out of the
./bin/hive classpath to load the service entries in order.
was:
JDBC standalone contains many shaded jars, which tends to diverge in version
without being aware of the issue on the cluster.
{code}
$ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
1047 Tue Feb 23 17:21:04 PST 2016
org/slf4j/helpers/SubstituteLoggerFactory.class
933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
{code}
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> without being aware of the issue on the cluster.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13134:
---
Description:
JDBC standalone contains many shaded jars, which tends to diverge in version
without being aware of the issue on the cluster.
{code}
$ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
1047 Tue Feb 23 17:21:04 PST 2016
org/slf4j/helpers/SubstituteLoggerFactory.class
933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
{code}
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> without being aware of the issue on the cluster.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13133) Create initial InputFormat + record readers/writers
[ https://issues.apache.org/jira/browse/HIVE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-13133. --- Resolution: Fixed Committed to llap branch. > Create initial InputFormat + record readers/writers > --- > > Key: HIVE-13133 > URL: https://issues.apache.org/jira/browse/HIVE-13133 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13133.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13133) Create initial InputFormat + record readers/writers
[ https://issues.apache.org/jira/browse/HIVE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-13133: -- Attachment: HIVE-13133.1.patch > Create initial InputFormat + record readers/writers > --- > > Key: HIVE-13133 > URL: https://issues.apache.org/jira/browse/HIVE-13133 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13133.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13133) Create initial InputFormat + record readers/writers
[ https://issues.apache.org/jira/browse/HIVE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner reassigned HIVE-13133: - Assignee: Gunther Hagleitner > Create initial InputFormat + record readers/writers > --- > > Key: HIVE-13133 > URL: https://issues.apache.org/jira/browse/HIVE-13133 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160046#comment-15160046
]
Hive QA commented on HIVE-13102:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789243/HIVE-13102.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9820 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7072/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7072/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7072/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789243 - PreCommit-HIVE-TRUNK-Build
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160041#comment-15160041 ] Sergey Shelukhin commented on HIVE-13131: - +1 pending tests > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13131.1.patch > > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13132:
---
Attachment: HIVE-13132.1.patch
Uploaded patch.
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13131: Assignee: Jason Dere (was: Sergey Shelukhin) > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13131.1.patch > > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13131: -- Attachment: HIVE-13131.1.patch > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-13131.1.patch > > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13131: -- Status: Patch Available (was: Open) > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-13131.1.patch > > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13131: -- Assignee: Sergey Shelukhin (was: Vikram Dixit K) > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Sergey Shelukhin > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13054) LLAP: disable permanent fns by default (for now)
[ https://issues.apache.org/jira/browse/HIVE-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13054: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. > LLAP: disable permanent fns by default (for now) > > > Key: HIVE-13054 > URL: https://issues.apache.org/jira/browse/HIVE-13054 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Trivial > Fix For: 2.1.0 > > Attachments: HIVE-13054.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13126: - Attachment: HIVE-13126.3.patch patch 3, moving reset logic from closeOp to initializeOp as Gopal suggested. > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch, HIVE-13126.2.patch, > HIVE-13126.3.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[
https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159948#comment-15159948
]
Ashutosh Chauhan commented on HIVE-13096:
-
Instead of recursively adding cardinality of tree for each operator, I think
following heuristic might be better:
{code}
getCummCardinality (Operator op) {
if (op.type = join) {
cummCardinality += maxCardinality from all branches;
} else {
return cummCardinality;
}
}
{code}
That is to say, cardinality from any operator other than join does not
contribute in cumulative cardinality. And for join, max cardinality from its
input contribute in cummulative cardinality of tree. This is akin to what we
have on logical side, where getCumulativeCost() is overriden only for join and
is overriden in manner suggested here.
> Cost to choose side table in MapJoin conversion based on cumulative
> cardinality
> ---
>
> Key: HIVE-13096
> URL: https://issues.apache.org/jira/browse/HIVE-13096
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13096.01.patch, HIVE-13096.patch
>
>
> HIVE-11954 changed the logic to choose the side table in the MapJoin
> conversion algorithm. Initial heuristic for the cost was based on number of
> heavyweight operators.
> This extends that work so the heuristic is based on accumulate cardinality.
> In the future, we should choose the side based on total latency for the input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13120) propagate doAs when generating ORC splits
[ https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159921#comment-15159921 ] Sergey Shelukhin edited comment on HIVE-13120 at 2/24/16 12:12 AM: --- [~prasanth_j] can you take a look? https://reviews.apache.org/r/43921/ was (Author: sershe): [~prasanth_j] can you take a look? > propagate doAs when generating ORC splits > - > > Key: HIVE-13120 > URL: https://issues.apache.org/jira/browse/HIVE-13120 > Project: Hive > Issue Type: Improvement >Reporter: Yi Zhang >Assignee: Sergey Shelukhin > Attachments: HIVE-13120.patch > > > ORC+HS2+doAs+FetchTask conversion = weird permission errors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13120) propagate doAs when generating ORC splits
[ https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159921#comment-15159921 ] Sergey Shelukhin commented on HIVE-13120: - [~prasanth_j] can you take a look? > propagate doAs when generating ORC splits > - > > Key: HIVE-13120 > URL: https://issues.apache.org/jira/browse/HIVE-13120 > Project: Hive > Issue Type: Improvement >Reporter: Yi Zhang >Assignee: Sergey Shelukhin > Attachments: HIVE-13120.patch > > > ORC+HS2+doAs+FetchTask conversion = weird permission errors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[
https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159920#comment-15159920
]
Hive QA commented on HIVE-12963:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789230/HIVE-12963.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9803 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby1_limit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7071/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7071/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7071/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789230 - PreCommit-HIVE-TRUNK-Build
> LIMIT statement with SORT BY creates additional MR job with hardcoded only
> one reducer
> --
>
> Key: HIVE-12963
> URL: https://issues.apache.org/jira/browse/HIVE-12963
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.0.0, 1.2.1, 0.13
>Reporter: Alina Abramova
>Assignee: Alina Abramova
> Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch,
> HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch
>
>
> I execute query:
> hive> select age from test1 sort by age.age limit 10;
> Total jobs = 2
> Launching Job 1 out of 2
> Number of reduce tasks not specified. Estimated from input data size: 1
> Launching Job 2 out of 2
> Number of reduce tasks determined at compile time: 1
> When I have a large number of rows then the last stage of the job takes a
> long time. I think we could allow to user choose number of reducers of last
> job or refuse extra MR job.
> The same behavior I observed with querie:
> hive> create table new_test as select age from test1 group by age.age limit
> 10;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13120) propagate doAs when generating ORC splits
[ https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13120: Target Version/s: 2.1.0, 2.0.1 Status: Patch Available (was: Open) > propagate doAs when generating ORC splits > - > > Key: HIVE-13120 > URL: https://issues.apache.org/jira/browse/HIVE-13120 > Project: Hive > Issue Type: Improvement >Reporter: Yi Zhang >Assignee: Sergey Shelukhin > Attachments: HIVE-13120.patch > > > ORC+HS2+doAs+FetchTask conversion = weird permission errors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13120) propagate doAs when generating ORC splits
[ https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13120: Attachment: HIVE-13120.patch The fix. Writing a test for an ORC split generation patch takes 5 times longer than making the patch itself... > propagate doAs when generating ORC splits > - > > Key: HIVE-13120 > URL: https://issues.apache.org/jira/browse/HIVE-13120 > Project: Hive > Issue Type: Improvement >Reporter: Yi Zhang >Assignee: Sergey Shelukhin > Attachments: HIVE-13120.patch > > > ORC+HS2+doAs+FetchTask conversion = weird permission errors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13110) LLAP: Package log4j2 jars into Slider pkg
[ https://issues.apache.org/jira/browse/HIVE-13110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13110: --- Resolution: Fixed Release Note: LLAP: Package log4j2 jars into Slider pkg (Gopal V, reviewed by Gunther Hagleitner) Status: Resolved (was: Patch Available) > LLAP: Package log4j2 jars into Slider pkg > - > > Key: HIVE-13110 > URL: https://issues.apache.org/jira/browse/HIVE-13110 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13110.1.patch > > > This forms the alternative path for HIVE-13015 (reverted), so that HIVE-13027 > can pick up the right logger impl always. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13122) LLAP: simple Model/View separation for UI
[ https://issues.apache.org/jira/browse/HIVE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13122: --- Status: Patch Available (was: Open) > LLAP: simple Model/View separation for UI > - > > Key: HIVE-13122 > URL: https://issues.apache.org/jira/browse/HIVE-13122 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13122.1.patch > > > The current LLAP UI in master uses a fixed loop to both extract data and to > display it in the same loop. > Split this up into a model-view, for modularity. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13122) LLAP: simple Model/View separation for UI
[ https://issues.apache.org/jira/browse/HIVE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13122: --- Attachment: HIVE-13122.1.patch > LLAP: simple Model/View separation for UI > - > > Key: HIVE-13122 > URL: https://issues.apache.org/jira/browse/HIVE-13122 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13122.1.patch > > > The current LLAP UI in master uses a fixed loop to both extract data and to > display it in the same loop. > Split this up into a model-view, for modularity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13122) LLAP: simple Model/View separation for UI
[ https://issues.apache.org/jira/browse/HIVE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13122: --- Description: The current LLAP UI in master uses a fixed loop to both extract data and to display it in the same loop. Split this up into a model-view, for modularity. NO PRECOMMIT TESTS was: The current LLAP UI in master uses a fixed loop to both extract data and to display it in the same loop. Split this up into a model-view, for modularity. > LLAP: simple Model/View separation for UI > - > > Key: HIVE-13122 > URL: https://issues.apache.org/jira/browse/HIVE-13122 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13122.1.patch > > > The current LLAP UI in master uses a fixed loop to both extract data and to > display it in the same loop. > Split this up into a model-view, for modularity. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13101) NullPointerException in HiveLexer.g
[
https://issues.apache.org/jira/browse/HIVE-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandeep akinapelli updated HIVE-13101:
--
Status: Patch Available (was: Open)
> NullPointerException in HiveLexer.g
> ---
>
> Key: HIVE-13101
> URL: https://issues.apache.org/jira/browse/HIVE-13101
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: sandeep akinapelli
>Assignee: sandeep akinapelli
>Priority: Minor
> Attachments: HIVE-13101.1.patch
>
>
> In HiveLexer.g:
> {code:Java}
> protected boolean allowQuotedId() {
> String supportedQIds = HiveConf.getVar(hiveConf,
> HiveConf.ConfVars.HIVE_QUOTEDID_SUPPORT);
> return !"none".equals(supportedQIds);
> }
> {code}
> NullPointerException is thrown when hiveConf is not set.
> Stack trace:
> {code:Java}java.lang.NullPointerException
> at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3142)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> ~[antlr-runtime-3.5.1.jar:3.5.1]
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at org.antlr.runtime.Lexer.nextToken(Lexer.java:85)
> ~[antlr-runtime-3.5.1.jar:3.5.1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159813#comment-15159813 ] Gopal V commented on HIVE-13126: closeOp() is a dangerous place to put things in, since it can be called during a failure condition. This particular reset op should happen in initializeOp(). > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch, HIVE-13126.2.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: HIVE-13083.4.patch
Reuploading the patch to trigger precommit test run.
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch, HIVE-13083.4.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13128) NullScan fails on a secure setup
[
https://issues.apache.org/jira/browse/HIVE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159698#comment-15159698
]
Sergey Shelukhin commented on HIVE-13128:
-
+1
> NullScan fails on a secure setup
>
>
> Key: HIVE-13128
> URL: https://issues.apache.org/jira/browse/HIVE-13128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-13128.01.patch
>
>
> Nullscan provides uris of the form nullscan://null/ - which are added to the
> list of FileSystems for which Tez should obtain tokens.
> {code}
> 2016-02-19T02:48:04,481 ERROR [main]: exec.Task (TezTask.java:execute(219)) -
> Failed to execute tez graph.
> java.lang.IllegalArgumentException: java.net.UnknownHostException: null
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:291)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:302)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:524)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.addFileSystemCredentialsFromURIs(TezClientUtils.java:338)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.setupDAGCredentials(TezClientUtils.java:369)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:704)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:522)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAG(TezClient.java:468)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:466)
> ~[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:187)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1600)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1373)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1196)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:228)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:180)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:395)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:331)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:428)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:444)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:744)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at
[jira] [Updated] (HIVE-13128) NullScan fails on a secure setup
[
https://issues.apache.org/jira/browse/HIVE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-13128:
--
Status: Patch Available (was: Open)
> NullScan fails on a secure setup
>
>
> Key: HIVE-13128
> URL: https://issues.apache.org/jira/browse/HIVE-13128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-13128.01.patch
>
>
> Nullscan provides uris of the form nullscan://null/ - which are added to the
> list of FileSystems for which Tez should obtain tokens.
> {code}
> 2016-02-19T02:48:04,481 ERROR [main]: exec.Task (TezTask.java:execute(219)) -
> Failed to execute tez graph.
> java.lang.IllegalArgumentException: java.net.UnknownHostException: null
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:291)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:302)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:524)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.addFileSystemCredentialsFromURIs(TezClientUtils.java:338)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.setupDAGCredentials(TezClientUtils.java:369)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:704)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:522)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAG(TezClient.java:468)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:466)
> ~[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:187)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1600)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1373)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1196)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:228)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:180)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:395)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:331)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:428)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:444)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:744)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at
[jira] [Updated] (HIVE-13128) NullScan fails on a secure setup
[
https://issues.apache.org/jira/browse/HIVE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-13128:
--
Attachment: HIVE-13128.01.patch
Patch to add relevant addTokens,getToken APIs to NullFileSystem. There's some
additional changes around logging / misplaced log lines which I ran into while
debugging this.
[~sershe] - please review.
> NullScan fails on a secure setup
>
>
> Key: HIVE-13128
> URL: https://issues.apache.org/jira/browse/HIVE-13128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-13128.01.patch
>
>
> Nullscan provides uris of the form nullscan://null/ - which are added to the
> list of FileSystems for which Tez should obtain tokens.
> {code}
> 2016-02-19T02:48:04,481 ERROR [main]: exec.Task (TezTask.java:execute(219)) -
> Failed to execute tez graph.
> java.lang.IllegalArgumentException: java.net.UnknownHostException: null
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:291)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:302)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:524)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
> ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.addFileSystemCredentialsFromURIs(TezClientUtils.java:338)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.setupDAGCredentials(TezClientUtils.java:369)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at
> org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:704)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:522)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.tez.client.TezClient.submitDAG(TezClient.java:468)
> ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:466)
> ~[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:187)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1600)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1373)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1196)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> [hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:228)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:180)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:395)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:331)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:428)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:444)
> [hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:744)
>
[jira] [Commented] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159672#comment-15159672 ] Vikram Dixit K commented on HIVE-13126: --- +1. Nit: The comment should say something of the order that: Reset grace hashjoin context so that there is no state maintained when operator/work is retrieved from cache. > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159661#comment-15159661 ] Wei Zheng commented on HIVE-13126: -- [~vikram.dixit] Can you take a look? > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13126: - Attachment: (was: HIVE-13126.1.patch) > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[
https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159643#comment-15159643
]
Hive QA commented on HIVE-12244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789238/HIVE-12244.8.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 471 failed/errored test(s), 9669 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-auto_join18_multi_distinct.q-interval_udf.q-list_bucket_query_multiskew_2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-avro_decimal_native.q-alter_file_format.q-groupby3_map_skew.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cte_4.q-nullscript.q-filter_join_breaktask.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-gby_star.q-udf_regexp_replace.q-orc_merge10.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-groupby3_map.q-udf_least.q-orc_merge9.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-index_compact_2.q-vector_grouping_sets.q-lateral_view_cp.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-input17.q-vector_auto_smb_mapjoin_14.q-auto_join25.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-input44.q-drop_partitions_filter2.q-smb_mapjoin_4.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-orc_createas1.q-encryption_join_with_different_encryption_keys.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_show_grant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
[jira] [Updated] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states
[ https://issues.apache.org/jira/browse/HIVE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13126: - Attachment: HIVE-13126.1.patch > Clean up MapJoinOperator properly to avoid object cache reuse with > unintentional states > --- > > Key: HIVE-13126 > URL: https://issues.apache.org/jira/browse/HIVE-13126 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13126.1.patch > > > For a given job, one task may reuse other task's object cache (plan cache) > such as MapJoinOperator. This is fine. But if we have some dirty states left > over, it may cause issue like wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13127) Rebase LLAP branch with master
[ https://issues.apache.org/jira/browse/HIVE-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-13127. --- Resolution: Fixed Fix Version/s: llap llap branch has been reset back to master > Rebase LLAP branch with master > -- > > Key: HIVE-13127 > URL: https://issues.apache.org/jira/browse/HIVE-13127 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > > Update llap branch with the current state of master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13051) Deadline class has numerous issues
[ https://issues.apache.org/jira/browse/HIVE-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13051: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for review! > Deadline class has numerous issues > -- > > Key: HIVE-13051 > URL: https://issues.apache.org/jira/browse/HIVE-13051 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13051.01.patch, HIVE-13051.patch > > > currentTimeMillis is not a correct way to measure intervals of time; it can > easily be adjusted e.g. by ntpd. System.nanoTime should be used. > It's also unsafe for failure cases, and doesn't appear to update from config > updates correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13118) add some logging to LLAP token related paths
[ https://issues.apache.org/jira/browse/HIVE-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13118: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > add some logging to LLAP token related paths > > > Key: HIVE-13118 > URL: https://issues.apache.org/jira/browse/HIVE-13118 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13118.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159422#comment-15159422 ] Austin Lee commented on HIVE-12679: --- Could someone please review the attached patch? Thanks in advance for your time. > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11607) Export tables broken for data > 32 MB
[
https://issues.apache.org/jira/browse/HIVE-11607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159401#comment-15159401
]
Ravi Prakash commented on HIVE-11607:
-
Hi Sergio! For now we chose to :
{code}
export HIVE_CLASSPATH=$HADOOP_PREFIX/share/hadoop/tools/lib/*
{code}
This makes the assumption that the distcp jars are in that location. Also,
since we are including everything in the tools directory now, there may be
conflicts. I'll update the JIRA if we see problems rolling this out
> Export tables broken for data > 32 MB
> -
>
> Key: HIVE-11607
> URL: https://issues.apache.org/jira/browse/HIVE-11607
> Project: Hive
> Issue Type: Bug
> Components: Import/Export
>Affects Versions: 1.0.0, 1.2.0, 1.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11607.2.patch, HIVE-11607.3.patch, HIVE-11607.patch
>
>
> Broken for both hadoop-1 as well as hadoop-2 line
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Status: Patch Available (was: In Progress) The patch contains changes to move the IMetaStoreClient construction logic into a factory class and use a HiveConf to load this factory class at runtime. > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Fix Version/s: (was: 1.2.1) > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Affects Version/s: 2.1.0 > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Affects Versions: 2.1.0 >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Attachment: HIVE-12679.patch The patch contains changes to move the IMetaStoreClient construction logic into a factory class and use a HiveConf to load this factory class at runtime. > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Reporter: Austin Lee >Assignee: Austin Lee >Priority: Minor > Labels: metastore > Fix For: 1.2.1 > > Attachments: HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[
https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159329#comment-15159329
]
Hive QA commented on HIVE-12994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789258/HIVE-12994.09.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/121/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/121/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-121/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789258 - PreCommit-HIVE-METASTORE-Test
> Implement support for NULLS FIRST/NULLS LAST
>
>
> Key: HIVE-12994
> URL: https://issues.apache.org/jira/browse/HIVE-12994
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Parser, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch,
> HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch,
> HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch,
> HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.patch
>
>
> From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to
> determine whether nulls appear before or after non-null data values when the
> ORDER BY clause is used.
> SQL standard does not specify the behavior by default. Currently in Hive,
> null values sort as if lower than any non-null value; that is, NULLS FIRST is
> the default for ASC order, and NULLS LAST for DESC order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13094) CBO: Assertion error in Case expression
[
https://issues.apache.org/jira/browse/HIVE-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159317#comment-15159317
]
Hive QA commented on HIVE-13094:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789187/HIVE-13094.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9803 tests executed
*Failed tests:*
{noformat}
TestCliDriver-show_conf.q-groupby2_noskew_multi_distinct.q-ptf_rcfile.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7069/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7069/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7069/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789187 - PreCommit-HIVE-TRUNK-Build
> CBO: Assertion error in Case expression
>
>
> Key: HIVE-13094
> URL: https://issues.apache.org/jira/browse/HIVE-13094
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13094.01.patch, HIVE-13094.patch
>
>
> Triggered by a trap case in the case evaluation
> {code}
> CASE WHEN (-2) >= 0 THEN SUBSTRING(str0, 1,CAST((-2) AS INT)) ELSE NULL
> {code}
> {code}
> Exception in thread "b367ad08-d900-4672-8e75-a4e90a52141b
> b367ad08-d900-4672-8e75-a4e90a52141b main" java.lang.AssertionError: Internal
> error: Cannot add expression of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "ISO-8859-1" COLLATE
> "ISO-8859-1$en_US$primary" $f0, VARCHAR(2147483647) CHARACTER SET
> "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary" $f1, VARCL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "ISO-8859-1"
> COLLATE "ISO-8859-1$en_US$primary" $f0, VARCHAR(2147483647) CHARACTER SET
> "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary" $fL
> set is
> rel#12408:HiveProject.HIVE.[](input=HepRelVertex#12407,$f0=$0,$f1=$6,$f2=CASE(>=(-(2),
> 0), substring($6, 1, -(2)), null))
> expression is HiveProject#12414
> at org.apache.calcite.util.Util.newInternal(Util.java:774)
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:317)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:224)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(HiveReduceExpressionsRule.java:208)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:318)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:514)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:392)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:285)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:207)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:194)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1265)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:938)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:878)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:969)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Status: Open (was: Patch Available) > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-12994 started by Jesus Camacho Rodriguez. -- > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Attachment: HIVE-12994.09.patch > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13070: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) No need to waste QA on this one; pushed to master. Thanks for the help [~spena]! > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159282#comment-15159282 ] Alan Gates commented on HIVE-12064: --- +1 > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.5.patch, HIVE-12064.6.patch, > HIVE-12064.7.patch, HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159276#comment-15159276 ] Sergio Peña commented on HIVE-13070: Got it. The change looks good to me. +1 It is not necessary to restart Jenkins once this is committed. Jenkins always downloads a new copy of the jenkins-*.sh scripts everytime a new job is triggered. > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11607) Export tables broken for data > 32 MB
[
https://issues.apache.org/jira/browse/HIVE-11607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159267#comment-15159267
]
Sergio Peña commented on HIVE-11607:
[~raviprak] How is that issue fix? Removing {{provided}] would work?
> Export tables broken for data > 32 MB
> -
>
> Key: HIVE-11607
> URL: https://issues.apache.org/jira/browse/HIVE-11607
> Project: Hive
> Issue Type: Bug
> Components: Import/Export
>Affects Versions: 1.0.0, 1.2.0, 1.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11607.2.patch, HIVE-11607.3.patch, HIVE-11607.patch
>
>
> Broken for both hadoop-1 as well as hadoop-2 line
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[
https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159261#comment-15159261
]
Jesus Camacho Rodriguez commented on HIVE-13070:
[~spena], thanks for checking.
The problem with the change you proposed is that process\_jira is called in
_jenkins-submit-build.sh_ script too, before the call to the tests execution is
done: at that point, {{BUILD_TAG}} is not set, and hence {{BUILD_POSTFIX}}
cannot be either. Thus, we cannot add {{test -n}}.
Nevertheless, I have uploaded a new patch to derive {{build_postfix}} within
process\_jira iff {{BUILD_TAG}} is set.
> Precommit HMS tests should run in addition to precommit normal tests, not
> instead of
>
>
> Key: HIVE-13070
> URL: https://issues.apache.org/jira/browse/HIVE-13070
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13070.01.patch, HIVE-13070.patch
>
>
> When a certain patch makes changes in the metastore upgrade scripts folder,
> precommit HMS tests are triggered. The problem is that precommit HMS marks
> the patch as tested, thus normal precommit tests are never triggered.
> I hit the issue while testing HIVE-12994.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13070: --- Attachment: HIVE-13070.01.patch > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13070: --- Status: Open (was: Patch Available) > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13070 started by Jesus Camacho Rodriguez. -- > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13070: --- Status: Patch Available (was: In Progress) > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.01.patch, HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13114) parquet filter fails for type float when hive.optimize.index.filter=true
[
https://issues.apache.org/jira/browse/HIVE-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña resolved HIVE-13114.
Resolution: Duplicate
Fix Version/s: 2.0.0
> parquet filter fails for type float when hive.optimize.index.filter=true
>
>
> Key: HIVE-13114
> URL: https://issues.apache.org/jira/browse/HIVE-13114
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Gabriel C Balan
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Hive fails when selecting from a table 'stored as parquet' with a row filter
> based on a float column and hive.optimize.index.filter=true.
> The following example fails in hive 1.2.1 (HDP 2.3.2), but works fine in hive
> 1.1.0 (CDH 5.5.0):
> {code}
> create table p(f float)stored as parquet;
> insert into table p values (1), (2), (3);
> select * from p where f >= 2;
> set hive.optimize.index.filter=true;
> select * from p where f >= 2;
> {code}
> The first select query works fine, the second fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException:
> FilterPredicate column: f's declared type (java.lang.Double) does not match
> the schema found in file metadata. Column f is of type:
> FullTypeDescriptor(PrimitiveType: FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> {code}
> Here's the stack trace from log4j:
> {code}
> 2016-02-22 12:18:30,691 ERROR [main]: CliDriver
> (SessionState.java:printError(960)) - Failed with exception
> java.io.IOException:java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> java.io.IOException: java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: FilterPredicate column: f's
> declared type (java.lang.Double) does not match the schema found in file
> metadata. Column f is of type: FullTypeDescriptor(PrimitiveType: FLOAT,
> OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:132)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:185)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:124)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)
> at parquet.filter2.predicate.Operators$GtEq.accept(Operators.java:248)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)
> at
> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)
> at
>
[jira] [Commented] (HIVE-13114) parquet filter fails for type float when hive.optimize.index.filter=true
[
https://issues.apache.org/jira/browse/HIVE-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159245#comment-15159245
]
Sergio Peña commented on HIVE-13114:
This patch was fixed for Hive 2.0
https://issues.apache.org/jira/browse/HIVE-11504
> parquet filter fails for type float when hive.optimize.index.filter=true
>
>
> Key: HIVE-13114
> URL: https://issues.apache.org/jira/browse/HIVE-13114
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Gabriel C Balan
>Priority: Trivial
>
> Hive fails when selecting from a table 'stored as parquet' with a row filter
> based on a float column and hive.optimize.index.filter=true.
> The following example fails in hive 1.2.1 (HDP 2.3.2), but works fine in hive
> 1.1.0 (CDH 5.5.0):
> {code}
> create table p(f float)stored as parquet;
> insert into table p values (1), (2), (3);
> select * from p where f >= 2;
> set hive.optimize.index.filter=true;
> select * from p where f >= 2;
> {code}
> The first select query works fine, the second fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException:
> FilterPredicate column: f's declared type (java.lang.Double) does not match
> the schema found in file metadata. Column f is of type:
> FullTypeDescriptor(PrimitiveType: FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> {code}
> Here's the stack trace from log4j:
> {code}
> 2016-02-22 12:18:30,691 ERROR [main]: CliDriver
> (SessionState.java:printError(960)) - Failed with exception
> java.io.IOException:java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> java.io.IOException: java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: FilterPredicate column: f's
> declared type (java.lang.Double) does not match the schema found in file
> metadata. Column f is of type: FullTypeDescriptor(PrimitiveType: FLOAT,
> OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:132)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:185)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:124)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)
> at parquet.filter2.predicate.Operators$GtEq.accept(Operators.java:248)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)
> at
>
[jira] [Comment Edited] (HIVE-13114) parquet filter fails for type float when hive.optimize.index.filter=true
[
https://issues.apache.org/jira/browse/HIVE-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159245#comment-15159245
]
Sergio Peña edited comment on HIVE-13114 at 2/23/16 5:30 PM:
-
This patch was fixed for Hive 2.0
https://issues.apache.org/jira/browse/HIVE-11504
I run a quick test on the master branch and it is working.
was (Author: spena):
This patch was fixed for Hive 2.0
https://issues.apache.org/jira/browse/HIVE-11504
> parquet filter fails for type float when hive.optimize.index.filter=true
>
>
> Key: HIVE-13114
> URL: https://issues.apache.org/jira/browse/HIVE-13114
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Gabriel C Balan
>Priority: Trivial
>
> Hive fails when selecting from a table 'stored as parquet' with a row filter
> based on a float column and hive.optimize.index.filter=true.
> The following example fails in hive 1.2.1 (HDP 2.3.2), but works fine in hive
> 1.1.0 (CDH 5.5.0):
> {code}
> create table p(f float)stored as parquet;
> insert into table p values (1), (2), (3);
> select * from p where f >= 2;
> set hive.optimize.index.filter=true;
> select * from p where f >= 2;
> {code}
> The first select query works fine, the second fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException:
> FilterPredicate column: f's declared type (java.lang.Double) does not match
> the schema found in file metadata. Column f is of type:
> FullTypeDescriptor(PrimitiveType: FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> {code}
> Here's the stack trace from log4j:
> {code}
> 2016-02-22 12:18:30,691 ERROR [main]: CliDriver
> (SessionState.java:printError(960)) - Failed with exception
> java.io.IOException:java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> java.io.IOException: java.lang.IllegalArgumentException: FilterPredicate
> column: f's declared type (java.lang.Double) does not match the schema found
> in file metadata. Column f is of type: FullTypeDescriptor(PrimitiveType:
> FLOAT, OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: FilterPredicate column: f's
> declared type (java.lang.Double) does not match the schema found in file
> metadata. Column f is of type: FullTypeDescriptor(PrimitiveType: FLOAT,
> OriginalType: null)
> Valid types for this column are: [class java.lang.Float]
> at
> parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:132)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:185)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:124)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)
> at parquet.filter2.predicate.Operators$GtEq.accept(Operators.java:248)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)
> at
[jira] [Commented] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159236#comment-15159236
]
Jesus Camacho Rodriguez commented on HIVE-13102:
Done, thanks
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159228#comment-15159228
]
Ashutosh Chauhan commented on HIVE-13102:
-
[~jcamachorodriguez] Can you create a RB for this?
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159223#comment-15159223 ] Ashutosh Chauhan commented on HIVE-13040: - This patch addresses two distinct issues: * Don't create empty buckets for Tez. We know for sure Tez can handle missing bucket files while doing BMJ & SMBJ. However, MR does explicitly checks for number of files before attempting BMJ & SMBJ, so if we don't create empty files for MR, we risk running into disabling BMJ & SMBJ later on for MR. * Above means, we do end up creating logically empty files for MR (which is majority of test cases). For such cases, ORC currently writes header & footer. This patch includes a change to not write anything at all (ie create 0-length file) in such cases. While reading there are two ways to handle such 0-length files, either make ORC reader resilient to it or exclude such files altogether while doing split generation. I choose second approach as thats more efficient since we avoid wasteful processing for that. So, there are changes related to that as well. > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13102:
---
Status: Patch Available (was: In Progress)
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13102:
---
Attachment: HIVE-13102.01.patch
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13102:
---
Status: Open (was: Patch Available)
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13102) CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
[
https://issues.apache.org/jira/browse/HIVE-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13102 started by Jesus Camacho Rodriguez.
--
> CBO: Reduce operations in Calcite do not fold as tight as rule-based folding
>
>
> Key: HIVE-13102
> URL: https://issues.apache.org/jira/browse/HIVE-13102
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13102.01.patch, HIVE-13102.patch
>
>
> With CBO
> {code}
> create temporary table table1(id int, val int, val1 int, dimid int);
> create temporary table table3(id int, val int, val1 int);
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> Warning: Map Join MAPJOIN[14][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_11]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] llap
> BROADCAST [RS_8]
> Select Operator [SEL_5] (rows=1 width=0)
> Filter Operator [FIL_13] (rows=1 width=0)
> predicate:(id = 1)
> TableScan [TS_3] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:NONE,Output:["id"]
> <-Select Operator [SEL_2] (rows=1 width=0)
> Output:["_col0","_col1","_col2"]
> Filter Operator [FIL_12] (rows=1 width=0)
> predicate:((dimid = 1) and (dimid <> 1))
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1","dimid"]
> {code}
> without CBO
> {code}
> hive> explain select table1.id, table1.val, table1.val1 from table1 inner
> join table3 on table1.dimid = table3.id and table3.id = 1 where table1.dimid
> <>1 ;
> OK
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1 llap
> File Output Operator [FS_9]
> Map Join Operator [MAPJOIN_14] (rows=1 width=0)
> Conds:FIL_12.1=RS_17.1(Inner),Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_17]
> PartitionCols:1
> Filter Operator [FIL_16] (rows=1 width=0)
> predicate:false
> TableScan [TS_1] (rows=1 width=0)
> default@table3,table3,Tbl:PARTIAL,Col:COMPLETE
> <-Filter Operator [FIL_12] (rows=1 width=0)
> predicate:false
> TableScan [TS_0] (rows=1 width=0)
>
> default@table1,table1,Tbl:PARTIAL,Col:NONE,Output:["id","val","val1"]
> Time taken: 0.044 seconds, Fetched: 23 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13118) add some logging to LLAP token related paths
[ https://issues.apache.org/jira/browse/HIVE-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159189#comment-15159189 ] Siddharth Seth commented on HIVE-13118: --- +1. Looks good. Only think to look at is whether there's too much information logged as part of the token. i.e. can the token be reconstructed from the log message. If that's the case - should shorten the log string. > add some logging to LLAP token related paths > > > Key: HIVE-13118 > URL: https://issues.apache.org/jira/browse/HIVE-13118 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13118.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[
https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159178#comment-15159178
]
Hive QA commented on HIVE-12244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789238/HIVE-12244.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/120/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/120/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-120/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789238 - PreCommit-HIVE-METASTORE-Test
> Refactoring code for avoiding of comparison of Strings and do comparison on
> Path
>
>
> Key: HIVE-12244
> URL: https://issues.apache.org/jira/browse/HIVE-12244
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1
>Reporter: Alina Abramova
>Assignee: Alina Abramova
>Priority: Minor
> Labels: patch
> Fix For: 1.2.1
>
> Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch,
> HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch,
> HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, HIVE-12244.8.patch
>
>
> In Hive often String is used for representation path and it causes new issues.
> We need to compare it with equals() but comparing Strings often is not right
> in terms comparing paths .
> I think if we use Path from org.apache.hadoop.fs we will avoid new problems
> in future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.8.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, HIVE-12244.8.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
