[jira] [Assigned] (IMPALA-3531) Implement FK/PK "rely novalidate" constraints for better CBO
[ https://issues.apache.org/jira/browse/IMPALA-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada reassigned IMPALA-3531: Assignee: Anurag Mantripragada > Implement FK/PK "rely novalidate" constraints for better CBO > > > Key: IMPALA-3531 > URL: https://issues.apache.org/jira/browse/IMPALA-3531 > Project: IMPALA > Issue Type: New Feature > Components: Catalog, Frontend, Perf Investigation >Affects Versions: Impala 2.5.0, Impala 2.6.0 > Environment: CDH >Reporter: Ruslan Dautkhanov >Assignee: Anurag Mantripragada >Priority: Minor > Labels: CBO, performance, ramp-up > > Oracle has "RELY NOVALIDATE" option for constraints.. Could be easier for > Hive to start with something like that for PK/FK constraints. So CBO has more > information for optimizations. It does not have to actually check if that > constraint is relationship is true; it can just "rely" on that constraint. > https://docs.oracle.com/database/121/SQLRF/clauses002.htm#sthref2289 > So it would be helpful with join cardinality estimates, and with cases like > IMPALA-2929. > https://docs.oracle.com/database/121/DWHSG/schemas.htm#DWHSG9053 > "Overview of Constraint States": > - Enforcement > - Validation > - Belief > So FK/PK with "rely novalidate" will have Enforcement disabled but > Belief = RELY as it is possible to do in Oracle and now in Hive (HIVE-13076). > It opens a lot of ways to do additional ways to optimize execution plans. > As exxplined in Tom Kyte's "Metadata matters" > http://www.peoug.org/wp-content/uploads/2009/12/MetadataMatters_PEOUG_Day2009_TKyte.pdf > pp.30 - "Tell us how the tables relate and we can remove them from the > plan...". > pp.35 - "Tell us how the tables relate and we have more access paths > available...". > Also it might be helpful when Impala is being integrated with Kudu as the > latter have to have a PK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7484) Unrecognized hint interpreted as straight_join
[ https://issues.apache.org/jira/browse/IMPALA-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada reassigned IMPALA-7484: Assignee: Anurag Mantripragada > Unrecognized hint interpreted as straight_join > -- > > Key: IMPALA-7484 > URL: https://issues.apache.org/jira/browse/IMPALA-7484 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Lars Volker >Assignee: Anurag Mantripragada >Priority: Major > > The call to setIsStraightJoin() in > [SelectList.java:87|https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87] > should be wrapped in an else-clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-7921) Hive JVM aborts during data load
[
https://issues.apache.org/jira/browse/IMPALA-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-7921.
Resolution: Fixed
Could not reproduce. Closing this issue.
> Hive JVM aborts during data load
>
>
> Key: IMPALA-7921
> URL: https://issues.apache.org/jira/browse/IMPALA-7921
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0
>Reporter: Lars Volker
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: broken-build, flaky
>
> During a core test run I observed Hive's JVM crashing. Here is the stack
> trace from the core file it left behind:
> {noformat}
> CORE: ./core.1543762524.3579.java
> BINARY: /usr/java/jdk1.8.0_144/bin/java
> Core was generated by `/usr/java/jdk1.8.0_144/bin/java -Dproc_jar -Xmx2048m
> -Dhive.log.file=hive-serve'.
> Program terminated with signal SIGABRT, Aborted.
> ...
> #0 0x7f5b72fc81f7 in raise () from /lib64/libc.so.6
> #1 0x7f5b72fc98e8 in abort () from /lib64/libc.so.6
> #2 0x7f5b728c5185 in os::abort(bool) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #3 0x7f5b72a67593 in VMError::report_and_die() () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #4 0x7f5b728ca68f in JVM_handle_linux_signal () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 0x7f5b728c0be3 in signalHandler(int, siginfo*, void*) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x080721b0 in ?? ()
> #8 0x in ?? ()
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8291) Describe extended does not display constraint information
Anurag Mantripragada created IMPALA-8291: Summary: Describe extended does not display constraint information Key: IMPALA-8291 URL: https://issues.apache.org/jira/browse/IMPALA-8291 Project: IMPALA Issue Type: Sub-task Components: Frontend Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada Currently, DESCRIBE EXTENDED table_name command does not display constraint information like primary key / Foreign key information for tables created through Hive. This work must also be extended to tables created through Impala once we have support for pk/fk in create table syntax. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8291) 'DESCRIBE EXTENDED ..' does not display constraint information
[ https://issues.apache.org/jira/browse/IMPALA-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-8291: - Summary: 'DESCRIBE EXTENDED ..' does not display constraint information (was: Describe extended does not display constraint information) > 'DESCRIBE EXTENDED ..' does not display constraint information > -- > > Key: IMPALA-8291 > URL: https://issues.apache.org/jira/browse/IMPALA-8291 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > > Currently, DESCRIBE EXTENDED table_name command does not display constraint > information like primary key / Foreign key information for tables created > through Hive. > This work must also be extended to tables created through Impala once we have > support for pk/fk in create table syntax. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8290) Display constraint information in 'SHOW CREATE' statement
Anurag Mantripragada created IMPALA-8290: Summary: Display constraint information in 'SHOW CREATE' statement Key: IMPALA-8290 URL: https://issues.apache.org/jira/browse/IMPALA-8290 Project: IMPALA Issue Type: Sub-task Components: Frontend Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada Show create statement should display primary key and foreign key information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-2112:
-
Description:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
To be compatible with Hive:
* We neither enforce or validate integrity constraints. Hence, DISABLE and
NOVALIDATE options are mandatory.
* RELY/NORELY is optional. The CBO is expected to use this information when a
user specifies “RELY”. The default is NORELY.
* Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
Key column in parent table.
Support create table syntax like hive does:
* {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
DISABLE NOVALIDATE);}}
* {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
* {{create table T1(id integer, name string, primary key(id) DISABLE
NOVALIDATE RELY}}
was:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
To be compatible with Hive:
* We neither enforce or validate integrity constraints. Hence, DISABLE and
NOVALIDATE options are mandatory.
* RELY/NORELY is optional. The CBO is expected to use this information when a
user specifies “RELY”. The default is NORELY.
* Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
Key column in parent table.
Support create table syntax like hive does:
* {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
DIASBLE NOVALIDATE);}}
* {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
* {{create table T1(id integer, name string, primary key(id) DISABLE
NOVALIDATE RELY}}
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kornacker
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
> DISABLE NOVALIDATE);}}
> * {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
> key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8163) Local catalog mode needs to be visible on catalogd web UI if turned on
[ https://issues.apache.org/jira/browse/IMPALA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-8163. -- Resolution: Fixed Fix Version/s: Impala 3.2.0 Change committed to master. > Local catalog mode needs to be visible on catalogd web UI if turned on > --- > > Key: IMPALA-8163 > URL: https://issues.apache.org/jira/browse/IMPALA-8163 > Project: IMPALA > Issue Type: Bug >Reporter: Adrian Ng >Assignee: Anurag Mantripragada >Priority: Major > Fix For: Impala 3.2.0 > > > Once local catalog mode (i.e. metadata v2) is turned on (off by default), it > needs to be visible on catalogd web UI because it is an experimental protocol > so admins need to be aware of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7935) /catalog_object end point broken in LocalCatalog mode
[
https://issues.apache.org/jira/browse/IMPALA-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7935 started by Anurag Mantripragada.
> /catalog_object end point broken in LocalCatalog mode
> -
>
> Key: IMPALA-7935
> URL: https://issues.apache.org/jira/browse/IMPALA-7935
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: observability
>
> Start Impala coordinator in LocalCatalog mode (v2)
> {code}
> start-impala-cluster.py -s 1 --impalad_args="--use_local_catalog=true"
> --catalogd_args="--catalog_topic_mode=minimal"
> {code}
> Check the following URL.
> http://:25000/catalog_object?object_type=TABLE_name=functional_text_lzo.alltypesaggmultifiles
> It says,
> {noformat}
> Error: UnsupportedOperationException: LocalCatalog.getTCatalogObject
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7910) COMPUTE STATS does an unnecessary REFRESH after writing to the Metastore
[
https://issues.apache.org/jira/browse/IMPALA-7910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-7910:
Assignee: Anurag Mantripragada (was: Tianyi Wang)
> COMPUTE STATS does an unnecessary REFRESH after writing to the Metastore
>
>
> Key: IMPALA-7910
> URL: https://issues.apache.org/jira/browse/IMPALA-7910
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.9.0, Impala 2.11.0, Impala 2.12.0
>Reporter: Michael Brown
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> COMPUTE STATS and possibly other DDL operations unnecessarily do the
> equivalent of a REFRESH after writing to the Hive Metastore. This unnecessary
> operation can be very expensive, so should be avoided.
> The behavior can be confirmed from the catalogd logs:
> {code}
> compute stats functional_parquet.alltypes;
> +---+
> | summary |
> +---+
> | Updated 24 partition(s) and 11 column(s). |
> +---+
> Relevant catalogd.INFO snippet
> I0413 14:40:24.210749 27295 HdfsTable.java:1263] Incrementally loading table
> metadata for: functional_parquet.alltypes
> I0413 14:40:24.242122 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.244634 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.247174 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.249713 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.252288 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.254629 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.256991 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.259464 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.262197 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.264463 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.266736 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.269210 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.271800 27295 HdfsTable.java:555] Refreshed file metadata for
> functional_parquet.alltypes Path:
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1:
> Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0
> I0413 14:40:24.274348 27295 HdfsTable.java:555]
[jira] [Assigned] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-7971:
Assignee: Anurag Mantripragada
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8208) Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
Anurag Mantripragada created IMPALA-8208:
Summary: Support ALTER commands for Rely/Norely novalidate for
primary key/foreign key
Key: IMPALA-8208
URL: https://issues.apache.org/jira/browse/IMPALA-8208
Project: IMPALA
Issue Type: Sub-task
Reporter: Anurag Mantripragada
Assignee: Anurag Mantripragada
Support for ALTER commands such as:
* {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
* {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
DISABLE NOVALIDATE RELY;}}
* {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
DISABLE NOVALIDATE;}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8208) Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
[
https://issues.apache.org/jira/browse/IMPALA-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8208:
-
Description:
Support for ALTER commands such as:
* {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
* {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
DISABLE NOVALIDATE RELY;}}
* {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
DISABLE NOVALIDATE;}}
* {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
* {{ALTER TABLE product DROP CONSTRAINT fk1;}}
was:
Support for ALTER commands such as:
* {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
* {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
DISABLE NOVALIDATE RELY;}}
* {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
DISABLE NOVALIDATE;}}
* {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
* {{ALTER TABLE product DROP CONSTRAINT fk1;}}{{
}}
**
> Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
> -
>
> Key: IMPALA-8208
> URL: https://issues.apache.org/jira/browse/IMPALA-8208
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
>
> Support for ALTER commands such as:
> * {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
> * {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
> DISABLE NOVALIDATE RELY;}}
> * {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
> DISABLE NOVALIDATE;}}
> * {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
> * {{ALTER TABLE product DROP CONSTRAINT fk1;}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8208) Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
[
https://issues.apache.org/jira/browse/IMPALA-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8208:
-
Description:
Support for ALTER commands such as:
* {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
* {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
DISABLE NOVALIDATE RELY;}}
* {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
DISABLE NOVALIDATE;}}
* {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
* {{ALTER TABLE product DROP CONSTRAINT fk1;}}{{
}}
**
was:
Support for ALTER commands such as:
* {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
* {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
DISABLE NOVALIDATE RELY;}}
* {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
DISABLE NOVALIDATE;}}
> Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
> -
>
> Key: IMPALA-8208
> URL: https://issues.apache.org/jira/browse/IMPALA-8208
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
>
> Support for ALTER commands such as:
> * {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
> * {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
> DISABLE NOVALIDATE RELY;}}
> * {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
> DISABLE NOVALIDATE;}}
> * {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
> * {{ALTER TABLE product DROP CONSTRAINT fk1;}}{{
> }}
> **
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Add PRIMARY/FOREIGN KEY keywords to CREATE TABLE
[ https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-2112: - Issue Type: Sub-task (was: New Feature) External issue ID: (was: IMPALA-2112) Parent: IMPALA-3531 > Add PRIMARY/FOREIGN KEY keywords to CREATE TABLE > > > Key: IMPALA-2112 > URL: https://issues.apache.org/jira/browse/IMPALA-2112 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Affects Versions: Impala 2.2 >Reporter: Marcel Kornacker >Priority: Minor > Labels: planner > > These would be advisory, ie, Impala would not attempt to enforce them. > However, they could be used for cardinality estimation during query planning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[ https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-2112: - Summary: Support primary key/foreign key constraint as part of create table in Impala (was: Add PRIMARY/FOREIGN KEY keywords to CREATE TABLE) > Support primary key/foreign key constraint as part of create table in Impala > > > Key: IMPALA-2112 > URL: https://issues.apache.org/jira/browse/IMPALA-2112 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Affects Versions: Impala 2.2 >Reporter: Marcel Kornacker >Priority: Minor > Labels: planner > > These would be advisory, ie, Impala would not attempt to enforce them. > However, they could be used for cardinality estimation during query planning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[ https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-2112: - Component/s: Catalog > Support primary key/foreign key constraint as part of create table in Impala > > > Key: IMPALA-2112 > URL: https://issues.apache.org/jira/browse/IMPALA-2112 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Affects Versions: Impala 2.2 >Reporter: Marcel Kornacker >Priority: Minor > Labels: planner > > These would be advisory, ie, Impala would not attempt to enforce them. > However, they could be used for cardinality estimation during query planning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[ https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-2112: - Description: These would be advisory, ie, Impala would not attempt to enforce them. However, they could be used for cardinality estimation during query planning. Support was:These would be advisory, ie, Impala would not attempt to enforce them. However, they could be used for cardinality estimation during query planning. > Support primary key/foreign key constraint as part of create table in Impala > > > Key: IMPALA-2112 > URL: https://issues.apache.org/jira/browse/IMPALA-2112 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Affects Versions: Impala 2.2 >Reporter: Marcel Kornacker >Priority: Minor > Labels: planner > > These would be advisory, ie, Impala would not attempt to enforce them. > However, they could be used for cardinality estimation during query planning. > > Support -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-2112:
-
Description:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
To be compatible with Hive:
* We neither enforce or validate integrity constraints. Hence, DISABLE and
NOVALIDATE options are mandatory.
* RELY/NORELY is optional. The CBO is expected to use this information when a
user specifies “RELY”. The default is NORELY.
* Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
Key column in parent table.
Support create table syntax like hive does:
* create table pk(id1 integer, id2 integer, }}{{primary key(id1,
id2) DIASBLE NOVALIDATE);{{ }}
* create table fk(id1 integer, id2 integer, }}{{constraint c1
foreign key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);
* {{create table T1(id integer, name string, primary key(id) DISABLE
NOVALIDATE RELY}}
was:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
Support
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kornacker
>Priority: Minor
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * create table pk(id1 integer, id2 integer, }}{{primary
> key(id1, id2) DIASBLE NOVALIDATE);{{ }}
> * create table fk(id1 integer, id2 integer, }}{{constraint c1
> foreign key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-2112:
-
Description:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
To be compatible with Hive:
* We neither enforce or validate integrity constraints. Hence, DISABLE and
NOVALIDATE options are mandatory.
* RELY/NORELY is optional. The CBO is expected to use this information when a
user specifies “RELY”. The default is NORELY.
* Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
Key column in parent table.
Support create table syntax like hive does:
* {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
DIASBLE NOVALIDATE);}}
* {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
* {{create table T1(id integer, name string, primary key(id) DISABLE
NOVALIDATE RELY}}
was:
These would be advisory, ie, Impala would not attempt to enforce them. However,
they could be used for cardinality estimation during query planning.
To be compatible with Hive:
* We neither enforce or validate integrity constraints. Hence, DISABLE and
NOVALIDATE options are mandatory.
* RELY/NORELY is optional. The CBO is expected to use this information when a
user specifies “RELY”. The default is NORELY.
* Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
Key column in parent table.
Support create table syntax like hive does:
* create table pk(id1 integer, id2 integer, }}{{primary key(id1,
id2) DIASBLE NOVALIDATE);{{ }}
* create table fk(id1 integer, id2 integer, }}{{constraint c1
foreign key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);
* {{create table T1(id integer, name string, primary key(id) DISABLE
NOVALIDATE RELY}}
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kornacker
>Priority: Minor
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
> DIASBLE NOVALIDATE);}}
> * {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
> key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8208) Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
[
https://issues.apache.org/jira/browse/IMPALA-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8208:
-
Component/s: Frontend
Catalog
> Support ALTER commands for Rely/Norely novalidate for primary key/foreign key
> -
>
> Key: IMPALA-8208
> URL: https://issues.apache.org/jira/browse/IMPALA-8208
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
>
> Support for ALTER commands such as:
> * {{ALTER TABLE T1 ADD CONSTRAINT pk2 primary key (a) disable novalidate;}}
> * {{ALTER TABLE T2 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES T1(a)
> DISABLE NOVALIDATE RELY;}}
> * {{ALTER TABLE T3 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES T1(a)
> DISABLE NOVALIDATE;}}
> * {{ALTER TABLE sales DROP CONSTRAINT pk1;}}
> * {{ALTER TABLE product DROP CONSTRAINT fk1;}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-2112:
Assignee: Anurag Mantripragada
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kornacker
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
> DIASBLE NOVALIDATE);}}
> * {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
> key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7973) Add support for fine-grained updates at partition level
[ https://issues.apache.org/jira/browse/IMPALA-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7973 started by Anurag Mantripragada. > Add support for fine-grained updates at partition level > --- > > Key: IMPALA-7973 > URL: https://issues.apache.org/jira/browse/IMPALA-7973 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > > When data is inserted into a partition or a new partition is created in a > large table, we should not be invalidating the whole table. Instead it should > be possible to refresh/add/drop certain partitions on the table directly > based on the event information. This would help with the performance of > subsequent access to the table by avoiding reloading the large table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7484) Unrecognized hint interpreted as straight_join
[ https://issues.apache.org/jira/browse/IMPALA-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-7484. -- Resolution: Fixed The patch was merged a while back. Hence closing it. > Unrecognized hint interpreted as straight_join > -- > > Key: IMPALA-7484 > URL: https://issues.apache.org/jira/browse/IMPALA-7484 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Lars Volker >Assignee: Anurag Mantripragada >Priority: Major > > The call to setIsStraightJoin() in > [SelectList.java:87|https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87] > should be wrapped in an else-clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8592) Add support for insert events for 'LOAD DATA..' statements from Impala.
[ https://issues.apache.org/jira/browse/IMPALA-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-8592: - Summary: Add support for insert events for 'LOAD DATA..' statements from Impala. (was: Add support for insert events from 'LOAD DATA..' statements from Impala.) > Add support for insert events for 'LOAD DATA..' statements from Impala. > --- > > Key: IMPALA-8592 > URL: https://issues.apache.org/jira/browse/IMPALA-8592 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Priority: Major > > Hive generates INSERT events for LOAD DATA.. statements. We should support > the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8592) Add support for insert events from 'LOAD DATA..' statements from Impala.
Anurag Mantripragada created IMPALA-8592: Summary: Add support for insert events from 'LOAD DATA..' statements from Impala. Key: IMPALA-8592 URL: https://issues.apache.org/jira/browse/IMPALA-8592 Project: IMPALA Issue Type: Sub-task Reporter: Anurag Mantripragada Hive generates INSERT events for LOAD DATA.. statements. We should support the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8592) Add support for insert events for 'LOAD DATA..' statements from Impala.
[ https://issues.apache.org/jira/browse/IMPALA-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada reassigned IMPALA-8592: Assignee: Anurag Mantripragada > Add support for insert events for 'LOAD DATA..' statements from Impala. > --- > > Key: IMPALA-8592 > URL: https://issues.apache.org/jira/browse/IMPALA-8592 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > > Hive generates INSERT events for LOAD DATA.. statements. We should support > the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7973) Add support for fine-grained updates at partition level
[ https://issues.apache.org/jira/browse/IMPALA-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-7973. -- Resolution: Fixed The patch is merged. > Add support for fine-grained updates at partition level > --- > > Key: IMPALA-7973 > URL: https://issues.apache.org/jira/browse/IMPALA-7973 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > > When data is inserted into a partition or a new partition is created in a > large table, we should not be invalidating the whole table. Instead it should > be possible to refresh/add/drop certain partitions on the table directly > based on the event information. This would help with the performance of > subsequent access to the table by avoiding reloading the large table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada resolved IMPALA-7971.
--
Resolution: Fixed
The patch is merged.
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-7971:
-
Fix Version/s: Impala 3.3.0
Added fix version.
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
> Fix For: Impala 3.3.0
>
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7973) Add support for fine-grained updates at partition level
[ https://issues.apache.org/jira/browse/IMPALA-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-7973: - Fix Version/s: Impala 3.3.0 Added fix version. > Add support for fine-grained updates at partition level > --- > > Key: IMPALA-7973 > URL: https://issues.apache.org/jira/browse/IMPALA-7973 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > Fix For: Impala 3.3.0 > > > When data is inserted into a partition or a new partition is created in a > large table, we should not be invalidating the whole table. Instead it should > be possible to refresh/add/drop certain partitions on the table directly > based on the event information. This would help with the performance of > subsequent access to the table by avoiding reloading the large table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8725) Improve usability when HMS is configured with strict managed tables
Anurag Mantripragada created IMPALA-8725:
Summary: Improve usability when HMS is configured with strict
managed tables
Key: IMPALA-8725
URL: https://issues.apache.org/jira/browse/IMPALA-8725
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Anurag Mantripragada
Users tend to create and query managed tables often and when HMS is configured
with strict managed tables they get:
{code:java}
Table default.foo failed strict managed table checks due to the following
reason: Table is marked as a managed table but is not transactional{code}
We should improve usability in these scenarios.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7973) Add support for fine-grained updates at partition level
[ https://issues.apache.org/jira/browse/IMPALA-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884521#comment-16884521 ] Anurag Mantripragada commented on IMPALA-7973: -- [~arodoni_cloudera], Before this change, any partition level events would invalidate the entire table. After this change, such events selectively refresh the partitions which were involved. This applies to create, drop and alter partitions. > Add support for fine-grained updates at partition level > --- > > Key: IMPALA-7973 > URL: https://issues.apache.org/jira/browse/IMPALA-7973 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > Fix For: Impala 3.3.0 > > > When data is inserted into a partition or a new partition is created in a > large table, we should not be invalidating the whole table. Instead it should > be possible to refresh/add/drop certain partitions on the table directly > based on the event information. This would help with the performance of > subsequent access to the table by avoiding reloading the large table. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8740:
-
Summary: TestCodegen.test_disable_codegen flakiness (was:
TestCodegen.test_disable_codegen flakiness in exhaustive build)
> TestCodegen.test_disable_codegen flakiness
> ---
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Summary: Multiple test failures in S3 due to incorrect loading of
functional.alltypes (was:
TestExplain.test_explain_validate_cardinality_estimates flakiness )
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Could be a data loading issue, cardinality estimates are off.
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness in exhaustive build
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8740:
-
Target Version: Impala 3.3.0
Affects Version/s: Impala 3.3.0
Labels: broken-build (was: )
> TestCodegen.test_disable_codegen flakiness in exhaustive build
> --
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness in exhaustive build
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-8740:
Assignee: Tim Armstrong (was: Anurag Mantripragada)
> TestCodegen.test_disable_codegen flakiness in exhaustive build
> --
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Reporter: Anurag Mantripragada
>Assignee: Tim Armstrong
>Priority: Major
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness in exhaustive build
Anurag Mantripragada created IMPALA-8740:
Summary: TestCodegen.test_disable_codegen flakiness in exhaustive
build
Key: IMPALA-8740
URL: https://issues.apache.org/jira/browse/IMPALA-8740
Project: IMPALA
Issue Type: Bug
Reporter: Anurag Mantripragada
Looks like the test cannot find "Codegen Disabled: disabled due to optimization
hints" in the profile and fails the assertion:
{code:java}
query_test/test_codegen.py:44: in test_disable_codegen
self.run_test_case('QueryTest/disable-codegen', vector)
common/impala_test_suite.py:617: in run_test_case
update_section=pytest.config.option.update_results)
common/test_result_verifier.py:605: in verify_runtime_profile
actual))
E AssertionError: Did not find matches for lines in runtime profile:
E EXPECTED LINES:
E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
...
ACTUAL PROFILE FOLLOWS{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness in exhaustive build
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-8740:
Assignee: Anurag Mantripragada
> TestCodegen.test_disable_codegen flakiness in exhaustive build
> --
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-8730.
Resolution: Not A Bug
Multiple other tests failed. This is probably a data loading issue on S3.
Closing this for now.
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Could be a data loading issue, cardinality estimates are off.
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8489) TestRecoverPartitions.test_post_invalidate fails with IllegalStateException when HMS polling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8489 started by Anurag Mantripragada.
> TestRecoverPartitions.test_post_invalidate fails with IllegalStateException
> when HMS polling is enabled
> ---
>
> Key: IMPALA-8489
> URL: https://issues.apache.org/jira/browse/IMPALA-8489
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: catalog-v2
>
> {noformat}
> metadata/test_recover_partitions.py:279: in test_post_invalidate
> "INSERT INTO TABLE %s PARTITION(i=002, p='p2') VALUES(4)" % FQ_TBL_NAME)
> common/impala_test_suite.py:620: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:628: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:722: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:364: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:385: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:IllegalArgumentException: no such partition id 6244
> {noformat}
> The failure is reproducible for me locally with catalog v2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8489) TestRecoverPartitions.test_post_invalidate fails with IllegalStateException when HMS polling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-8489:
Assignee: Anurag Mantripragada (was: Vihang Karajgaonkar)
> TestRecoverPartitions.test_post_invalidate fails with IllegalStateException
> when HMS polling is enabled
> ---
>
> Key: IMPALA-8489
> URL: https://issues.apache.org/jira/browse/IMPALA-8489
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: catalog-v2
>
> {noformat}
> metadata/test_recover_partitions.py:279: in test_post_invalidate
> "INSERT INTO TABLE %s PARTITION(i=002, p='p2') VALUES(4)" % FQ_TBL_NAME)
> common/impala_test_suite.py:620: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:628: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:722: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:364: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:385: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:IllegalArgumentException: no such partition id 6244
> {noformat}
> The failure is reproducible for me locally with catalog v2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8579) Ignore trivial alter events
[
https://issues.apache.org/jira/browse/IMPALA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-8579:
Assignee: Anurag Mantripragada
> Ignore trivial alter events
> ---
>
> Key: IMPALA-8579
> URL: https://issues.apache.org/jira/browse/IMPALA-8579
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: catalog-v2
>
> Certain alter table and alter partition events are actually just to update
> the {{lastaccesstime}}. We should ignore such alter events to avoid
> unnecessary refresh/invalidates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Description:
Multiple tests involving functional.alltypes failed. Most likely due to data
loading issue on S3. Most are assertion failures because of row_count mismatch.
Below is the test failures:
{code:java}
1.
metadata.test_metadata_query_statements.TestMetadataQueryStatements.test_show_stats
2.
metadata.test_explain.TestExplain.test_explain_validate_cardinality_estimates
3. query_test.test_insert.TestInsertQueries.test_insert
4. query_test.test_runtime_filters.TestRuntimeFilters.test_basic_filters{code}
was:
Could be a data loading issue, cardinality estimates are off.
Below is the error message:
{code:java}
metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
check_cardinality query_result, expected_cardinality=expected_cardinality)
metadata/test_explain.py:86: in check_row_size_and_cardinality assert
m.groups()[1] == expected_cardinality E assert '7.00K' == '7.30K' E -
7.00K E ? ^ E + 7.30K E ? ^{code}
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Multiple tests involving functional.alltypes failed. Most likely due to data
> loading issue on S3. Most are assertion failures because of row_count
> mismatch.
>
> Below is the test failures:
> {code:java}
> 1.
> metadata.test_metadata_query_statements.TestMetadataQueryStatements.test_show_stats
>
> 2.
> metadata.test_explain.TestExplain.test_explain_validate_cardinality_estimates
> 3. query_test.test_insert.TestInsertQueries.test_insert
> 4. query_test.test_runtime_filters.TestRuntimeFilters.test_basic_filters{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Comment: was deleted
(was: Thank you for testing it locally [~fangyurao]. My guess is this could be
a dataloading issue. Let's wait for next run. Could be a one-off thing.
)
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Multiple tests involving functional.alltypes failed. Most likely due to data
> loading issue on S3. Most are assertion failures because of row_count
> mismatch.
>
> Below is the test failures:
> {code:java}
> 1.
> metadata.test_metadata_query_statements.TestMetadataQueryStatements.test_show_stats
>
> 2.
> metadata.test_explain.TestExplain.test_explain_validate_cardinality_estimates
> 3. query_test.test_insert.TestInsertQueries.test_insert
> 4. query_test.test_runtime_filters.TestRuntimeFilters.test_basic_filters{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878233#comment-16878233
]
Anurag Mantripragada commented on IMPALA-8730:
--
Thanks [~fangyurao], this was probably a data loading issue, updated the title
and description accordingly. Please feel free to reassign.
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Multiple tests involving functional.alltypes failed. Most likely due to data
> loading issue on S3. Most are assertion failures because of row_count
> mismatch.
>
> Below is the test failures:
> {code:java}
> 1.
> metadata.test_metadata_query_statements.TestMetadataQueryStatements.test_show_stats
>
> 2.
> metadata.test_explain.TestExplain.test_explain_validate_cardinality_estimates
> 3. query_test.test_insert.TestInsertQueries.test_insert
> 4. query_test.test_runtime_filters.TestRuntimeFilters.test_basic_filters{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reopened IMPALA-8730:
--
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Could be a data loading issue, cardinality estimates are off.
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-8730) Multiple test failures in S3 due to incorrect loading of functional.alltypes
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Comment: was deleted
(was: Multiple other tests failed. This is probably a data loading issue on S3.
Closing this for now.)
> Multiple test failures in S3 due to incorrect loading of functional.alltypes
>
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Could be a data loading issue, cardinality estimates are off.
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8740:
-
Attachment: full_profile.txt
> TestCodegen.test_disable_codegen flakiness
> ---
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build
> Attachments: full_profile.txt
>
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8740) TestCodegen.test_disable_codegen flakiness
[
https://issues.apache.org/jira/browse/IMPALA-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8740:
-
Attachment: (was: full_profile.txt)
> TestCodegen.test_disable_codegen flakiness
> ---
>
> Key: IMPALA-8740
> URL: https://issues.apache.org/jira/browse/IMPALA-8740
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build
>
> Looks like the test cannot find "Codegen Disabled: disabled due to
> optimization hints" in the profile and fails the assertion:
> {code:java}
> query_test/test_codegen.py:44: in test_disable_codegen
> self.run_test_case('QueryTest/disable-codegen', vector)
> common/impala_test_suite.py:617: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:605: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*Codegen Disabled: disabled due to optimization hints.*
> ...
> ACTUAL PROFILE FOLLOWS{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8730) TestExplain.test_explain_validate_cardinality_estimates flakiness due to reliance on exact cardinality numbers.
Anurag Mantripragada created IMPALA-8730:
Summary: TestExplain.test_explain_validate_cardinality_estimates
flakiness due to reliance on exact cardinality numbers.
Key: IMPALA-8730
URL: https://issues.apache.org/jira/browse/IMPALA-8730
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 3.3.0
Reporter: Anurag Mantripragada
Assignee: Fang-Yu Rao
Another flaky test uncovered due to IMPALA-7608. The test relies on exact
cardinality numbers but after IMPALA-7608, these could be indeterministic.
Below is the error message:
{code:java}
metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
check_cardinality query_result, expected_cardinality=expected_cardinality)
metadata/test_explain.py:86: in check_row_size_and_cardinality assert
m.groups()[1] == expected_cardinality E assert '7.00K' == '7.30K' E -
7.00K E ? ^ E + 7.30K E ? ^{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8730) TestExplain.test_explain_validate_cardinality_estimates flakiness
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Summary: TestExplain.test_explain_validate_cardinality_estimates flakiness
(was: TestExplain.test_explain_validate_cardinality_estimates flakiness due to
reliance on exact cardinality numbers.)
> TestExplain.test_explain_validate_cardinality_estimates flakiness
> --
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Another flaky test uncovered due to IMPALA-7608. The test relies on exact
> cardinality numbers but after IMPALA-7608, these could be indeterministic.
>
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8730) TestExplain.test_explain_validate_cardinality_estimates flakiness
[
https://issues.apache.org/jira/browse/IMPALA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8730:
-
Description:
Could be a data loading issue, cardinality estimates are off.
Below is the error message:
{code:java}
metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
check_cardinality query_result, expected_cardinality=expected_cardinality)
metadata/test_explain.py:86: in check_row_size_and_cardinality assert
m.groups()[1] == expected_cardinality E assert '7.00K' == '7.30K' E -
7.00K E ? ^ E + 7.30K E ? ^{code}
was:
Another flaky test uncovered due to IMPALA-7608. The test relies on exact
cardinality numbers but after IMPALA-7608, these could be indeterministic.
Below is the error message:
{code:java}
metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
check_cardinality query_result, expected_cardinality=expected_cardinality)
metadata/test_explain.py:86: in check_row_size_and_cardinality assert
m.groups()[1] == expected_cardinality E assert '7.00K' == '7.30K' E -
7.00K E ? ^ E + 7.30K E ? ^{code}
> TestExplain.test_explain_validate_cardinality_estimates flakiness
> --
>
> Key: IMPALA-8730
> URL: https://issues.apache.org/jira/browse/IMPALA-8730
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Anurag Mantripragada
>Assignee: Fang-Yu Rao
>Priority: Critical
> Labels: broken-build
>
> Could be a data loading issue, cardinality estimates are off.
> Below is the error message:
> {code:java}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
>check_cardinality(result.data, '7.30K') metadata/test_explain.py:98: in
> check_cardinality query_result,
> expected_cardinality=expected_cardinality) metadata/test_explain.py:86: in
> check_row_size_and_cardinality assert m.groups()[1] ==
> expected_cardinality E assert '7.00K' == '7.30K' E - 7.00K E ? ^
> E + 7.30K E ? ^{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8489) TestRecoverPartitions.test_post_invalidate fails with IllegalStateException when HMS polling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-8489.
Resolution: Fixed
Committed to master.
> TestRecoverPartitions.test_post_invalidate fails with IllegalStateException
> when HMS polling is enabled
> ---
>
> Key: IMPALA-8489
> URL: https://issues.apache.org/jira/browse/IMPALA-8489
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: catalog-v2
>
> {noformat}
> metadata/test_recover_partitions.py:279: in test_post_invalidate
> "INSERT INTO TABLE %s PARTITION(i=002, p='p2') VALUES(4)" % FQ_TBL_NAME)
> common/impala_test_suite.py:620: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:628: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:722: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:364: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:385: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:IllegalArgumentException: no such partition id 6244
> {noformat}
> The failure is reproducible for me locally with catalog v2.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-8124.
Resolution: Fixed
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Closed] (IMPALA-7935) /catalog_object end point broken in LocalCatalog mode
[
https://issues.apache.org/jira/browse/IMPALA-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-7935.
Resolution: Fixed
> /catalog_object end point broken in LocalCatalog mode
> -
>
> Key: IMPALA-7935
> URL: https://issues.apache.org/jira/browse/IMPALA-7935
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: catalog-v2, observability
>
> Start Impala coordinator in LocalCatalog mode (v2)
> {code}
> start-impala-cluster.py -s 1 --impalad_args="--use_local_catalog=true"
> --catalogd_args="--catalog_topic_mode=minimal"
> {code}
> Check the following URL.
> http://:25000/catalog_object?object_type=TABLE_name=functional_text_lzo.alltypesaggmultifiles
> It says,
> {noformat}
> Error: UnsupportedOperationException: LocalCatalog.getTCatalogObject
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8901) # links on /catalog page broken
[ https://issues.apache.org/jira/browse/IMPALA-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8901 started by Anurag Mantripragada. > # links on /catalog page broken > --- > > Key: IMPALA-8901 > URL: https://issues.apache.org/jira/browse/IMPALA-8901 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Thomas Tauber-Marshall >Assignee: Anurag Mantripragada >Priority: Minor > > Due to IMPALA-7935, the database links at the top of the /catalog page which > use '#' links to jump to the part of the page corresponding to the particular > database no longer work if local catalog is enabled because the tag with > the id corresponding to the database name is omitted from the page. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8579) Ignore trivial alter events
[
https://issues.apache.org/jira/browse/IMPALA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8579 started by Anurag Mantripragada.
> Ignore trivial alter events
> ---
>
> Key: IMPALA-8579
> URL: https://issues.apache.org/jira/browse/IMPALA-8579
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: catalog-v2
>
> Certain alter table and alter partition events are actually just to update
> the {{lastaccesstime}}. We should ignore such alter events to avoid
> unnecessary refresh/invalidates.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8901) # links on /catalog page broken
[ https://issues.apache.org/jira/browse/IMPALA-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-8901. -- Resolution: Fixed > # links on /catalog page broken > --- > > Key: IMPALA-8901 > URL: https://issues.apache.org/jira/browse/IMPALA-8901 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Thomas Tauber-Marshall >Assignee: Anurag Mantripragada >Priority: Minor > > Due to IMPALA-7935, the database links at the top of the /catalog page which > use '#' links to jump to the part of the page corresponding to the particular > database no longer work if local catalog is enabled because the tag with > the id corresponding to the database name is omitted from the page. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8579) Ignore trivial alter events
[
https://issues.apache.org/jira/browse/IMPALA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada resolved IMPALA-8579.
--
Resolution: Fixed
> Ignore trivial alter events
> ---
>
> Key: IMPALA-8579
> URL: https://issues.apache.org/jira/browse/IMPALA-8579
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: catalog-v2
>
> Certain alter table and alter partition events are actually just to update
> the {{lastaccesstime}}. We should ignore such alter events to avoid
> unnecessary refresh/invalidates.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8761) Configuration validation introduced in IMPALA-8559 can be improved
[ https://issues.apache.org/jira/browse/IMPALA-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8761 started by Anurag Mantripragada. > Configuration validation introduced in IMPALA-8559 can be improved > -- > > Key: IMPALA-8761 > URL: https://issues.apache.org/jira/browse/IMPALA-8761 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > > The issue with configuration validation in IMPALA-8559 is that it validates > one configuration at a time and fails as soon as there is a validation error. > Since there are more than one configuration keys to validate, user may have > to restart HMS again and again if there are multiple configuration changes > which are needed. This is not a great user experience. A simple improvement > that can be made is do all the configuration validations together and then > present the results together in case of failures so that user can change all > the required changes in one go. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8761) Configuration validation introduced in IMPALA-8559 can be improved
[ https://issues.apache.org/jira/browse/IMPALA-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada reassigned IMPALA-8761: Assignee: Anurag Mantripragada > Configuration validation introduced in IMPALA-8559 can be improved > -- > > Key: IMPALA-8761 > URL: https://issues.apache.org/jira/browse/IMPALA-8761 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Anurag Mantripragada >Priority: Major > > The issue with configuration validation in IMPALA-8559 is that it validates > one configuration at a time and fails as soon as there is a validation error. > Since there are more than one configuration keys to validate, user may have > to restart HMS again and again if there are multiple configuration changes > which are needed. This is not a great user experience. A simple improvement > that can be made is do all the configuration validations together and then > present the results together in case of failures so that user can change all > the required changes in one go. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903416#comment-16903416
]
Anurag Mantripragada commented on IMPALA-8124:
--
Thanks [~Xiaomeng Zhang] for letting us know, I will take a look at it.
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Work started] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8124 started by Anurag Mantripragada.
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8489) TestRecoverPartitions.test_post_invalidate fails with IllegalStateException when HMS polling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888331#comment-16888331
]
Anurag Mantripragada commented on IMPALA-8489:
--
[~bharathv] - As per your gerrit comments, I reproduced this locally and here
is the sequence of steps to reproduce.
{code:java}
CREATE TABLE foo (c int) PARTITIONED BY (i int, p string);
INSERT INTO TABLE foo PARTITION(i=1, p='p1') VALUES(1);
-- Create a partition directory manually and add a file there
ALTER TABLE foo RECOVER PARTITIONS;
INVALIDATE METADATA;
INSERT INTO TABLE test_recover_partitions PARTITION(i=002, p='p2')
VALUES(4);{code}
The final insert throws the exception as the partition ids have changed after a
drop + recreate of partition (i=002, p='p2'). Catalogd logs below:
{code:java}
I0718 13:52:47.996079 20026 jni-util.cc:288]
java.lang.IllegalArgumentException: no such partition id 6
at org.apache.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:343)
at
org.apache.impala.service.CatalogOpExecutor.createInsertEvents(CatalogOpExecutor.java:3885)
at
org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:3843)
at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:322)
I0718 13:52:48.129122 20026 status.cc:124] IllegalArgumentException: no such
partition id 6{code}
> TestRecoverPartitions.test_post_invalidate fails with IllegalStateException
> when HMS polling is enabled
> ---
>
> Key: IMPALA-8489
> URL: https://issues.apache.org/jira/browse/IMPALA-8489
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: catalog-v2
>
> {noformat}
> metadata/test_recover_partitions.py:279: in test_post_invalidate
> "INSERT INTO TABLE %s PARTITION(i=002, p='p2') VALUES(4)" % FQ_TBL_NAME)
> common/impala_test_suite.py:620: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:628: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:722: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:364: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:385: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:IllegalArgumentException: no such partition id 6244
> {noformat}
> The failure is reproducible for me locally with catalog v2.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8489) TestRecoverPartitions.test_post_invalidate fails with IllegalStateException when HMS polling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888331#comment-16888331
]
Anurag Mantripragada edited comment on IMPALA-8489 at 7/18/19 9:23 PM:
---
[~bharathv] - As per your gerrit comments, I reproduced this locally and here
is the sequence of steps to reproduce.
{code:java}
CREATE TABLE foo (c int) PARTITIONED BY (i int, p string);
INSERT INTO TABLE foo PARTITION(i=1, p='p1') VALUES(1);
-- Create a partition (i=002, p='p2') directory manually and add a file there --
ALTER TABLE foo RECOVER PARTITIONS;
INVALIDATE METADATA;
INSERT INTO TABLE test_recover_partitions PARTITION(i=002, p='p2')
VALUES(4);{code}
The final insert throws the exception as the partition ids have changed after a
drop + recreate of partition (i=002, p='p2'). Catalogd logs below:
{code:java}
I0718 13:52:47.996079 20026 jni-util.cc:288]
java.lang.IllegalArgumentException: no such partition id 6
at org.apache.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:343)
at
org.apache.impala.service.CatalogOpExecutor.createInsertEvents(CatalogOpExecutor.java:3885)
at
org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:3843)
at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:322)
I0718 13:52:48.129122 20026 status.cc:124] IllegalArgumentException: no such
partition id 6{code}
was (Author: anuragmantri):
[~bharathv] - As per your gerrit comments, I reproduced this locally and here
is the sequence of steps to reproduce.
{code:java}
CREATE TABLE foo (c int) PARTITIONED BY (i int, p string);
INSERT INTO TABLE foo PARTITION(i=1, p='p1') VALUES(1);
-- Create a partition directory manually and add a file there
ALTER TABLE foo RECOVER PARTITIONS;
INVALIDATE METADATA;
INSERT INTO TABLE test_recover_partitions PARTITION(i=002, p='p2')
VALUES(4);{code}
The final insert throws the exception as the partition ids have changed after a
drop + recreate of partition (i=002, p='p2'). Catalogd logs below:
{code:java}
I0718 13:52:47.996079 20026 jni-util.cc:288]
java.lang.IllegalArgumentException: no such partition id 6
at org.apache.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:343)
at
org.apache.impala.service.CatalogOpExecutor.createInsertEvents(CatalogOpExecutor.java:3885)
at
org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:3843)
at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:322)
I0718 13:52:48.129122 20026 status.cc:124] IllegalArgumentException: no such
partition id 6{code}
> TestRecoverPartitions.test_post_invalidate fails with IllegalStateException
> when HMS polling is enabled
> ---
>
> Key: IMPALA-8489
> URL: https://issues.apache.org/jira/browse/IMPALA-8489
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: catalog-v2
>
> {noformat}
> metadata/test_recover_partitions.py:279: in test_post_invalidate
> "INSERT INTO TABLE %s PARTITION(i=002, p='p2') VALUES(4)" % FQ_TBL_NAME)
> common/impala_test_suite.py:620: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:628: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:722: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:364: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:385: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:IllegalArgumentException: no such partition id 6244
> {noformat}
> The failure is reproducible for me locally with catalog v2.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8968) Fix self-event detection on database events.
[
https://issues.apache.org/jira/browse/IMPALA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada closed IMPALA-8968.
Resolution: Fixed
> Fix self-event detection on database events.
>
>
> Key: IMPALA-8968
> URL: https://issues.apache.org/jira/browse/IMPALA-8968
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When event processing is turned on, self-events are not detected for DATABASE
> level events (create/alter/drop database). Reproduced using the below
> statements:
> {code:java}
> CREATE DATABASE test;
> CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
> '/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
> DROP FUNCTION test.fn_test(INT, STRING);
> DROP DATABASE test CASCADE; {code}
> Since the events processor could not detect self-events, it will try to
> process the ALTER_DATABASE event created by dropping the function. However,
> it doesn't find the DB and errors out like below:
> {code:java}
> I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
> events. Start event id : 30077
> I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
> events. Start event id : 30077
> I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
> EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
> database test
> I0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId: 30079
> EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
> database test
> I0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number of events
> received: 2 Total number of events filtered out: 0
> I0923 16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078
> EventType: ALTER_DATABASE Received exception Database test not found.
> Ignoring self-event evaluation
> E0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525] Unexpected
> exception received while processing event
> Java exception
> follows:org.apache.impala.catalog.events.MetastoreNotificationException:
> Unable to process event 30078 of type ALTER_DATABASE. Event processing will
> be stopped.
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.impala.catalog.DatabaseNotFoundException: Database test
> not found
> at
> org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
> at
> org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
> ... 8 more
> E0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631] Notification
> event is null
> W0923 16:09:56.057376 6727 MetastoreEventsProcessor.java:504] Event
> processing is skipped since status is ERROR. Last synced event id is
> 30077{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8968) Fix self-event detection on database events.
Anurag Mantripragada created IMPALA-8968:
Summary: Fix self-event detection on database events.
Key: IMPALA-8968
URL: https://issues.apache.org/jira/browse/IMPALA-8968
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Anurag Mantripragada
Assignee: Anurag Mantripragada
Self-events are not detected for DATABASE level events (create/alter/drop
database). Reproduced using the below statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the event's processor could not detect self-events, it will try to
process the ALTER_DATABASE event created by dropping the function. However, it
doesn't find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525]
Unexpected exception received while processing eventJava exception
follows:org.apache.impala.catalog.events.MetastoreNotificationException: Unable
to process event 30078 of type ALTER_DATABASE. Event processing will be
stopped. at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.impala.catalog.DatabaseNotFoundException: Database test not found at
org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
at
org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
Notification event is nullW0923 16:09:56.057376 6727
MetastoreEventsProcessor.java:504] Event processing is skipped since status is
ERROR. Last synced event id is 30077{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8968) Fix self-event detection on database events.
[
https://issues.apache.org/jira/browse/IMPALA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8968:
-
Description:
Self-events are not detected for DATABASE level events (create/alter/drop
database). Reproduced using the below statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the event's processor could not detect self-events, it will try to
process the ALTER_DATABASE event created by dropping the function. However, it
doesn't find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525]
Unexpected exception received while processing eventJava exception
follows:org.apache.impala.catalog.events.MetastoreNotificationException: Unable
to process event 30078 of type ALTER_DATABASE. Event processing will be
stopped. at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.impala.catalog.DatabaseNotFoundException: Database test not found at
org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
at
org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
Notification event is nullW0923 16:09:56.057376 6727
MetastoreEventsProcessor.java:504] Event processing is skipped since status is
ERROR. Last synced event id is 30077{code}
was:
Self-events are not detected for DATABASE level events (create/alter/drop
database). Reproduced using the below statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the event's processor could not detect self-events, it will try to
process the ALTER_DATABASE event created by dropping the function. However, it
doesn't find the DB and errors out like below:
{noformat}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727
[jira] [Updated] (IMPALA-8968) Fix self-event detection on database events.
[
https://issues.apache.org/jira/browse/IMPALA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8968:
-
Description:
When event processing is turned on, self-events are not detected for DATABASE
level events (create/alter/drop database). Reproduced using the below
statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the events processor could not detect self-events, it will try to process
the ALTER_DATABASE event created by dropping the function. However, it doesn't
find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525]
Unexpected exception received while processing eventJava exception
follows:org.apache.impala.catalog.events.MetastoreNotificationException: Unable
to process event 30078 of type ALTER_DATABASE. Event processing will be
stopped. at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.impala.catalog.DatabaseNotFoundException: Database test not found at
org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
at
org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
Notification event is nullW0923 16:09:56.057376 6727
MetastoreEventsProcessor.java:504] Event processing is skipped since status is
ERROR. Last synced event id is 30077{code}
was:
Self-events are not detected for DATABASE level events (create/alter/drop
database). Reproduced using the below statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the events processor could not detect self-events, it will try to process
the ALTER_DATABASE event created by dropping the function. However, it doesn't
find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.01 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923
[jira] [Created] (IMPALA-9119) query_test.test_tpch_queries.TestTpchQuery.test_tpch failed with "Memory limit Exceeded."
Anurag Mantripragada created IMPALA-9119:
Summary: query_test.test_tpch_queries.TestTpchQuery.test_tpch
failed with "Memory limit Exceeded."
Key: IMPALA-9119
URL: https://issues.apache.org/jira/browse/IMPALA-9119
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.3.0
Reporter: Anurag Mantripragada
Observed during a GVD run here:
[https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1529/]
Part of the error message:
{code:java}
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EQuery aborted:Memory limit exceeded: Cannot perform aggregation at
aggregator with id 4. Failed to allocate 25 bytes for intermediate tuple.
E Fragment 85444bc502604143:43739df8000b could not allocate 25.00 B
without exceeding limit.
E Error occurred on backend e6a3e789bfca:22000 by fragment
85444bc502604143:43739df8000b
E Memory left in process limit: 6.82 GB
E Memory left in query limit: 10.48 KB {code}
Aggregation node id 4:
{code:java}
E Fragment 85444bc502604143:43739df8000b: Reservation=42.00 MB
OtherMemory=1.64 MB Total=43.64 MB Peak=43.70 MB
E AGGREGATION_NODE (id=4): Reservation=42.00 MB OtherMemory=42.12 KB
Total=42.04 MB Peak=42.04 MB
E GroupingAggregator 0: Reservation=42.00 MB OtherMemory=17.12 KB
Total=42.02 MB Peak=42.02 MB
E Exprs: Total=17.12 KB Peak=17.12 KB
E KUDU_SCAN_NODE (id=3): Total=1.49 MB Peak=1.51 MB
E Queued Batches: Total=1.49 MB Peak=1.51 MB
E KrpcDataStreamSender (dst_id=13): Total=83.35 KB Peak=99.35 KB
E CodeGen: Total=5.29 KB Peak=596.50 KB {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9142) Failure in PlannerTest.testSpillableBufferSizing: SCAN HDFS columns missing.
Anurag Mantripragada created IMPALA-9142:
Summary: Failure in PlannerTest.testSpillableBufferSizing: SCAN
HDFS columns missing.
Key: IMPALA-9142
URL: https://issues.apache.org/jira/browse/IMPALA-9142
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Anurag Mantripragada
Assignee: Tim Armstrong
Not sure if the mt_dop planner changes has anything to do with this. Please
feel free to reassign this.
It seems like the columns are missing for functional_parquet.alltypestiny table.
Happened here:
[https://master-02.jenkins.cloudera.com/job/impala-asf-master-exhaustive/866]
{code:java}
Actual:
|--03:EXCHANGE [BROADCAST]
| | mem-estimate=251.92KB mem-reservation=0B thread-reservation=0
| | tuple-ids=1 row-size=80B cardinality=unavailable
| | in pipelines: 01(GETNEXT)
| |
| F01:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
| Per-Host Resources: mem-estimate=16.00MB mem-reservation=88.00KB
thread-reservation=2
| 01:SCAN HDFS [functional_parquet.alltypestiny, RANDOM]
| HDFS partitions=4/4 files=4 size=11.92KB
| stored statistics:
| table: rows=unavailable size=unavailable
| partitions: 0/4 rows=unavailable
| columns: unavailable
Expected:
|--03:EXCHANGE [BROADCAST]
| | mem-estimate=251.92KB mem-reservation=0B thread-reservation=0
| | tuple-ids=1 row-size=80B cardinality=unavailable
| | in pipelines: 01(GETNEXT)
| |
| F01:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
| Per-Host Resources: mem-estimate=16.00MB mem-reservation=88.00KB
thread-reservation=2
| 01:SCAN HDFS [functional_parquet.alltypestiny, RANDOM]
| HDFS partitions=4/4 files=4 size=11.67KB
| stored statistics:
| table: rows=unavailable size=unavailable
| partitions: 0/4 rows=unavailable
| columns missing stats: id, bool_col, tinyint_col, smallint_col,
int_col, bigint_col, float_col, double_col, date_string_col, string_col,
timestamp_col {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9122) TestEventProcessing.test_insert_events flaky in precommit
[
https://issues.apache.org/jira/browse/IMPALA-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned IMPALA-9122:
Assignee: Anurag Mantripragada (was: Vihang Karajgaonkar)
> TestEventProcessing.test_insert_events flaky in precommit
> -
>
> Key: IMPALA-9122
> URL: https://issues.apache.org/jira/browse/IMPALA-9122
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: broken-build, flaky
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8613/
> {noformat}
> custom_cluster.test_event_processing.TestEventProcessing.test_insert_events
> (from pytest)
> Failing for the past 1 build (Since Failed#8613 )
> Took 1 min 20 sec.
> add description
> Error Message
> assert >(18421)
> is True + where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> Stacktrace
> custom_cluster/test_event_processing.py:82: in test_insert_events
> self.run_test_insert_events()
> custom_cluster/test_event_processing.py:143: in run_test_insert_events
> assert self.wait_for_insert_event_processing(last_synced_event_id) is True
> E assert of 0x7f7fa86ec6d0>>(18421) is True
> E+ where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9104) Add support for retrieval of primary keys and foreign keys in impala-hs2-server.
[ https://issues.apache.org/jira/browse/IMPALA-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9104 started by Anurag Mantripragada. > Add support for retrieval of primary keys and foreign keys in > impala-hs2-server. > > > Key: IMPALA-9104 > URL: https://issues.apache.org/jira/browse/IMPALA-9104 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > > Impala's TCLIService.thrift files was taken from Hive-1 days and does not > contain APIs related to primary keys and foreign keys that was added to Hive > jdbc driver start Hive-2.1. We need to port these APIs inorder to be able to > use them in Impala JDBC clients. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9122) TestEventProcessing.test_insert_events flaky in precommit
[
https://issues.apache.org/jira/browse/IMPALA-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967818#comment-16967818
]
Anurag Mantripragada commented on IMPALA-9122:
--
Sure, taking a look at it.
> TestEventProcessing.test_insert_events flaky in precommit
> -
>
> Key: IMPALA-9122
> URL: https://issues.apache.org/jira/browse/IMPALA-9122
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Labels: broken-build, flaky
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8613/
> {noformat}
> custom_cluster.test_event_processing.TestEventProcessing.test_insert_events
> (from pytest)
> Failing for the past 1 build (Since Failed#8613 )
> Took 1 min 20 sec.
> add description
> Error Message
> assert >(18421)
> is True + where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> Stacktrace
> custom_cluster/test_event_processing.py:82: in test_insert_events
> self.run_test_insert_events()
> custom_cluster/test_event_processing.py:143: in run_test_insert_events
> assert self.wait_for_insert_event_processing(last_synced_event_id) is True
> E assert of 0x7f7fa86ec6d0>>(18421) is True
> E+ where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9132) Explain statements should not cause NPE in LogLineageRecord()
Anurag Mantripragada created IMPALA-9132: Summary: Explain statements should not cause NPE in LogLineageRecord() Key: IMPALA-9132 URL: https://issues.apache.org/jira/browse/IMPALA-9132 Project: IMPALA Issue Type: Bug Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada For DDLs, LogLineageRecord() adds certain fields to the lineageGraph in the backend. However, explain statements do not have a catalogOpExecutor causing a NPE. We should, in general, avoid creating lineage records for Explain as Atlas currently does not use them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9116) SASL server fails when FQDN is greater than 63 characters long in Kudu RPC
Anurag Mantripragada created IMPALA-9116: Summary: SASL server fails when FQDN is greater than 63 characters long in Kudu RPC Key: IMPALA-9116 URL: https://issues.apache.org/jira/browse/IMPALA-9116 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 3.3.0 Reporter: Anurag Mantripragada Fix For: Impala 3.4.0 In the current Kudu RPC implementation, we don't explicitly pass the host's FQDN into the SASL library. Due to an upstream SASL bug ([https://github.com/cyrusimap/cyrus-sasl/issues/583]) the FQDN gets truncated when trying to determine the server's principal, in the case that the server's fQDN is longer than 64 characters. This results in startup failures where the preflight checks fail due to not finding the appropriate keytab entry (after searching for a truncated host name) To work around this, we should use our own code to compute the FQDN. Kudu is making the changes in it's own implementation here: https://issues.apache.org/jira/browse/KUDU-2989, we should do the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9132) Explain statements should not cause NPE in LogLineageRecord()
[ https://issues.apache.org/jira/browse/IMPALA-9132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-9132. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed > Explain statements should not cause NPE in LogLineageRecord() > - > > Key: IMPALA-9132 > URL: https://issues.apache.org/jira/browse/IMPALA-9132 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Blocker > Labels: crash > Fix For: Impala 3.4.0 > > > For DDLs, LogLineageRecord() adds certain fields to the lineageGraph in the > backend. However, explain statements do not have a catalogOpExecutor causing > a NPE. > We should, in general, avoid creating lineage records for Explain as Atlas > currently does not use them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9158) Support loading PK/FK constraints in LocalCatalog.
[ https://issues.apache.org/jira/browse/IMPALA-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9158 started by Anurag Mantripragada. > Support loading PK/FK constraints in LocalCatalog. > -- > > Key: IMPALA-9158 > URL: https://issues.apache.org/jira/browse/IMPALA-9158 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > > Currently, we only added support for loading PK/FK information for Catalog > V1. Supporting it in LocalCatlog needs implementing loading in > CatalogMetaProvider and DirectMetaProvider. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada resolved IMPALA-2112.
--
Resolution: Fixed
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kinard
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
> DISABLE NOVALIDATE);}}
> * {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
> key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9122) TestEventProcessing.test_insert_events flaky in precommit
[
https://issues.apache.org/jira/browse/IMPALA-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976830#comment-16976830
]
Anurag Mantripragada commented on IMPALA-9122:
--
Sorry for the delay in investigating this. A quick look at the logs reveals
that during the partition refresh, one of the files in .staging directory
caused a FileNotFoundException, throwing events processor into the ERROR state.
IIUC, files in .statging directories should be ignored. I will try to reproduce
this locally and investigate further.
> TestEventProcessing.test_insert_events flaky in precommit
> -
>
> Key: IMPALA-9122
> URL: https://issues.apache.org/jira/browse/IMPALA-9122
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: broken-build, flaky
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8613/
> {noformat}
> custom_cluster.test_event_processing.TestEventProcessing.test_insert_events
> (from pytest)
> Failing for the past 1 build (Since Failed#8613 )
> Took 1 min 20 sec.
> add description
> Error Message
> assert >(18421)
> is True + where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> Stacktrace
> custom_cluster/test_event_processing.py:82: in test_insert_events
> self.run_test_insert_events()
> custom_cluster/test_event_processing.py:143: in run_test_insert_events
> assert self.wait_for_insert_event_processing(last_synced_event_id) is True
> E assert of 0x7f7fa86ec6d0>>(18421) is True
> E+ where TestEventProcessing.wait_for_insert_event_processing of
> > =
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9188) Dataload is failing when USE_CDP_HIVE=true
[
https://issues.apache.org/jira/browse/IMPALA-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980636#comment-16980636
]
Anurag Mantripragada commented on IMPALA-9188:
--
I think the bug is at this line:
[https://github.com/apache/impala/blob/e716e76cccf59c2780571429b1b945d6bbc61b8d/fe/src/main/java/org/apache/impala/analysis/TableDef.java#L497]
For a composite primary key like (id, year) we are generating unique constraint
names for each column whereas, they should have the same constraint name. In
Hive, the comparator first sorts using constraint name and then key_seq if
constraint names are same.This is why the hive comparator is giving different
results. We should generate a new name only if key_seq is 1, if not, we should
use existing constraint name. We already do something similar for foreign keys.
[https://github.com/apache/impala/blob/e716e76cccf59c2780571429b1b945d6bbc61b8d/fe/src/main/java/org/apache/impala/analysis/TableDef.java#L565]
> Dataload is failing when USE_CDP_HIVE=true
> --
>
> Key: IMPALA-9188
> URL: https://issues.apache.org/jira/browse/IMPALA-9188
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> When USE_CDP_HIVE=true, Impala builds are failing during dataload when
> creating tables with PK/FK constraints.
> The error is:
> {code:java}
> ERROR: CREATE EXTERNAL TABLE IF NOT EXISTS
> functional_seq_record_snap.child_table (
> seq int, id int, year string, a int, primary key(seq) DISABLE NOVALIDATE
> RELY, foreign key
> (id, year) references functional_seq_record_snap.parent_table(id, year)
> DISABLE NOVALIDATE
> RELY, foreign key(a) references functional_seq_record_snap.parent_table_2(a)
> DISABLE
> NOVALIDATE RELY)
> row format delimited fields terminated by ','
> LOCATION '/test-warehouse/child_table'
> Traceback (most recent call last):
> File "Impala/bin/load-data.py", line 208, in exec_impala_query_from_file
> result = impala_client.execute(query)
> File "Impala/tests/beeswax/impala_beeswax.py", line 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "Impala/tests/beeswax/impala_beeswax.py", line 362, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File "Impala/tests/beeswax/impala_beeswax.py", line 356, in
> execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File "Impala/tests/beeswax/impala_beeswax.py", line 519, in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:
> CAUSED BY: MetaException: Foreign key references id:int;year:string; but no
> corresponding primary key or unique key exists. Possible keys:
> [year:string;id:int;]{code}
> The corresponding error in HMS is:
> {code:java}
> 2019-11-22T06:36:59,937 INFO [pool-10-thread-13] metastore.HiveMetaStore:
> 18: source:127.0.0.1 create_table_req: Table(tableName:child_table,
> dbName:functional_seq_record_gzip, owner:jenkins, createTime:0,
> lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:seq, type:int, comment:null),
> FieldSchema(name:id, type:int, comment:null), FieldSchema(name:year,
> type:string, comment:null), FieldSchema(name:a, type:int, comment:null)],
> location:hdfs://localhost:20500/test-warehouse/child_table,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=,, field.delim=,}), bucketCols:null,
> sortCols:null, parameters:null), partitionKeys:[], parameters:{EXTERNAL=TRUE,
> OBJCAPABILITIES=EXTREAD,EXTWRITE}, viewOriginalText:null,
> viewExpandedText:null, tableType:EXTERNAL_TABLE, catName:hive,
> ownerType:USER, accessType:8)
> 2019-11-22T06:36:59,937 INFO [pool-10-thread-13] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 create_table_req:
> Table(tableName:child_table, dbName:functional_seq_record_gzip,
> owner:jenkins, createTime:0, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:seq, type:int, comment:null),
> FieldSchema(name:id, type:int, comment:null), FieldSchema(name:year,
> type:string, comment:null), FieldSchema(name:a, type:int, comment:null)],
> location:hdfs://localhost:20500/test-warehouse/child_table,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
>
[jira] [Commented] (IMPALA-9188) Dataload is failing when USE_CDP_HIVE=true
[
https://issues.apache.org/jira/browse/IMPALA-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980597#comment-16980597
]
Anurag Mantripragada commented on IMPALA-9188:
--
>From the logs, it looks like there was a mismatch in the order of the PK
>column names[expecting (year,id), but got (id, year)]:
{code:java}
2019-11-22T06:36:59,945 ERROR [pool-10-thread-13]
metastore.RetryingHMSHandler: MetaException(message:Foreign key references
id:int;year:string; but no corresponding primary key or unique key exists.
Possible keys: [year:string;id:int;]){code}
I looked at the differences in what Hive is doing in Hive 2 and Hive 3. Looks
like Hive 3 introduced a canonical way to get PK column names (using a
combination of PKName and Key_Seq). I do not have access to my dev machine to
check why HMS would get the different order of PK columns from what it is
expecting. Below is the code that does that.
[https://github.infra.cloudera.com/CDH/hive/blob/95c21a5a1671b27d55b30d8abe5ad185616a4457/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L4988]
I can take a look at it, but I do not know how soon. If no one has cycles to
check this, I think it is best to revert the change.
> Dataload is failing when USE_CDP_HIVE=true
> --
>
> Key: IMPALA-9188
> URL: https://issues.apache.org/jira/browse/IMPALA-9188
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> When USE_CDP_HIVE=true, Impala builds are failing during dataload when
> creating tables with PK/FK constraints.
> The error is:
> {code:java}
> ERROR: CREATE EXTERNAL TABLE IF NOT EXISTS
> functional_seq_record_snap.child_table (
> seq int, id int, year string, a int, primary key(seq) DISABLE NOVALIDATE
> RELY, foreign key
> (id, year) references functional_seq_record_snap.parent_table(id, year)
> DISABLE NOVALIDATE
> RELY, foreign key(a) references functional_seq_record_snap.parent_table_2(a)
> DISABLE
> NOVALIDATE RELY)
> row format delimited fields terminated by ','
> LOCATION '/test-warehouse/child_table'
> Traceback (most recent call last):
> File "Impala/bin/load-data.py", line 208, in exec_impala_query_from_file
> result = impala_client.execute(query)
> File "Impala/tests/beeswax/impala_beeswax.py", line 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "Impala/tests/beeswax/impala_beeswax.py", line 362, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File "Impala/tests/beeswax/impala_beeswax.py", line 356, in
> execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File "Impala/tests/beeswax/impala_beeswax.py", line 519, in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:
> CAUSED BY: MetaException: Foreign key references id:int;year:string; but no
> corresponding primary key or unique key exists. Possible keys:
> [year:string;id:int;]{code}
> The corresponding error in HMS is:
> {code:java}
> 2019-11-22T06:36:59,937 INFO [pool-10-thread-13] metastore.HiveMetaStore:
> 18: source:127.0.0.1 create_table_req: Table(tableName:child_table,
> dbName:functional_seq_record_gzip, owner:jenkins, createTime:0,
> lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:seq, type:int, comment:null),
> FieldSchema(name:id, type:int, comment:null), FieldSchema(name:year,
> type:string, comment:null), FieldSchema(name:a, type:int, comment:null)],
> location:hdfs://localhost:20500/test-warehouse/child_table,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=,, field.delim=,}), bucketCols:null,
> sortCols:null, parameters:null), partitionKeys:[], parameters:{EXTERNAL=TRUE,
> OBJCAPABILITIES=EXTREAD,EXTWRITE}, viewOriginalText:null,
> viewExpandedText:null, tableType:EXTERNAL_TABLE, catName:hive,
> ownerType:USER, accessType:8)
> 2019-11-22T06:36:59,937 INFO [pool-10-thread-13] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 create_table_req:
> Table(tableName:child_table, dbName:functional_seq_record_gzip,
> owner:jenkins, createTime:0, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:seq, type:int, comment:null),
> FieldSchema(name:id, type:int, comment:null), FieldSchema(name:year,
>
[jira] [Created] (IMPALA-9256) Refactor constraint information into a separate class.
Anurag Mantripragada created IMPALA-9256: Summary: Refactor constraint information into a separate class. Key: IMPALA-9256 URL: https://issues.apache.org/jira/browse/IMPALA-9256 Project: IMPALA Issue Type: Sub-task Components: Frontend Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada We recently added support for primary keys and foreign keys information for tables. However, it is cleaner to have an SQLConstraint class as a container for these constraints just like hive does here: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/constraint/Constraints.java.|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/constraint/Constraints.java] This can be extended to support other kinds of constraints in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9116) SASL server fails when FQDN is greater than 63 characters long in Kudu RPC
[ https://issues.apache.org/jira/browse/IMPALA-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-9116. -- Assignee: Anurag Mantripragada Resolution: Fixed > SASL server fails when FQDN is greater than 63 characters long in Kudu RPC > -- > > Key: IMPALA-9116 > URL: https://issues.apache.org/jira/browse/IMPALA-9116 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > Fix For: Impala 3.4.0 > > > In the current Kudu RPC implementation, we don't explicitly pass the host's > FQDN into the SASL library. Due to an upstream SASL bug > ([https://github.com/cyrusimap/cyrus-sasl/issues/583]) the FQDN gets > truncated when trying to determine the server's principal, in the case that > the server's fQDN is longer than 64 characters. > This results in startup failures where the preflight checks fail due to not > finding the appropriate keytab entry (after searching for a truncated host > name) > To work around this, we should use our own code to compute the FQDN. > Kudu is making the changes in it's own implementation here: > https://issues.apache.org/jira/browse/KUDU-2989, we should do the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9104) Add support for retrieval of primary keys and foreign keys in impala-hs2-server.
[ https://issues.apache.org/jira/browse/IMPALA-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-9104. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed > Add support for retrieval of primary keys and foreign keys in > impala-hs2-server. > > > Key: IMPALA-9104 > URL: https://issues.apache.org/jira/browse/IMPALA-9104 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > Fix For: Impala 3.4.0 > > > Impala's TCLIService.thrift files was taken from Hive-1 days and does not > contain APIs related to primary keys and foreign keys that was added to Hive > jdbc driver start Hive-2.1. We need to port these APIs inorder to be able to > use them in Impala JDBC clients. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9073) Failed test during pre-commit: custom_cluster.test_executor_groups.TestExecutorGroups.test_executor_concurrency
Anurag Mantripragada created IMPALA-9073:
Summary: Failed test during pre-commit:
custom_cluster.test_executor_groups.TestExecutorGroups.test_executor_concurrency
Key: IMPALA-9073
URL: https://issues.apache.org/jira/browse/IMPALA-9073
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Anurag Mantripragada
Attachments: TEST-impala-custom-cluster.log,
TEST-impala-custom-cluster.xml,
impalad.ip-172-31-3-83.ubuntu.log.INFO.20191020-182539.109469
Observed this test failure in pre-commit test of an unrelated change. Looks
like the expected number of concurrent queries reached 4 while it is expected
to be 3 or less.
{code:java}
custom_cluster/test_executor_groups.py:293: in test_executor_concurrency
assert max(num_running) == 3, \
E AssertionError: Unexpected number of running queries: [3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
E assert 4 == 3
E+ where 4 = max([3, 3, 3, 3, 3, 3, ...]) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9073) Failed test during pre-commit: custom_cluster.test_executor_groups.TestExecutorGroups.test_executor_concurrency
[
https://issues.apache.org/jira/browse/IMPALA-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-9073:
-
Description:
Observed this test failure in pre-commit test of an unrelated change. Looks
like the expected number of concurrent queries reached 4 while it is expected
to be 3 or less.
{code:java}
custom_cluster/test_executor_groups.py:293: in test_executor_concurrency
assert max(num_running) == 3, \
E AssertionError: Unexpected number of running queries: [3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
E assert 4 == 3
E+ where 4 = max([3, 3, 3, 3, 3, 3, ...]) {code}
was:
Observed this test failure in pre-commit test of an unrelated change. Looks
like the expected number of concurrent queries reached 4 while it is expected
to be 3 or less.
{code:java}
custom_cluster/test_executor_groups.py:293: in test_executor_concurrency
assert max(num_running) == 3, \
E AssertionError: Unexpected number of running queries: [3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
E assert 4 == 3
E+ where 4 = max([3, 3, 3, 3, 3, 3, ...]) {code}
> Failed test during pre-commit:
> custom_cluster.test_executor_groups.TestExecutorGroups.test_executor_concurrency
> ---
>
> Key: IMPALA-9073
> URL: https://issues.apache.org/jira/browse/IMPALA-9073
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Anurag Mantripragada
>Priority: Major
> Labels: build-failure
> Attachments: TEST-impala-custom-cluster.log,
> TEST-impala-custom-cluster.xml,
> impalad.ip-172-31-3-83.ubuntu.log.INFO.20191020-182539.109469
>
>
> Observed this test failure in pre-commit test of an unrelated change. Looks
> like the expected number of concurrent queries reached 4 while it is expected
> to be 3 or less.
> {code:java}
> custom_cluster/test_executor_groups.py:293: in test_executor_concurrency
> assert max(num_running) == 3, \
> E AssertionError: Unexpected number of running queries: [3, 3, 3, 3, 3, 3,
> 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
> E assert 4 == 3
> E+ where 4 = max([3, 3, 3, 3, 3, 3, ...]) {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9053) DDLs should generate lineage graphs.
Anurag Mantripragada created IMPALA-9053:
Summary: DDLs should generate lineage graphs.
Key: IMPALA-9053
URL: https://issues.apache.org/jira/browse/IMPALA-9053
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Anurag Mantripragada
Assignee: Anurag Mantripragada
{{DDLs like create table should generate minimal lineage for consumers like
Atlas to establish their relationship with the underlying table locations.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9053) DDLs should generate lineage graphs.
[
https://issues.apache.org/jira/browse/IMPALA-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-9053:
-
Description:
{{DDLs like create table should generate minimal lineage for consumers like
Atlas to establish their relationship with the underlying table locations.}}
{{For example, in a create external table DDL, Atlas has no way to establish
lineage between the query and the table created.}}
was:{{DDLs like create table should generate minimal lineage for consumers
like Atlas to establish their relationship with the underlying table
locations.}}
> DDLs should generate lineage graphs.
>
>
> Key: IMPALA-9053
> URL: https://issues.apache.org/jira/browse/IMPALA-9053
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> {{DDLs like create table should generate minimal lineage for consumers like
> Atlas to establish their relationship with the underlying table locations.}}
> {{For example, in a create external table DDL, Atlas has no way to establish
> lineage between the query and the table created.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-2112) Support primary key/foreign key constraint as part of create table in Impala
[
https://issues.apache.org/jira/browse/IMPALA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-2112 started by Anurag Mantripragada.
> Support primary key/foreign key constraint as part of create table in Impala
>
>
> Key: IMPALA-2112
> URL: https://issues.apache.org/jira/browse/IMPALA-2112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog, Frontend
>Affects Versions: Impala 2.2
>Reporter: Marcel Kinard
>Assignee: Anurag Mantripragada
>Priority: Critical
> Labels: planner
>
> These would be advisory, ie, Impala would not attempt to enforce them.
> However, they could be used for cardinality estimation during query planning.
> To be compatible with Hive:
> * We neither enforce or validate integrity constraints. Hence, DISABLE and
> NOVALIDATE options are mandatory.
> * RELY/NORELY is optional. The CBO is expected to use this information when
> a user specifies “RELY”. The default is NORELY.
> * Since we do not yet have UNIQUE in Hive, the FK mentioned must be Primary
> Key column in parent table.
> Support create table syntax like hive does:
> * {{create table pk(id1 integer, id2 integer, }}{{primary key(id1, id2)
> DISABLE NOVALIDATE);}}
> * {{create table fk(id1 integer, id2 integer, }}{{constraint c1 foreign
> key(id1, id2) references pk(id2, id1) DISABLE NOVALIDATE);}}
> * {{create table T1(id integer, name string, primary key(id) DISABLE
> NOVALIDATE RELY}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9095) Alter table events generated by renames are not renaming the table to a different DB.
[
https://issues.apache.org/jira/browse/IMPALA-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada resolved IMPALA-9095.
--
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> Alter table events generated by renames are not renaming the table to a
> different DB.
> -
>
> Key: IMPALA-9095
> URL: https://issues.apache.org/jira/browse/IMPALA-9095
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Critical
> Fix For: Impala 3.4.0
>
>
> Alter table renames was recently refactored. This introduced a bug where
> rename to a different database is not applied correctly.
> Steps to reproduce:
> From Hive:
> {code:java}
> create database bug1;
> create table bug1.foo (id int);
> create database bug2;
> alter table bug1.foo rename to bug2.foo;{code}
>
> From Impala:
> {code:java}
> use bug2;
> show tables;{code}
>
> Expect foo to show up in bug2, it doesn't.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9104) Add support for retrieval of primary keys and foreign keys in HS2.
Anurag Mantripragada created IMPALA-9104: Summary: Add support for retrieval of primary keys and foreign keys in HS2. Key: IMPALA-9104 URL: https://issues.apache.org/jira/browse/IMPALA-9104 Project: IMPALA Issue Type: Sub-task Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada Impala's TCLIService.thrift files was taken from Hive-1 days and does not contain APIs related to primary keys and foreign keys that was added to Hive jdbc driver start Hive-2.1. We need to port these APIs inorder to be able to use them in Impala JDBC clients. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9104) Add support for retrieval of primary keys and foreign keys in impala-hs2-server.
[ https://issues.apache.org/jira/browse/IMPALA-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-9104: - Summary: Add support for retrieval of primary keys and foreign keys in impala-hs2-server. (was: Add support for retrieval of primary keys and foreign keys in HS2.) > Add support for retrieval of primary keys and foreign keys in > impala-hs2-server. > > > Key: IMPALA-9104 > URL: https://issues.apache.org/jira/browse/IMPALA-9104 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > > Impala's TCLIService.thrift files was taken from Hive-1 days and does not > contain APIs related to primary keys and foreign keys that was added to Hive > jdbc driver start Hive-2.1. We need to port these APIs inorder to be able to > use them in Impala JDBC clients. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9070) Include table location information in lineage graphs for external tables.
[ https://issues.apache.org/jira/browse/IMPALA-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-9070. -- Resolution: Fixed > Include table location information in lineage graphs for external tables. > - > > Key: IMPALA-9070 > URL: https://issues.apache.org/jira/browse/IMPALA-9070 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > > Atlas needs table location information to establish lineage between table > location and newly created external tables in Impala. This information should > be sent to backend after successful table creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9070) Include table location information in lineage graphs for external tables.
Anurag Mantripragada created IMPALA-9070: Summary: Include table location information in lineage graphs for external tables. Key: IMPALA-9070 URL: https://issues.apache.org/jira/browse/IMPALA-9070 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada Atlas needs table location information to establish lineage between table location and newly created external tables in Impala. This information should be sent to backend after successful table creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9053) DDLs should generate lineage graphs.
[
https://issues.apache.org/jira/browse/IMPALA-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada resolved IMPALA-9053.
--
Resolution: Fixed
> DDLs should generate lineage graphs.
>
>
> Key: IMPALA-9053
> URL: https://issues.apache.org/jira/browse/IMPALA-9053
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> {{DDLs like create table should generate minimal lineage for consumers like
> Atlas to establish their relationship with the underlying table locations.}}
> {{For example, in a create external table DDL, Atlas has no way to establish
> lineage between the query and the table created.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9095) Alter table events generated by renames are not renaming the table to a different DB.
[ https://issues.apache.org/jira/browse/IMPALA-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-9095: - Priority: Blocker (was: Critical) > Alter table events generated by renames are not renaming the table to a > different DB. > - > > Key: IMPALA-9095 > URL: https://issues.apache.org/jira/browse/IMPALA-9095 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Blocker > > Alter table renames was recently refactored. This introduced a bug where > rename to a different database is not applied correctly. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9095) Alter table events generated by renames are not renaming the table to a different DB.
[ https://issues.apache.org/jira/browse/IMPALA-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-9095: - Priority: Critical (was: Blocker) > Alter table events generated by renames are not renaming the table to a > different DB. > - > > Key: IMPALA-9095 > URL: https://issues.apache.org/jira/browse/IMPALA-9095 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Critical > > Alter table renames was recently refactored. This introduced a bug where > rename to a different database is not applied correctly. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9095) Alter table events generated by renames are not renaming the table to a different DB.
Anurag Mantripragada created IMPALA-9095: Summary: Alter table events generated by renames are not renaming the table to a different DB. Key: IMPALA-9095 URL: https://issues.apache.org/jira/browse/IMPALA-9095 Project: IMPALA Issue Type: Bug Components: Frontend Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada Alter table renames was recently refactored. This introduced a bug where rename to a different database is not applied correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
