[jira] [Updated] (IMPALA-2581) Push down LIMIT past DISTINCT
[ https://issues.apache.org/jira/browse/IMPALA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyao updated IMPALA-2581: --- Labels: backend frontend performance (was: performance) > Push down LIMIT past DISTINCT > -- > > Key: IMPALA-2581 > URL: https://issues.apache.org/jira/browse/IMPALA-2581 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.5.0 >Reporter: Jim Apple >Assignee: liuyao >Priority: Minor > Labels: backend, frontend, performance > Fix For: Impala 4.1.0 > > > In a table t with a column x with no null values, "SELECT DISTINCT x FROM t > LIMIT 1" should be roughly instant. Instead, it finds *all* the distinct > values, then returns one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2581) Push down LIMIT past DISTINCT
[ https://issues.apache.org/jira/browse/IMPALA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyao updated IMPALA-2581: --- Fix Version/s: Impala 4.1.0 > Push down LIMIT past DISTINCT > -- > > Key: IMPALA-2581 > URL: https://issues.apache.org/jira/browse/IMPALA-2581 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.5.0 >Reporter: Jim Apple >Assignee: liuyao >Priority: Minor > Labels: performance > Fix For: Impala 4.1.0 > > > In a table t with a column x with no null values, "SELECT DISTINCT x FROM t > LIMIT 1" should be roughly instant. Instead, it finds *all* the distinct > values, then returns one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10897) TestEventProcessing.test_event_based_replication is flaky
[
https://issues.apache.org/jira/browse/IMPALA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved IMPALA-10897.
--
Fix Version/s: Impala 4.0.1
Resolution: Fixed
This test has been disabled as part of IMPALA-9857
> TestEventProcessing.test_event_based_replication is flaky
> -
>
> Key: IMPALA-10897
> URL: https://issues.apache.org/jira/browse/IMPALA-10897
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Fix For: Impala 4.0.1
>
>
> Saw this in an ASAN build:
> {code:python}
> metadata/test_event_processing.py:185: in test_event_based_replication
> self.__run_event_based_replication_tests()
> metadata/test_event_processing.py:326: in __run_event_based_replication_tests
> EventProcessorUtils.wait_for_event_processing(self)
> util/event_processor_utils.py:61: in wait_for_event_processing
> within {1} seconds".format(current_event_id, timeout))
> E Exception: Event processor did not sync till last known event id 34722
>within 10 seconds {code}
> Standard Error
> {code}
> SET
> client_identifier=metadata/test_event_processing.py::TestEventProcessing::()::test_event_based_replication;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-08-28 23:43:40,300 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-08-28 23:43:40,323 INFO MainThread: Closing active operation
> -- connecting to localhost:11050 with impyla
> -- 2021-08-28 23:43:48,026 INFO MainThread: Waiting until events
> processor syncs to event id:31451
> -- 2021-08-28 23:43:48,759 DEBUGMainThread: Metric last-synced-event-id
> has reached the desired value:31455
> -- 2021-08-28 23:43:48,790 DEBUGMainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-08-28 23:43:48,820 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25000
> -- 2021-08-28 23:43:48,824 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-28 23:43:49,825 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25000
> -- 2021-08-28 23:43:49,829 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-28 23:43:50,830 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25000
> -- 2021-08-28 23:43:50,835 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-28 23:43:51,836 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25000
> -- 2021-08-28 23:43:51,840 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-28 23:43:52,841 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25000
> -- 2021-08-28 23:43:52,846 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2364
> -- 2021-08-28 23:43:52,846 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25001
> -- 2021-08-28 23:43:52,851 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2364
> -- 2021-08-28 23:43:52,851 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-centos74-m5-4xlarge-ondemand-1787.vpc.cloudera.com:25002
> -- 2021-08-28 23:43:52,855 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2364
> -- executing against localhost:21000
> create table repl_source_ugchr.unpart_tbl (a string, b string) stored as
> parquet tblproperties
> ('transactional'='true','transactional_properties'='insert_only');
> -- 2021-08-28 23:43:52,878 INFO MainThread: Started query
> 394339b6db812c59:a5e5039a
> -- executing against localhost:21000
> create table repl_source_ugchr.part_tbl (id int, bool_col boolean,
> tinyint_col tinyint, smallint_col smallint, int_col int, bigint_col bigint,
> float_col float, double_col double, date_string string, string_col string,
> timestamp_col timestamp) partitioned by (year int, month int) stored as
> parquet tblproperties
> ('transactional'='true','transactional_properties'='insert_only');
> -- 2021-08-28 23:43:52,900 INFO MainThread: Started query
> b74f5e32e4c1790a:46410750
> -- executing against localhost:21000
> insert into repl_source_ugchr.unpart_tbl select * from functional.tinytable;
> -- 2021-08-28 23:43:56,132 INFO MainThread: Started query
>
[jira] [Created] (IMPALA-10979) Reduce pre-aggregation memory if cardinality is very low
Csaba Ringhofer created IMPALA-10979: Summary: Reduce pre-aggregation memory if cardinality is very low Key: IMPALA-10979 URL: https://issues.apache.org/jira/browse/IMPALA-10979 Project: IMPALA Issue Type: Bug Components: Backend, Frontend Reporter: Csaba Ringhofer >From an execution summery an aggregation: #Rows Est. #RowsPeak Mem Est. Peak Mem 0 0 34.04 MB 128.00 MB While we knew exactly the 0 rows will arrive (due to scan node child with 0 files) we still allocated 34.04 MB of memory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10978) Add a developer query option MEM_LIMIT_COORDINATOR
Bikramjeet Vig created IMPALA-10978: --- Summary: Add a developer query option MEM_LIMIT_COORDINATOR Key: IMPALA-10978 URL: https://issues.apache.org/jira/browse/IMPALA-10978 Project: IMPALA Issue Type: Task Reporter: Bikramjeet Vig Add a developer query option MEM_LIMIT_COORDINATOR to independently control the mem_limit applied to the coordinator. This would be similar in functionality to MEM_LIMIT_EXECUTORS added in IMPALA-8928 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10972) Adapt Impala's tests to behavior change in HIVE-24920
[
https://issues.apache.org/jira/browse/IMPALA-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-10972:
--
Assignee: Joe McDonnell
> Adapt Impala's tests to behavior change in HIVE-24920
> -
>
> Key: IMPALA-10972
> URL: https://issues.apache.org/jira/browse/IMPALA-10972
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.1.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> In HIVE-24920, the behavior changed so that creating a managed table that
> points to an existing location without specifying the location explicitly
> will give an error.
> This impacts a lot of our tests that use tests/common/file_utils.py's
> create_table_from_file() or create_table_and_copy_files(), causing a lot of
> test issues like this:
> {noformat}
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:
> E CAUSED BY: MetaException: Exception determining external table
> location:Default location is not available for table:
> hdfs://localhost:20500/test-warehouse/test_error_propagation_race_ccd6c80c.db/bad_magic_number{noformat}
> tests/common/file_utils.py populates a table's directory before using a
> normal create table without the location specified. This is now prohibited.
> tests/common/file_utils.py behaves this way as an attempt to make s3 more
> consistent back when it did not have full consistency. See IMPALA-7804:
> [https://github.com/apache/impala/commit/f1a3c47959e17d301f970b7ea2755067d00ae986]
> This is no longer a concern, so these functions should change back to
> something compatible with HIVE-24920.
> Marking this as a blocker, because it prevents moving to a new Hive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10481) Lack of TServer affinity in remote Kudu scans results in bad OS buffer cache behavior on tablet servers
[ https://issues.apache.org/jira/browse/IMPALA-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431415#comment-17431415 ] David Rorke commented on IMPALA-10481: -- KUDU-3248 changed the behavior of the default CLOSEST_REPLICA option to implement deterministic selection from a given client process. This improves affinity/locality and resolves the buffer cache warmup problem without requiring any changes to Impala. Updated results for the TPC-DS query 33 warmup test below showing the impact of the Kudu fix: ||Config||Iteration 1||Iter 2||Iter 3||Iter 4||Iter 5||Iter 6||Iter 7||Iter 8||Iter 9|| |Local|111.4|14.6| | | | | | | | |Remote (default config without KUDU-3248)|110.8|56.9|49.9|43.3|37.3|44.0|20.0|28.9|14.9| |Remote (LEADER_ONLY)|120.1|16.2|15.7|14.2| | | | | | |Remote with KUDU-3248 fix|144.0|23.9|18.8| | | | | | | Also overall benchmark results with a concurrent TPC-DS workload show a major improvement in remote reads from the KUDU-3248 fix, with remote throughput now exceeding local (note the remote configuration used separate 10 node clusters for Kudu and Impala vs a single 10 node cluster in the local case): || ||Local||Remote without KUDU-3248||Remote with KUDU-3248|| |Queries per Hour|378|237|409| > Lack of TServer affinity in remote Kudu scans results in bad OS buffer cache > behavior on tablet servers > --- > > Key: IMPALA-10481 > URL: https://issues.apache.org/jira/browse/IMPALA-10481 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0.0 >Reporter: David Rorke >Priority: Major > Labels: kudu, performance > > Remote Kudu scans can take many iterations against the same scan range before > achieving good performance if the OS buffer cache is initially cold on the > tablet servers. The slow warmup of the buffer cache is exacerbated by the > fact that remote scans in the default Impala config choose a tablet server at > random from the replica candidates. The Kudu client supports a LEADER_ONLY > option that provides hard affinity to the leader replica, and Impala allows > this to be configured using the --pick_only_leaders_for_tests option, but > this is currently considered a testing only option and by default Impala will > connect to a random replica. > The following is a series of iterations of TPC-DS query 33 (times in > seconds), against a freshly started Kudu cluster, in 3 configurations (1) > local reads, with Impala running on Kudu cluster, (2) remote reads from > separate Impala cluster with default config, (3) remote reads with > pick_only_leaders_for_tests=true (LEADER_ONLY affinity) > > ||Config||Iteration 1||Iter 2||Iter 3||Iter 4||Iter 5||Iter 6||Iter 7||Iter > 8||Iter 9|| > |Local|111.4|14.6| | | | | | | | > |Remote (default config)|110.8|56.9|49.9|43.3|37.3|44.0|20.0|28.9|14.9| > |Remote (LEADER_ONLY)|120.1|16.2|15.7|14.2| | | | | | > With pick_only_leaders_for_tests, the remote performance improves quickly, > approaching local performance on the second iteration and warming up fully by > iteration 4. In the default config it takes 9 iterations of the query > before we see the same performance. > Running similar experiments after explicitly dropping the buffer cache on the > tablet servers confirmed that this slow warmup is caused by poor buffer cache > hit rates until the cache is fully warm. > I suspect that slow warmup isn't the only consequence of this. Caching a > given tablet in the buffer cache on multiple tablet servers increases the > overall buffer cache footprint and will increase tserver memory pressure > under load. > We should consider setting the LEADER_ONLY option by default for remote Kudu > reads. The only concern would be that this might result in worse load > balancing and hotspots, in which case Kudu might need to implement some > additional connection option that provides a better mix of affinity and load > balancing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10481) Lack of TServer affinity in remote Kudu scans results in bad OS buffer cache behavior on tablet servers
[ https://issues.apache.org/jira/browse/IMPALA-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Rorke resolved IMPALA-10481. -- Resolution: Fixed > Lack of TServer affinity in remote Kudu scans results in bad OS buffer cache > behavior on tablet servers > --- > > Key: IMPALA-10481 > URL: https://issues.apache.org/jira/browse/IMPALA-10481 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0.0 >Reporter: David Rorke >Priority: Major > Labels: kudu, performance > > Remote Kudu scans can take many iterations against the same scan range before > achieving good performance if the OS buffer cache is initially cold on the > tablet servers. The slow warmup of the buffer cache is exacerbated by the > fact that remote scans in the default Impala config choose a tablet server at > random from the replica candidates. The Kudu client supports a LEADER_ONLY > option that provides hard affinity to the leader replica, and Impala allows > this to be configured using the --pick_only_leaders_for_tests option, but > this is currently considered a testing only option and by default Impala will > connect to a random replica. > The following is a series of iterations of TPC-DS query 33 (times in > seconds), against a freshly started Kudu cluster, in 3 configurations (1) > local reads, with Impala running on Kudu cluster, (2) remote reads from > separate Impala cluster with default config, (3) remote reads with > pick_only_leaders_for_tests=true (LEADER_ONLY affinity) > > ||Config||Iteration 1||Iter 2||Iter 3||Iter 4||Iter 5||Iter 6||Iter 7||Iter > 8||Iter 9|| > |Local|111.4|14.6| | | | | | | | > |Remote (default config)|110.8|56.9|49.9|43.3|37.3|44.0|20.0|28.9|14.9| > |Remote (LEADER_ONLY)|120.1|16.2|15.7|14.2| | | | | | > With pick_only_leaders_for_tests, the remote performance improves quickly, > approaching local performance on the second iteration and warming up fully by > iteration 4. In the default config it takes 9 iterations of the query > before we see the same performance. > Running similar experiments after explicitly dropping the buffer cache on the > tablet servers confirmed that this slow warmup is caused by poor buffer cache > hit rates until the cache is fully warm. > I suspect that slow warmup isn't the only consequence of this. Caching a > given tablet in the buffer cache on multiple tablet servers increases the > overall buffer cache footprint and will increase tserver memory pressure > under load. > We should consider setting the LEADER_ONLY option by default for remote Kudu > reads. The only concern would be that this might result in worse load > balancing and hotspots, in which case Kudu might need to implement some > additional connection option that provides a better mix of affinity and load > balancing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10973) Empty scan nodes are scheduled to the (exclusive) coordinator
[ https://issues.apache.org/jira/browse/IMPALA-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer resolved IMPALA-10973. -- Resolution: Implemented > Empty scan nodes are scheduled to the (exclusive) coordinator > - > > Key: IMPALA-10973 > URL: https://issues.apache.org/jira/browse/IMPALA-10973 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Critical > Labels: scalability, scheduler > > Currently fragments with scan nodes that have no scan ranges are scheduled to > the coordinator, even if it is an exclusive coordinator: > https://github.com/apache/impala/blob/master/be/src/scheduling/scheduler.cc#L805 > As "parent" fragments are often scheduled to be collocated with their > children, the condition of "being scheduled to the coordinator" can spread > through the plan tree. > This can be disastrous to scalability in clusters with lot of executors but > few coordinators and is also very counter-intuitive, as scanning an empty > table shouldn't have a major effect on the query. > > To reproduce locally: > bin/start-impala-cluster.py --use_exclusive_coordinators -c 1 > in Impala shell: > select id from functional.alltypes; > profile; -- scan nodes will be scheduled to 2 hosts > select f2 from functional.emptytable union all select id from > functional.alltypes; > profile; -- scan nodes will be scheduled to 3 hosts -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-10860) Allow setting separate mem_limit for coordinators and executors
[
https://issues.apache.org/jira/browse/IMPALA-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Rawat reopened IMPALA-10860:
-
> Allow setting separate mem_limit for coordinators and executors
> ---
>
> Key: IMPALA-10860
> URL: https://issues.apache.org/jira/browse/IMPALA-10860
> Project: IMPALA
> Issue Type: Task
>Reporter: Abhishek Rawat
>Assignee: Abhishek Rawat
>Priority: Major
>
> The mem_limit query option applies to all impala coordinators and executors.
> This may not be ideal for dedicated coordinators, since they generally need
> less memory per query and having same memory limit will reduce system wide
> query concurrency.
> We could add new query options:
>
> {code:java}
> mem_limit_coordinators
> mem_limit_executors
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10569) Impala should determine Iceberg data file format from Iceberg metadata
[ https://issues.apache.org/jira/browse/IMPALA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431218#comment-17431218 ] Tamas Mate commented on IMPALA-10569: - In the current implementation the IcebergTable encapsulates an HdfsTable while the LocalIcebergTable encapsulates a LocalFsTable. While local catalog mode is active the local catalog could receive the file format from the catalog, but it could not be propagated to the internal HdfsTable and its partition which would need it later. Due to this limitation, to resolve this Jira we would possibly need some refactoring around this area. > Impala should determine Iceberg data file format from Iceberg metadata > -- > > Key: IMPALA-10569 > URL: https://issues.apache.org/jira/browse/IMPALA-10569 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Zoltán Borók-Nagy >Assignee: Tamas Mate >Priority: Major > Labels: impala-iceberg > > When Impala creates an Iceberg table it sets HMS table property > 'iceberg.file_format' to indicate the underlying data file format. > However, when the table was created by Hive or Spark, we don't have this > property and Impala assumes that the data file format is PARQUET. This > assumption is just a wild guess, and when it's wrong Impala raises an error > during query execution. > Instead of only checking the table property, Impala could also try to > determine the file format based on Iceberg metadata. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10569) Impala should determine Iceberg data file format from Iceberg metadata
[ https://issues.apache.org/jira/browse/IMPALA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-10569 started by Tamas Mate. --- > Impala should determine Iceberg data file format from Iceberg metadata > -- > > Key: IMPALA-10569 > URL: https://issues.apache.org/jira/browse/IMPALA-10569 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Zoltán Borók-Nagy >Assignee: Tamas Mate >Priority: Major > Labels: impala-iceberg > > When Impala creates an Iceberg table it sets HMS table property > 'iceberg.file_format' to indicate the underlying data file format. > However, when the table was created by Hive or Spark, we don't have this > property and Impala assumes that the data file format is PARQUET. This > assumption is just a wild guess, and when it's wrong Impala raises an error > during query execution. > Instead of only checking the table property, Impala could also try to > determine the file format based on Iceberg metadata. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-10569) Impala should determine Iceberg data file format from Iceberg metadata
[ https://issues.apache.org/jira/browse/IMPALA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-10569 stopped by Tamas Mate. --- > Impala should determine Iceberg data file format from Iceberg metadata > -- > > Key: IMPALA-10569 > URL: https://issues.apache.org/jira/browse/IMPALA-10569 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Zoltán Borók-Nagy >Assignee: Tamas Mate >Priority: Major > Labels: impala-iceberg > > When Impala creates an Iceberg table it sets HMS table property > 'iceberg.file_format' to indicate the underlying data file format. > However, when the table was created by Hive or Spark, we don't have this > property and Impala assumes that the data file format is PARQUET. This > assumption is just a wild guess, and when it's wrong Impala raises an error > during query execution. > Instead of only checking the table property, Impala could also try to > determine the file format based on Iceberg metadata. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10977) Make zipping unnest work with absolute paths
Gabor Kaszab created IMPALA-10977:
-
Summary: Make zipping unnest work with absolute paths
Key: IMPALA-10977
URL: https://issues.apache.org/jira/browse/IMPALA-10977
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 4.1.0
Reporter: Gabor Kaszab
[IMPALA-10920|https://issues.apache.org/jira/browse/IMPALA-10920] introduced
zipping unnest that has the following syntax:
{code:java}
SELECT arr1.item, arr2.item FROM tbl t, UNNEST(t.arr1, t.arr2)
{code}
However, If the arrays are given with absolute path then the query doesn't work:
{code:java}
SELECT arr1.item, arr2.item FROM UNNEST(tbl.arr1, tbl.arr2)
{code}
The reason partly is because in [SingleNodePlanner.createTableRefNode()|
https://github.com/apache/impala/blob/cd902d8c22d22ea8ebdb85e31ddc143ee0d0bf69/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L2138]
the UNNEST node is only created when relative path is provided. As a result
the memory layout for the tuples related to the unnest won't be computed and a
check will fail.
But even if we create an UNNEST node even for absolute paths the plan tree will
be wrong as there won't be any SUBPLAN and SCAN nodes, so in this case it
should be handled separately to also add that part into the tree not just the
UNNEST node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10925) Improved self event detection for event processor in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sourabh Goyal updated IMPALA-10925: --- Description: h3. Problem Statement Impala catalogd has Events processor which polls metastore events at regular intervals to automatically apply changes to the metadata in the catalogd. However, the current design to detect the self-generated events (DDL/DMLs coming from the same catalogd) have consistency problems which can cause query failures under certain circumstances. h3. Current Design The current design of self-event detection is based on adding markers to the HMS objects which are detected when the event is received later to determine if the event is self-generated or not. These markers constitute a serviceID which is unique to the catalogd instance and a catalog version number which is unique for each catalog object. When a DDL is executed, catalogd adds these as object parameters. When the event is received, Events processor checks the serviceID and if the catalog version of the current object with the same name in the catalogd cache and makes a decision of whether to ignore the event or not. h3. Problems with the current design The approach is problematic under some circumstances where there are conflicting DDLs repeated at a faster interval . For example, consider the following sequence of alter table operations on a table t1 as per HMS 1. alter table t1 from source s1 say another Impala cluster 2. alter table t1 from source s2 say another Hive cluster 3. alter table t1 from local The #3 alter table ddl operation would get reflected in the local cache immediately. However, event processor later on would process events from #1 and #2 above and try to alter the table. In an ideal scenario, these alters should have been applied before #3 (i.e in the same order as that in HMS). This leaves table t1 in an inconsistent state. h3. Proposed Solution The main idea of the solution is to keep track of the last event id for a given table as eventId which the catalogd has synced to in the Table object. The events processor ignores any event whose EVENT_ID is less than or equal to the eventId stored in the table. Once the events processor successfully processes a given event, it updates the value of eventId in the table before releasing the table lock. Also, any DDL or refresh operation on the catalogd (from both catalog metastore server and Impala shell) will follow the following steps to update the event id for the table: 1. Acquire write lock on the table 2. Perform ddl operation in HMS 3. Sync table till the latest event id (as per HMS) since its last synced event id The above steps ensure that any concurrent updates applied on a same db/table from multiple sources like Hive, Impala or say multiple Impala clusters, get reflected in the local catalogd cache (in the same order as they appear in HMS) thus removing any inconsistencies. Also the solution relies on the existing locking mechanism in the catalogd to prevent any other concurrent updates to the table (even via EventsProcessor). In case of database objects, we will also have a similar eventId which represents the events on the database object (CREATE, DROP, ALTER database) and to which the catalogd as synced to. Since there is no refresh database command, catalogOpExecutor will only update the database eventId when there are DDLs at the database level (e.g CREATE, DROP, ALTER database) cc - [~vihangk1] [~kishendas] was: h3. Problem Statement Impala catalogd has Events processor which polls metastore events at regular intervals to automatically apply changes to the metadata in the catalogd. However, the current design to detect the self-generated events (DDL/DMLs coming from the same catalogd) have consistency problems which can cause query failures under certain circumstances. h3. Current Design The current design of self-event detection is based on adding markers to the HMS objects which are detected when the event is received later to determine if the event is self-generated or not. These markers constitute a serviceID which is unique to the catalogd instance and a catalog version number which is unique for each catalog object. When a DDL is executed, catalogd adds these as object parameters. When the event is received, Events processor checks the serviceID and if the catalog version of the current object with the same name in the catalogd cache and makes a decision of whether to ignore the event or not. h3. Problems with the current design The approach is problematic under some circumstances where there are conflicting DDLs repeated at a faster interval. For example, a sequence of create/drop table DDLs will generate CREATE_TABLE and DROP_TABLE events. When the events are received, it is possible that the CREATE_TABLE event is
[jira] [Commented] (IMPALA-10925) Improved self event detection for event processor in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431135#comment-17431135 ] Sourabh Goyal commented on IMPALA-10925: Thanks for the feedback [~vihangk1]. I have updated the description. > Improved self event detection for event processor in catalogd > -- > > Key: IMPALA-10925 > URL: https://issues.apache.org/jira/browse/IMPALA-10925 > Project: IMPALA > Issue Type: Epic > Components: Catalog >Reporter: Sourabh Goyal >Assignee: Sourabh Goyal >Priority: Major > > h3. Problem Statement > Impala catalogd has Events processor which polls metastore events at regular > intervals to automatically apply changes to the metadata in the catalogd. > However, the current design to detect the self-generated events (DDL/DMLs > coming from the same catalogd) have consistency problems which can cause > query failures under certain circumstances. > > h3. Current Design > The current design of self-event detection is based on adding markers to the > HMS objects which are detected when the event is received later to determine > if the event is self-generated or not. These markers constitute a serviceID > which is unique to the catalogd instance and a catalog version number which > is unique for each catalog object. When a DDL is executed, catalogd adds > these as object parameters. When the event is received, Events processor > checks the serviceID and if the catalog version of the current object with > the same name in the catalogd cache and makes a decision of whether to ignore > the event or not. > > h3. Problems with the current design > The approach is problematic under some circumstances where there are > conflicting DDLs repeated at a faster interval . For example, consider the > following sequence of alter table operations on a table t1 as per HMS > 1. alter table t1 from source s1 say another Impala cluster > 2. alter table t1 from source s2 say another Hive cluster > 3. alter table t1 from local > The #3 alter table ddl operation would get reflected in the local cache > immediately. However, event processor later on would process events from #1 > and #2 above and try to alter the table. In an ideal scenario, these alters > should have been applied before #3 (i.e in the same order as that in HMS). > This leaves table t1 in an inconsistent state. > h3. Proposed Solution > The main idea of the solution is to keep track of the last event id for a > given table as eventId which the catalogd has synced to in the Table object. > The events processor ignores any event whose EVENT_ID is less than or equal > to the eventId stored in the table. Once the events processor successfully > processes a given event, it updates the value of eventId in the table before > releasing the table lock. Also, any DDL or refresh operation on the catalogd > (from both catalog metastore server and Impala shell) will follow the > following steps to update the event id for the table: > 1. Acquire write lock on the table > 2. Perform ddl operation in HMS > 3. Sync table till the latest event id (as per HMS) since its last synced > event id > The above steps ensure that any concurrent updates applied on a same db/table > from multiple sources like Hive, Impala or say multiple Impala clusters, get > reflected in the local catalogd cache (in the same order as they appear in > HMS) thus removing any inconsistencies. > Also the solution relies on the existing locking mechanism in the catalogd to > prevent any other concurrent updates to the table (even via EventsProcessor). > In case of database objects, we will also have a similar eventId which > represents the events on the database object (CREATE, DROP, ALTER database) > and to which the catalogd as synced to. Since there is no refresh database > command, catalogOpExecutor will only update the database eventId when there > are DDLs at the database level (e.g CREATE, DROP, ALTER database) > > cc - [~vihangk1] [~kishendas] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10976) Sync db/table to latest event id for all ddls in catalogOpExecutor
Sourabh Goyal created IMPALA-10976: -- Summary: Sync db/table to latest event id for all ddls in catalogOpExecutor Key: IMPALA-10976 URL: https://issues.apache.org/jira/browse/IMPALA-10976 Project: IMPALA Issue Type: Task Components: Catalog, Frontend Reporter: Sourabh Goyal Assignee: Sourabh Goyal -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10975) Minor refactoring in alter table DDL operation in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431106#comment-17431106 ] Sourabh Goyal commented on IMPALA-10975: Patch up for review: https://gerrit.cloudera.org/c/17919/ > Minor refactoring in alter table DDL operation in catalogd > -- > > Key: IMPALA-10975 > URL: https://issues.apache.org/jira/browse/IMPALA-10975 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Frontend >Reporter: Sourabh Goyal >Assignee: Sourabh Goyal >Priority: Major > > For almost all alter table DDL operations in catalogOpExecutor, we add table > to catalog update if reloadMetadata is true in the end. However for certain > sub ddl operations like ADD and DROP partitions, the table to update catalog > is performed locally. This Jira is to refactor addTableToCatalogUpdate() and > call it from one place for all the sub ddls. This refactoring would be > helpful when we introduce code changes of syncing db/table to latest event id > from catalogOpExecutor in future. > > cc - [~vihangk1] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10975) Minor refactoring in alter table DDL operation in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-10975 started by Sourabh Goyal. -- > Minor refactoring in alter table DDL operation in catalogd > -- > > Key: IMPALA-10975 > URL: https://issues.apache.org/jira/browse/IMPALA-10975 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Frontend >Reporter: Sourabh Goyal >Assignee: Sourabh Goyal >Priority: Major > > For almost all alter table DDL operations in catalogOpExecutor, we add table > to catalog update if reloadMetadata is true in the end. However for certain > sub ddl operations like ADD and DROP partitions, the table to update catalog > is performed locally. This Jira is to refactor addTableToCatalogUpdate() and > call it from one place for all the sub ddls. This refactoring would be > helpful when we introduce code changes of syncing db/table to latest event id > from catalogOpExecutor in future. > > cc - [~vihangk1] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10975) Minor refactoring in alter table DDL operation in catalogd
Sourabh Goyal created IMPALA-10975: -- Summary: Minor refactoring in alter table DDL operation in catalogd Key: IMPALA-10975 URL: https://issues.apache.org/jira/browse/IMPALA-10975 Project: IMPALA Issue Type: Improvement Components: Catalog, Frontend Reporter: Sourabh Goyal Assignee: Sourabh Goyal For almost all alter table DDL operations in catalogOpExecutor, we add table to catalog update if reloadMetadata is true in the end. However for certain sub ddl operations like ADD and DROP partitions, the table to update catalog is performed locally. This Jira is to refactor addTableToCatalogUpdate() and call it from one place for all the sub ddls. This refactoring would be helpful when we introduce code changes of syncing db/table to latest event id from catalogOpExecutor in future. cc - [~vihangk1] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10566) Change the default value of FETCH_ROWS_TIMEOUT_MS to 0
[ https://issues.apache.org/jira/browse/IMPALA-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431078#comment-17431078 ] Zoltán Borók-Nagy commented on IMPALA-10566: [~Yuchen Fan] what client do you use to connect to Impala? You might need to consider upgrading your ODBC/JDBC driver. > Change the default value of FETCH_ROWS_TIMEOUT_MS to 0 > -- > > Key: IMPALA-10566 > URL: https://issues.apache.org/jira/browse/IMPALA-10566 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Aman Sinha >Assignee: Zoltán Borók-Nagy >Priority: Major > > The current default value of FETCH_ROWS_TIMEOUT_MS is 10 secs. This was done > in IMPALA-7312 and IMPALA-8962 to introduce a non-blocking fetch() api > behavior such that clients are not blocked indefinitely. In some cases, > especially with the supported JDBC/ODBC drivers, this can cause a regression > by returning either empty or partial results to the client based on the > following sequence: > * Client starts to fetch rows > * Impala is unable to produce rows in 10s. so to not make the client > block, Impala returns an empty result set with hasMoreRows=true. > * Query’s state can be either RUNNING or FINISHED > * Client sees empty result set, ignores hasMoreRows=true > * Client closes the query thinking it got the whole result set > This issue was observed in internal testing and is also reported in the > following JIRA: > https://issues.apache.org/jira/browse/IMPALA-8962?focusedCommentId=17240474=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17240474 > In contrast, impala-shell works correctly and does not have this partial > results problem. > Since this issue impacts various drivers, it is best to change the default > value of FETCH_ROWS_TIMEOUT_MS to 0 to revert to the blocking behavior. > Users can opt-in by changing the value. In the meantime, driver programs > also need to be updated to allow the non-blocking fetch behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
