[jira] [Updated] (IMPALA-13035) Querying metadata tables from non-Iceberg tables throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/IMPALA-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13035:
Fix Version/s: Impala 4.5.0
> Querying metadata tables from non-Iceberg tables throws
> IllegalArgumentException
>
>
> Key: IMPALA-13035
> URL: https://issues.apache.org/jira/browse/IMPALA-13035
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.3.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
> Fix For: Impala 4.5.0
>
>
> If a query targets an Iceberg metadata table like default.xy.`files` and the
> xy table is not an Iceberg table then the analyzer throws
> IllegalArgumentException.
> The main concern is that IcebergMetadataTable.java:isIcebergMetadataTable is
> called before it's validated that the table is indeed an IcebergTable.
> Example:
> {code:java}
> create table xy(a int);
> select * from default.xy.`files`;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13009) Potential leak of partition deletions in the catalog topic
[
https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-13009.
-
Fix Version/s: Impala 4.5.0
Resolution: Fixed
> Potential leak of partition deletions in the catalog topic
> --
>
> Key: IMPALA-13009
> URL: https://issues.apache.org/jira/browse/IMPALA-13009
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1,
> Impala 4.1.2, Impala 4.3.0
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
> Fix For: Impala 4.5.0
>
>
> Catalogd might not send partition deletions to the catalog topic in the
> following scenario:
> * Some partitions of a table are dropped.
> * The HdfsTable object is removed sequentially before catalogd collects the
> dropped partitions.
> In such case, catalogd loses track of the dropped partitions so their updates
> keep existing in the catalog topic, until the partition names are reused
> again.
> Note that the HdfsTable object can be removed by commands like DropTable or
> INVALIDATE.
> The leaked partitions will be detected when a coordinator restarts. An
> IllegalStateException complaining stale partitions will be reported, causing
> the table not being added to the catalog cache of coordinator.
> {noformat}
> E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
> object: Received stale partition in a statestore update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> Java exception follows:
> java.lang.IllegalStateException: Received stale partition in a statestore
> update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:512)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
> at
> org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
> at
>
[jira] [Commented] (IMPALA-13058) TestRuntimeFilters.test_basic_filters failed on ARM builds
[
https://issues.apache.org/jira/browse/IMPALA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844132#comment-17844132
]
Wenzhe Zhou commented on IMPALA-13058:
--
Thanks Riza to look in this. The unit-test was very old test. It failed only
for ARM builds, which were added to nightly builds since last week.
> TestRuntimeFilters.test_basic_filters failed on ARM builds
> --
>
> Key: IMPALA-13058
> URL: https://issues.apache.org/jira/browse/IMPALA-13058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Wenzhe Zhou
>Assignee: Riza Suminto
>Priority: Major
>
> Stacktrace:
> query_test/test_runtime_filters.py:98: in test_basic_filters
> self.run_test_case('QueryTest/runtime_filters', vector,
> test_file_vars=vars)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/impala_test_suite.py:851:
> in run_test_case
> update_section=pytest.config.option.update_results)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/test_result_verifier.py:701:
> in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*REMOTE.*ms.*ms.*true
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13009) Potential leak of partition deletions in the catalog topic
[
https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844099#comment-17844099
]
ASF subversion and git services commented on IMPALA-13009:
--
Commit ee21427d26620b40d38c706b4944d2831f84f6f5 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee21427d2 ]
IMPALA-13009: Fix catalogd not sending deletion updates for some dropped
partitions
*Background*
Since IMPALA-3127, catalogd sends incremental partition updates based on
the last sent table snapshot ('maxSentPartitionId_' to be specific).
Dropped partitions since the last catalog update are tracked in
'droppedPartitions_' of HdfsTable. When catalogd collects the next
catalog update, they will be collected. HdfsTable then clears the set.
See details in CatalogServiceCatalog#addHdfsPartitionsToCatalogDelta().
If an HdfsTable is invalidated, it's replaced with an IncompleteTable
which doesn't track any partitions. The HdfsTable object is then added
to the deleteLog so catalogd can send deletion updates for all its
partitions. The same if the HdfsTable is dropped. However, the
previously dropped partitions are not collected in this case, which
results in a leak in the catalog topic if the partition name is not
reused anymore. Note that in the catalog topic, the key of a partition
update consists of the table name and the partition name. So if the
partition is added back to the table, the topic key will be reused then
resolves the leak.
The leak will be observed when a coordinator restarts. In the initial
catalog update sent from statestore, coordinator will find some
partition updates that are not referenced by the HdfsTable (assuming the
table is used again after the INVALIDATE). Then a Precondition check
fails and the table is not added to the coordinator.
*Overview of the patch*
This patch fixes the leak by also collecting the dropped partitions when
adding the HdfsTable to the deleteLog. A new field, dropped_partitions,
is added in THdfsTable to collect them. It's only used when catalogd
collects catalog updates.
Removes the Precondition check in coordinator and just reports the stale
partitions since IMPALA-12831 could also introduce them.
Also adds a log line in CatalogOpExecutor.alterTableDropPartition() to
show the dropped partition names for better diagnostics.
Tests
- Added e2e tests
Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21
Reviewed-on: http://gerrit.cloudera.org:8080/21326
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Potential leak of partition deletions in the catalog topic
> --
>
> Key: IMPALA-13009
> URL: https://issues.apache.org/jira/browse/IMPALA-13009
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1,
> Impala 4.1.2, Impala 4.3.0
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
>
> Catalogd might not send partition deletions to the catalog topic in the
> following scenario:
> * Some partitions of a table are dropped.
> * The HdfsTable object is removed sequentially before catalogd collects the
> dropped partitions.
> In such case, catalogd loses track of the dropped partitions so their updates
> keep existing in the catalog topic, until the partition names are reused
> again.
> Note that the HdfsTable object can be removed by commands like DropTable or
> INVALIDATE.
> The leaked partitions will be detected when a coordinator restarts. An
> IllegalStateException complaining stale partitions will be reported, causing
> the table not being added to the catalog cache of coordinator.
> {noformat}
> E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
> object: Received stale partition in a statestore update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
>
[jira] [Commented] (IMPALA-12831) HdfsTable.toMinimalTCatalogObject() should hold table read lock to generate incremental updates
[
https://issues.apache.org/jira/browse/IMPALA-12831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844101#comment-17844101
]
ASF subversion and git services commented on IMPALA-12831:
--
Commit ee21427d26620b40d38c706b4944d2831f84f6f5 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee21427d2 ]
IMPALA-13009: Fix catalogd not sending deletion updates for some dropped
partitions
*Background*
Since IMPALA-3127, catalogd sends incremental partition updates based on
the last sent table snapshot ('maxSentPartitionId_' to be specific).
Dropped partitions since the last catalog update are tracked in
'droppedPartitions_' of HdfsTable. When catalogd collects the next
catalog update, they will be collected. HdfsTable then clears the set.
See details in CatalogServiceCatalog#addHdfsPartitionsToCatalogDelta().
If an HdfsTable is invalidated, it's replaced with an IncompleteTable
which doesn't track any partitions. The HdfsTable object is then added
to the deleteLog so catalogd can send deletion updates for all its
partitions. The same if the HdfsTable is dropped. However, the
previously dropped partitions are not collected in this case, which
results in a leak in the catalog topic if the partition name is not
reused anymore. Note that in the catalog topic, the key of a partition
update consists of the table name and the partition name. So if the
partition is added back to the table, the topic key will be reused then
resolves the leak.
The leak will be observed when a coordinator restarts. In the initial
catalog update sent from statestore, coordinator will find some
partition updates that are not referenced by the HdfsTable (assuming the
table is used again after the INVALIDATE). Then a Precondition check
fails and the table is not added to the coordinator.
*Overview of the patch*
This patch fixes the leak by also collecting the dropped partitions when
adding the HdfsTable to the deleteLog. A new field, dropped_partitions,
is added in THdfsTable to collect them. It's only used when catalogd
collects catalog updates.
Removes the Precondition check in coordinator and just reports the stale
partitions since IMPALA-12831 could also introduce them.
Also adds a log line in CatalogOpExecutor.alterTableDropPartition() to
show the dropped partition names for better diagnostics.
Tests
- Added e2e tests
Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21
Reviewed-on: http://gerrit.cloudera.org:8080/21326
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> HdfsTable.toMinimalTCatalogObject() should hold table read lock to generate
> incremental updates
> ---
>
> Key: IMPALA-12831
> URL: https://issues.apache.org/jira/browse/IMPALA-12831
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1,
> Impala 4.1.2, Impala 4.3.0
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
> Fix For: Impala 4.4.0
>
>
> When enable_incremental_metadata_updates=true (default), catalogd sends
> incremental partition updates to coordinators, which goes into
> HdfsTable.toMinimalTCatalogObject():
> {code:java}
> public TCatalogObject toMinimalTCatalogObject() {
> TCatalogObject catalogObject = super.toMinimalTCatalogObject();
> if (!BackendConfig.INSTANCE.isIncrementalMetadataUpdatesEnabled()) {
> return catalogObject;
> }
> catalogObject.getTable().setTable_type(TTableType.HDFS_TABLE);
> THdfsTable hdfsTable = new THdfsTable(hdfsBaseDir_, getColumnNames(),
> nullPartitionKeyValue_, nullColumnValue_,
> /*idToPartition=*/ new HashMap<>(),
> /*prototypePartition=*/ new THdfsPartition());
> for (HdfsPartition part : partitionMap_.values()) {
> hdfsTable.partitions.put(part.getId(), part.toMinimalTHdfsPartition());
> }
> hdfsTable.setHas_full_partitions(false);
> // The minimal catalog object of partitions contain the partition names.
> hdfsTable.setHas_partition_names(true);
> catalogObject.getTable().setHdfs_table(hdfsTable);
> return catalogObject;
> }{code}
> Accessing table fields without holding the table read lock might be failed by
> concurrent DDLs. All workloads that use this method (e.g. INVALIDATE
> commands) could hit this issue. We've saw event-processor failed in
> processing a RELOAD event that want to invalidates an HdfsTable:
> {noformat}
> E0216 16:23:44.283689 253 MetastoreEventsProcessor.java:899] Unexpected
> exception received while processing event
> Java exception follows:
> java.util.ConcurrentModificationException
> at
[jira] [Commented] (IMPALA-3127) Decouple partitions from tables
[
https://issues.apache.org/jira/browse/IMPALA-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844100#comment-17844100
]
ASF subversion and git services commented on IMPALA-3127:
-
Commit ee21427d26620b40d38c706b4944d2831f84f6f5 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee21427d2 ]
IMPALA-13009: Fix catalogd not sending deletion updates for some dropped
partitions
*Background*
Since IMPALA-3127, catalogd sends incremental partition updates based on
the last sent table snapshot ('maxSentPartitionId_' to be specific).
Dropped partitions since the last catalog update are tracked in
'droppedPartitions_' of HdfsTable. When catalogd collects the next
catalog update, they will be collected. HdfsTable then clears the set.
See details in CatalogServiceCatalog#addHdfsPartitionsToCatalogDelta().
If an HdfsTable is invalidated, it's replaced with an IncompleteTable
which doesn't track any partitions. The HdfsTable object is then added
to the deleteLog so catalogd can send deletion updates for all its
partitions. The same if the HdfsTable is dropped. However, the
previously dropped partitions are not collected in this case, which
results in a leak in the catalog topic if the partition name is not
reused anymore. Note that in the catalog topic, the key of a partition
update consists of the table name and the partition name. So if the
partition is added back to the table, the topic key will be reused then
resolves the leak.
The leak will be observed when a coordinator restarts. In the initial
catalog update sent from statestore, coordinator will find some
partition updates that are not referenced by the HdfsTable (assuming the
table is used again after the INVALIDATE). Then a Precondition check
fails and the table is not added to the coordinator.
*Overview of the patch*
This patch fixes the leak by also collecting the dropped partitions when
adding the HdfsTable to the deleteLog. A new field, dropped_partitions,
is added in THdfsTable to collect them. It's only used when catalogd
collects catalog updates.
Removes the Precondition check in coordinator and just reports the stale
partitions since IMPALA-12831 could also introduce them.
Also adds a log line in CatalogOpExecutor.alterTableDropPartition() to
show the dropped partition names for better diagnostics.
Tests
- Added e2e tests
Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21
Reviewed-on: http://gerrit.cloudera.org:8080/21326
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Decouple partitions from tables
> ---
>
> Key: IMPALA-3127
> URL: https://issues.apache.org/jira/browse/IMPALA-3127
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Affects Versions: Impala 2.2.4
>Reporter: Dimitris Tsirogiannis
>Assignee: Quanlong Huang
>Priority: Major
> Labels: catalog-server, performance
> Fix For: Impala 4.0.0
>
>
> Currently, partitions are tightly integrated into the HdfsTable objects,
> making incremental metadata updates difficult to perform. Furthermore, the
> catalog transmits entire table metadata even when only few partitions change,
> introducing significant latencies, wasting network bandwidth and CPU cycles
> while updating table metadata at the receiving impalads. As a first step, we
> should decouple partitions from tables and add them as a separate level in
> the hierarchy of catalog entities (server-db-table-partition). Subsequently,
> the catalog should transmit only entities that have changed after DDL/DML
> statements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13058) TestRuntimeFilters.test_basic_filters failed on ARM builds
[
https://issues.apache.org/jira/browse/IMPALA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844060#comment-17844060
]
Riza Suminto commented on IMPALA-13058:
---
This seems unrelated to runtime filter. It is strange that Query Compilation:
2.858ms, but Planning finished: 0.000ns.
{code:java}
E Query Compilation: 2.858ms
E - Metadata of all 2 tables cached: 274.516us (274.516us)
E - Analysis finished: 543.727us (269.211us)
E - Authorization finished (noop): 599.562us (55.835us)
E - Value transfer graph computed: 665.553us (65.991us)
E - Single node plan created: 1.105ms (439.572us)
E - Runtime filters computed: 1.748ms (643.246us)
E - Distributed plan created: 1.768ms (20.414us)
E - Parallel plans created: 1.782ms (13.834us)
E - Planning finished: 2.858ms (1.075ms)
E Query Timeline: 50.001ms
E - Query submitted: 0.000ns (0.000ns)
E - Planning finished: 0.000ns (0.000ns)
E - Submit for admission: 0.000ns (0.000ns)
E - Completed admission: 0.000ns (0.000ns)
E - Ready to start on 3 backends: 0.000ns (0.000ns)
E - All 3 execution backends (13 fragment instances) started: 0.000ns
(0.000ns)
E - First dynamic filter received: 0.000ns (0.000ns)
E - Rows available: 10.000ms (10.000ms)
E - First row fetched: 50.001ms (40.000ms)
E - Last row fetched: 50.001ms (0.000ns)
E - Released admission control resources: 50.001ms (0.000ns)
E - AdmissionControlTimeSinceLastUpdate: 26.000ms
E - ComputeScanRangeAssignmentTimer: 0.000ns {code}
> TestRuntimeFilters.test_basic_filters failed on ARM builds
> --
>
> Key: IMPALA-13058
> URL: https://issues.apache.org/jira/browse/IMPALA-13058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Wenzhe Zhou
>Assignee: Riza Suminto
>Priority: Major
>
> Stacktrace:
> query_test/test_runtime_filters.py:98: in test_basic_filters
> self.run_test_case('QueryTest/runtime_filters', vector,
> test_file_vars=vars)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/impala_test_suite.py:851:
> in run_test_case
> update_section=pytest.config.option.update_results)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/test_result_verifier.py:701:
> in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*REMOTE.*ms.*ms.*true
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13051) Speed up test_query_log test runs
[ https://issues.apache.org/jira/browse/IMPALA-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr reassigned IMPALA-13051: --- Assignee: Jason Fehr (was: Michael Smith) > Speed up test_query_log test runs > - > > Key: IMPALA-13051 > URL: https://issues.apache.org/jira/browse/IMPALA-13051 > Project: IMPALA > Issue Type: Task >Affects Versions: Impala 4.4.0 >Reporter: Michael Smith >Assignee: Jason Fehr >Priority: Minor > > test_query_log.py takes 11 minutes to run. Most of them use graceful > shutdown, and provide an unnecessary grace period. Optimize test_query_log > test runs, and do some other code cleanup around workload management. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13061) Query Live table fails to load if default_transactional_type=insert_only set globally
Michael Smith created IMPALA-13061:
--
Summary: Query Live table fails to load if
default_transactional_type=insert_only set globally
Key: IMPALA-13061
URL: https://issues.apache.org/jira/browse/IMPALA-13061
Project: IMPALA
Issue Type: Bug
Reporter: Michael Smith
If transactional type defaults to insert_only for all queries via
{code}
--default_query_options=default_transactional_type=insert_only
{code}
the table definition for {{sys.impala_query_live}} is set to transactional,
which causes an exception in catalogd
{code}
I0506 22:07:42.808758 3972 jni-util.cc:302] 4547b965aeebc5f0:8ba96c58]
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:496)
at org.apache.impala.catalog.Table.getPartialInfo(Table.java:851)
at
org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3818)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3714)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3681)
at
org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:431)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
at
org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:253)
at
org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:430)
{code}
We need to override that setting while creating {{sys.impala_query_live}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13061) Query Live table fails to load if default_transactional_type=insert_only set globally
[
https://issues.apache.org/jira/browse/IMPALA-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith reassigned IMPALA-13061:
--
Assignee: Michael Smith
> Query Live table fails to load if default_transactional_type=insert_only set
> globally
> -
>
> Key: IMPALA-13061
> URL: https://issues.apache.org/jira/browse/IMPALA-13061
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
>
> If transactional type defaults to insert_only for all queries via
> {code}
> --default_query_options=default_transactional_type=insert_only
> {code}
> the table definition for {{sys.impala_query_live}} is set to transactional,
> which causes an exception in catalogd
> {code}
> I0506 22:07:42.808758 3972 jni-util.cc:302]
> 4547b965aeebc5f0:8ba96c58] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at org.apache.impala.catalog.Table.getPartialInfo(Table.java:851)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3818)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3714)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3681)
> at
> org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:431)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
> at
> org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:253)
> at
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:430)
> {code}
> We need to override that setting while creating {{sys.impala_query_live}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13061) Query Live table fails to load if default_transactional_type=insert_only set globally
[
https://issues.apache.org/jira/browse/IMPALA-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-13061:
---
Epic Link: IMPALA-12427
> Query Live table fails to load if default_transactional_type=insert_only set
> globally
> -
>
> Key: IMPALA-13061
> URL: https://issues.apache.org/jira/browse/IMPALA-13061
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Critical
>
> If transactional type defaults to insert_only for all queries via
> {code}
> --default_query_options=default_transactional_type=insert_only
> {code}
> the table definition for {{sys.impala_query_live}} is set to transactional,
> which causes an exception in catalogd
> {code}
> I0506 22:07:42.808758 3972 jni-util.cc:302]
> 4547b965aeebc5f0:8ba96c58] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at org.apache.impala.catalog.Table.getPartialInfo(Table.java:851)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3818)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3714)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3681)
> at
> org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:431)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
> at
> org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:253)
> at
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:430)
> {code}
> We need to override that setting while creating {{sys.impala_query_live}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13061) Query Live table fails to load if default_transactional_type=insert_only set globally
[
https://issues.apache.org/jira/browse/IMPALA-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13061 started by Michael Smith.
--
> Query Live table fails to load if default_transactional_type=insert_only set
> globally
> -
>
> Key: IMPALA-13061
> URL: https://issues.apache.org/jira/browse/IMPALA-13061
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Critical
>
> If transactional type defaults to insert_only for all queries via
> {code}
> --default_query_options=default_transactional_type=insert_only
> {code}
> the table definition for {{sys.impala_query_live}} is set to transactional,
> which causes an exception in catalogd
> {code}
> I0506 22:07:42.808758 3972 jni-util.cc:302]
> 4547b965aeebc5f0:8ba96c58] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at org.apache.impala.catalog.Table.getPartialInfo(Table.java:851)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3818)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3714)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3681)
> at
> org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:431)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
> at
> org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:253)
> at
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:430)
> {code}
> We need to override that setting while creating {{sys.impala_query_live}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13061) Query Live table fails to load if default_transactional_type=insert_only set globally
[
https://issues.apache.org/jira/browse/IMPALA-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-13061:
---
Priority: Critical (was: Major)
> Query Live table fails to load if default_transactional_type=insert_only set
> globally
> -
>
> Key: IMPALA-13061
> URL: https://issues.apache.org/jira/browse/IMPALA-13061
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Critical
>
> If transactional type defaults to insert_only for all queries via
> {code}
> --default_query_options=default_transactional_type=insert_only
> {code}
> the table definition for {{sys.impala_query_live}} is set to transactional,
> which causes an exception in catalogd
> {code}
> I0506 22:07:42.808758 3972 jni-util.cc:302]
> 4547b965aeebc5f0:8ba96c58] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at org.apache.impala.catalog.Table.getPartialInfo(Table.java:851)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3818)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3714)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3681)
> at
> org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:431)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
> at
> org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:253)
> at
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:430)
> {code}
> We need to override that setting while creating {{sys.impala_query_live}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reassigned IMPALA-13034: --- Assignee: Quanlong Huang > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12795) TestWebPage.test_catalog_operation_fields is flaky
[
https://issues.apache.org/jira/browse/IMPALA-12795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-12795.
-
Fix Version/s: Impala 4.4.0
Resolution: Fixed
> TestWebPage.test_catalog_operation_fields is flaky
> --
>
> Key: IMPALA-12795
> URL: https://issues.apache.org/jira/browse/IMPALA-12795
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
> Fix For: Impala 4.4.0
>
>
> Saw the test failed in an internal job:
> {noformat}
> webserver/test_web_pages.py:942: in test_catalog_operation_fields
> assert matched
> E assert False{noformat}
> That means the CREATE DATABASE statement was not found in the coordinator
> webUI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12951) Investigate RETRY_FAILED_QUERIES with System Tables
[
https://issues.apache.org/jira/browse/IMPALA-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843919#comment-17843919
]
Michael Smith commented on IMPALA-12951:
Impala has a general query retry mechanism. It's not used for inserts. I
believe this problem only applies to system tables, but not 100% sure.
> Investigate RETRY_FAILED_QUERIES with System Tables
> ---
>
> Key: IMPALA-12951
> URL: https://issues.apache.org/jira/browse/IMPALA-12951
> Project: IMPALA
> Issue Type: Task
>Reporter: Michael Smith
>Priority: Major
>
> My theory is that {{RETRY_FAILED_QUERIES}} doesn't work well with querying
> System Tables because coordinators don't get added to a blacklist when they
> fail, so it reschedules them on the same set of nodes.
> Look into whether issues with {{RETRY_FAILED_QUERIES}} when a coordinator
> restarts (without graceful shutdown) are unique to coordinators, and how to
> improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12951) Investigate RETRY_FAILED_QUERIES with System Tables
[
https://issues.apache.org/jira/browse/IMPALA-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843918#comment-17843918
]
Jason Fehr commented on IMPALA-12951:
-
Is this issue limited to clients when they are reading from the system tables
or does it also cover situations where inserting into the sys.impala_query_log
table fails?
> Investigate RETRY_FAILED_QUERIES with System Tables
> ---
>
> Key: IMPALA-12951
> URL: https://issues.apache.org/jira/browse/IMPALA-12951
> Project: IMPALA
> Issue Type: Task
>Reporter: Michael Smith
>Priority: Major
>
> My theory is that {{RETRY_FAILED_QUERIES}} doesn't work well with querying
> System Tables because coordinators don't get added to a blacklist when they
> fail, so it reschedules them on the same set of nodes.
> Look into whether issues with {{RETRY_FAILED_QUERIES}} when a coordinator
> restarts (without graceful shutdown) are unique to coordinators, and how to
> improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13060) TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator flaky
Wenzhe Zhou created IMPALA-13060:
Summary:
TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator
flaky
Key: IMPALA-13060
URL: https://issues.apache.org/jira/browse/IMPALA-13060
Project: IMPALA
Issue Type: Bug
Reporter: Wenzhe Zhou
Assignee: Jason Fehr
The unit-test failed since the expected message "'Query successfully
unregistered: query_id=554dcf86d11dbd5f:0ea9f28d'" was not found in the
log file.
Stacktrace:
custom_cluster/test_query_log.py:414: in
test_query_log_table_query_select_dedicate_coordinator
client = self.get_client(vector.get_value('protocol'))
custom_cluster/test_query_log.py:73: in get_client
self.assert_impalad_log_contains("INFO", finish_re)
common/impala_test_suite.py:1271: in assert_impalad_log_contains
"impalad", level, line_regex, expected_count, timeout_s)
common/impala_test_suite.py:1322: in assert_log_contains
(expected_count, log_file_path, line_regex, found, line)
E AssertionError: Expected 1 lines in file
/data0/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-04d5.vpc.cloudera.com.jenkins.log.INFO.20240506-052112.12754
matching regex 'Query successfully unregistered:
query_id=554dcf86d11dbd5f:0ea9f28d', but found 0 lines. Last line was:
E I0506 05:21:32.810438 14218 TAcceptQueueServer.cpp:355] New connection to
server beeswax-frontend from client
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12977) add search and pagination to /hadoop-varz
[ https://issues.apache.org/jira/browse/IMPALA-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843870#comment-17843870 ] ASF subversion and git services commented on IMPALA-12977: -- Commit 7c98ebb7b149a03a1fdb10f0da124c3fd2265f5d in impala's branch refs/heads/master from Saurabh Katiyal [ https://gitbox.apache.org/repos/asf?p=impala.git;h=7c98ebb7b ] IMPALA-12977: add search and pagination to /hadoop-varz Added search and pagination feature to /hadoop-varz Change-Id: Ic8cac23b655fa58ce12d9857649705574614a5f0 Reviewed-on: http://gerrit.cloudera.org:8080/21329 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > add search and pagination to /hadoop-varz > -- > > Key: IMPALA-12977 > URL: https://issues.apache.org/jira/browse/IMPALA-12977 > Project: IMPALA > Issue Type: Improvement >Reporter: Saurabh Katiyal >Assignee: Saurabh Katiyal >Priority: Minor > Labels: newbie > > /hadoop-varz has 2000+ configurations , It'd be nice to have some of the > tools like search and pagination that we get on /varz (flags.tmpl) > existing template to /hadoop-varz: > [https://github.com/apache/impala/blob/ba4cb95b6251911fa9e057cea1cb37958d339fed/www/hadoop-varz.tmpl] > existing template to /varz: > [https://github.com/apache/impala/blob/ba4cb95b6251911fa9e057cea1cb37958d339fed/www/flags.tmpl] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13058) TestRuntimeFilters.test_basic_filters failed on ARM builds
[
https://issues.apache.org/jira/browse/IMPALA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riza Suminto reassigned IMPALA-13058:
-
Assignee: Riza Suminto
> TestRuntimeFilters.test_basic_filters failed on ARM builds
> --
>
> Key: IMPALA-13058
> URL: https://issues.apache.org/jira/browse/IMPALA-13058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Wenzhe Zhou
>Assignee: Riza Suminto
>Priority: Major
>
> Stacktrace:
> query_test/test_runtime_filters.py:98: in test_basic_filters
> self.run_test_case('QueryTest/runtime_filters', vector,
> test_file_vars=vars)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/impala_test_suite.py:851:
> in run_test_case
> update_section=pytest.config.option.update_results)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/test_result_verifier.py:701:
> in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*REMOTE.*ms.*ms.*true
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12264) Add configuration to limit concurrent connections
[ https://issues.apache.org/jira/browse/IMPALA-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Smith resolved IMPALA-12264. Fix Version/s: Impala 4.4.0 Resolution: Fixed > Add configuration to limit concurrent connections > - > > Key: IMPALA-12264 > URL: https://issues.apache.org/jira/browse/IMPALA-12264 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Quanlong Huang >Assignee: Andrew Sherman >Priority: Major > Fix For: Impala 4.4.0 > > > To prevent a rogue application from repeatedly connecting to and monopolizing > Impala, we can add configurations to limit concurrent connections. E.g. limit > the number of concurrent connections per user, IP address, or the user and IP > address combination. > CC [~mylogi...@gmail.com] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13058) TestRuntimeFilters.test_basic_filters failed on ARM builds
[
https://issues.apache.org/jira/browse/IMPALA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843833#comment-17843833
]
Riza Suminto commented on IMPALA-13058:
---
Coordinator received filter update very fast within 2ms, but column "First
arrived" and "Completed" stays "N/A" in "Final filter table".
{code:java}
E Filter routing table:
EID Src. Node Tgt. Node(s) Target type Partition filter Pending
(Expected) First arrived Completed Enabled Bloom Size Est fpp Min value
Max value In-list size
E
-
E 0 2 0 REMOTE true 1
(3)N/AN/A true 1.00 MB 1.13e-16
E Backend startup latencies: Count: 3, sum: 0, min / max: 0 / 0, 25th
%-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th
%-ile: 0
E Slowest backend to start up:
E Final filter table:
EID Src. Node Tgt. Node(s) Target type Partition filter Pending
(Expected) First arrived Completed Enabled Bloom Size Est fpp Min value
Max value In-list size
E
-
E 0 2 0 REMOTE true 0
(3)N/AN/A true 1.00 MB 1.13e-16 {code}
I think what happen is, the filter arrived so fast such that
first_arrival_time() and completion_time() stays at 0 and printed as "N/A"
[https://github.com/apache/impala/blob/0d01f5e82967b84b182d40de1ce213367e0b6181/be/src/runtime/coordinator.cc#L675-L684]
> TestRuntimeFilters.test_basic_filters failed on ARM builds
> --
>
> Key: IMPALA-13058
> URL: https://issues.apache.org/jira/browse/IMPALA-13058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Wenzhe Zhou
>Priority: Major
>
> Stacktrace:
> query_test/test_runtime_filters.py:98: in test_basic_filters
> self.run_test_case('QueryTest/runtime_filters', vector,
> test_file_vars=vars)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/impala_test_suite.py:851:
> in run_test_case
> update_section=pytest.config.option.update_results)
> /data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/test_result_verifier.py:701:
> in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*REMOTE.*ms.*ms.*true
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13059) Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number or metadata file for version 2 is missing
[ https://issues.apache.org/jira/browse/IMPALA-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenzhe Zhou updated IMPALA-13059: - Summary: Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number or metadata file for version 2 is missing (was: Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number) > Testcase fails at test_migrated_table_field_id_resolution due to unmatched > column number or metadata file for version 2 is missing > -- > > Key: IMPALA-13059 > URL: https://issues.apache.org/jira/browse/IMPALA-13059 > Project: IMPALA > Issue Type: Bug >Reporter: Wenzhe Zhou >Priority: Major > > Test failed due to the unmatched number of columns. > > Stacktrace: > query_test/test_iceberg.py:270: in test_migrated_table_field_id_resolution > vector, unique_database) > common/impala_test_suite.py:756: in run_test_case > self.__verify_results_and_errors(vector, test_section, result, use_db) > common/impala_test_suite.py:589: in __verify_results_and_errors > replace_filenames_with_placeholder) > common/test_result_verifier.py:443: in verify_raw_results > verify_results(expected_types, actual_types, order_matters=True) > common/test_result_verifier.py:339: in verify_results > assert expected_results == actual_results > E assert ['INT', 'DOUBLE'] == ['INT', 'DOUBLE', 'DOUBLE'] > E Right contains more items, first extra item: 'DOUBLE' > E Use -v to get the full diff -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13059) Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number
[
https://issues.apache.org/jira/browse/IMPALA-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843827#comment-17843827
]
Wenzhe Zhou commented on IMPALA-13059:
--
In another test case, it failed due to metadata file for version 2 is missing
Stacktrace
query_test/test_iceberg.py:264: in test_migrated_table_field_id_resolution
"iceberg_migrated_complex_test", "parquet")
common/file_utils.py:72: in create_iceberg_table_from_directory
'True');""".format(qualified_table_name))
common/impala_connection.py:216: in execute
fetch_profile_after_close=fetch_profile_after_close)
beeswax/impala_beeswax.py:191: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:384: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:405: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EQuery aborted:ImpalaRuntimeException: Failed to ALTER table
'iceberg_migrated_complex_test': Metadata file for version 2 is missing
> Testcase fails at test_migrated_table_field_id_resolution due to unmatched
> column number
>
>
> Key: IMPALA-13059
> URL: https://issues.apache.org/jira/browse/IMPALA-13059
> Project: IMPALA
> Issue Type: Bug
>Reporter: Wenzhe Zhou
>Priority: Major
>
> Test failed due to the unmatched number of columns.
>
> Stacktrace:
> query_test/test_iceberg.py:270: in test_migrated_table_field_id_resolution
> vector, unique_database)
> common/impala_test_suite.py:756: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:589: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:443: in verify_raw_results
> verify_results(expected_types, actual_types, order_matters=True)
> common/test_result_verifier.py:339: in verify_results
> assert expected_results == actual_results
> E assert ['INT', 'DOUBLE'] == ['INT', 'DOUBLE', 'DOUBLE']
> E Right contains more items, first extra item: 'DOUBLE'
> E Use -v to get the full diff
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13059) Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number
[ https://issues.apache.org/jira/browse/IMPALA-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenzhe Zhou updated IMPALA-13059: - Summary: Testcase fails at test_migrated_table_field_id_resolution due to unmatched column number (was: Testcase fails at test_migrated_table_field_id_resolution due to not matched number of columns) > Testcase fails at test_migrated_table_field_id_resolution due to unmatched > column number > > > Key: IMPALA-13059 > URL: https://issues.apache.org/jira/browse/IMPALA-13059 > Project: IMPALA > Issue Type: Bug >Reporter: Wenzhe Zhou >Priority: Major > > Test failed due to the unmatched number of columns. > > Stacktrace: > query_test/test_iceberg.py:270: in test_migrated_table_field_id_resolution > vector, unique_database) > common/impala_test_suite.py:756: in run_test_case > self.__verify_results_and_errors(vector, test_section, result, use_db) > common/impala_test_suite.py:589: in __verify_results_and_errors > replace_filenames_with_placeholder) > common/test_result_verifier.py:443: in verify_raw_results > verify_results(expected_types, actual_types, order_matters=True) > common/test_result_verifier.py:339: in verify_results > assert expected_results == actual_results > E assert ['INT', 'DOUBLE'] == ['INT', 'DOUBLE', 'DOUBLE'] > E Right contains more items, first extra item: 'DOUBLE' > E Use -v to get the full diff -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13059) Testcase fails at test_migrated_table_field_id_resolution due to not matched number of columns
Wenzhe Zhou created IMPALA-13059: Summary: Testcase fails at test_migrated_table_field_id_resolution due to not matched number of columns Key: IMPALA-13059 URL: https://issues.apache.org/jira/browse/IMPALA-13059 Project: IMPALA Issue Type: Bug Reporter: Wenzhe Zhou Test failed due to the unmatched number of columns. Stacktrace: query_test/test_iceberg.py:270: in test_migrated_table_field_id_resolution vector, unique_database) common/impala_test_suite.py:756: in run_test_case self.__verify_results_and_errors(vector, test_section, result, use_db) common/impala_test_suite.py:589: in __verify_results_and_errors replace_filenames_with_placeholder) common/test_result_verifier.py:443: in verify_raw_results verify_results(expected_types, actual_types, order_matters=True) common/test_result_verifier.py:339: in verify_results assert expected_results == actual_results E assert ['INT', 'DOUBLE'] == ['INT', 'DOUBLE', 'DOUBLE'] E Right contains more items, first extra item: 'DOUBLE' E Use -v to get the full diff -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13058) TestRuntimeFilters.test_basic_filters failed on ARM builds
Wenzhe Zhou created IMPALA-13058:
Summary: TestRuntimeFilters.test_basic_filters failed on ARM builds
Key: IMPALA-13058
URL: https://issues.apache.org/jira/browse/IMPALA-13058
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Wenzhe Zhou
Stacktrace:
query_test/test_runtime_filters.py:98: in test_basic_filters
self.run_test_case('QueryTest/runtime_filters', vector, test_file_vars=vars)
/data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/impala_test_suite.py:851:
in run_test_case
update_section=pytest.config.option.update_results)
/data/jenkins/workspace/impala-cdw-master-exhaustive-release-arm/repos/Impala/tests/common/test_result_verifier.py:701:
in verify_runtime_profile
actual))
E AssertionError: Did not find matches for lines in runtime profile:
E EXPECTED LINES:
E row_regex: .*REMOTE.*ms.*ms.*true
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13057) Incorporate tuple/slot information into the tuple cache key
Joe McDonnell created IMPALA-13057:
--
Summary: Incorporate tuple/slot information into the tuple cache
key
Key: IMPALA-13057
URL: https://issues.apache.org/jira/browse/IMPALA-13057
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 4.4.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
Since the tuple and slot information is kept separately in the descriptor
table, it does not get incorporated into the PlanNode thrift used for the tuple
cache key. This means that the tuple cache can't distinguish between these two
queries:
{noformat}
select int_col1 from table;
select int_col2 from table;{noformat}
To solve this, the tuple/slot information needs to be incorporated into the
cache key. PlanNode::initThrift() walks through each tuple, so this is a good
place to serialize the TupleDescriptor/SlotDescriptors and incorporate it into
the hash.
The tuple ids and slot ids are global ids, so the value is influenced by the
entirety of the query. This is a problem for matching cache results across
different queries. As part of incorporating the tuple/slot information, we
should also add an ability to translate tuple/slot ids into ids local to a
subtree.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13012) Completed queries write fails regularly under heavy load
[
https://issues.apache.org/jira/browse/IMPALA-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843794#comment-17843794
]
ASF subversion and git services commented on IMPALA-13012:
--
Commit 73f13f0e9f225400bb641c48da57a9871b5d8383 in impala's branch
refs/heads/branch-4.4.0 from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=73f13f0e9 ]
IMPALA-13012: Lower default query_log_max_queued
Sets the query_log_max_queued default such that
query_log_max_queued * num_columns(49) < statement_expression_limit
to avoid triggering e.g.
AnalysisException: Exceeded the statement expression limit (25)
Statement has 370039 expressions.
Also increases statement_expression_limit for insertion to avoid an
error if query_log_max_queued is changed.
Logs time taken to write to the queries table for help with debugging
and adds histogram "impala-server.completed-queries.write-durations".
Fixes InternalServer so it uses 'default_query_options'.
Change-Id: I6535675307d88cb65ba7d908f3c692e0cf3259d7
Reviewed-on: http://gerrit.cloudera.org:8080/21351
Reviewed-by: Michael Smith
Tested-by: Michael Smith
Reviewed-by: Riza Suminto
(cherry picked from commit ba32d70891fd68c5c1234ed543b74c51661bf272)
> Completed queries write fails regularly under heavy load
>
>
> Key: IMPALA-13012
> URL: https://issues.apache.org/jira/browse/IMPALA-13012
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Critical
> Fix For: Impala 4.4.0
>
>
> Under heavy test load (running EE tests), Impala regularly fails to write
> completed queries with errors like
> {code}
> W0411 19:11:07.764967 32713 workload-management.cc:435] failed to write
> completed queries table="sys.impala_query_log" record_count="10001"
> W0411 19:11:07.764983 32713 workload-management.cc:437] AnalysisException:
> Exceeded the statement expression limit (25)
> Statement has 370039 expressions.
> {code}
> After a few attempts, it floods logs with an error for each query that could
> not be written
> {code}
> E0411 19:11:24.646953 32713 workload-management.cc:376] could not write
> completed query table="sys.impala_query_log"
> query_id="3142ceb1380b58e6:715b83d9"
> {code}
> This seems like poor default behavior. Options for addressing it:
> # Decrease the default for {{query_log_max_queued}}. Inserts are pretty
> constant at 37 expressions per entry. I'm not sure why that isn't 49, since
> that's the number of columns we have; maybe some fields are frequently
> omitted. I would cap {{query_log_max_queued}} to {{statement_expression_limit
> / number_of_columns ~ 5100}}.
> # Allow workload management to {{set statement_expression_limit}} higher
> using a similar formula. This may be relatively safe as the expressions are
> simple.
> # Ideally we would skip expression parsing and construct TExecRequest
> directly, but that's a much larger effort.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13005) Query Live table not visible via metastore
[
https://issues.apache.org/jira/browse/IMPALA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843793#comment-17843793
]
ASF subversion and git services commented on IMPALA-13005:
--
Commit 91a9dd81ccd38e2e4b975e4ddf49f97c3e01b051 in impala's branch
refs/heads/branch-4.4.0 from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=91a9dd81c ]
IMPALA-13005: Create Query Live table in HMS
Creates the 'sys.impala_query_live' table in HMS using a similar 'CREATE
TABLE' command to 'sys.impala_query_log'. Updates frontend to identify a
System Table based on the '__IMPALA_SYSTEM_TABLE' property. Tables
improperly marked with '__IMPALA_SYSTEM_TABLE' will error when
attempting to scan them because no relevant scanner will be available.
Creating the table in HMS simplifies supporting 'SHOW CREATE TABLE' and
'DESCRIBE EXTENDED', so allows them for parity with Query Log.
Explicitly disables 'COMPUTE STATS' on system tables as it doesn't work
correctly.
Makes System Tables work with local catalog mode, fixing
LocalCatalogException: Unknown table type for table sys.impala_query_live
Updates workload management implementation to rely more on
SystemTables.thrift definition, and adds DCHECKs to verify completeness
and ordering.
Testing:
- adds additional test cases for changes to introspection commands
- passes existing test_query_live and test_query_log suites
Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4
Reviewed-on: http://gerrit.cloudera.org:8080/21302
Reviewed-by: Riza Suminto
Tested-by: Impala Public Jenkins
(cherry picked from commit 73a9ef9c4c00419b5082f355d96961c5971634d6)
> Query Live table not visible via metastore
> --
>
> Key: IMPALA-13005
> URL: https://issues.apache.org/jira/browse/IMPALA-13005
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
> Fix For: Impala 4.4.0
>
>
> Tools that inspect Hive Metastore to view information about tables are unable
> to see {{sys.impala_query_live}} when {{enable_workload_mgmt}} is enabled.
> This presents an inconsistent view depending on whether inspecting tables via
> Impala or Hive MetaStore.
> Register {{sys.impala_query_live}} with Hive MetaStore, similar to what
> {{sys.impala_query_log}} does, to make table listing consistent.
> It also doesn't work with local catalog mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13024) Several tests timeout waiting for admission
[
https://issues.apache.org/jira/browse/IMPALA-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843792#comment-17843792
]
ASF subversion and git services commented on IMPALA-13024:
--
Commit 7ca9798949ffd31b703c610e4eca20d0e0181ab3 in impala's branch
refs/heads/branch-4.4.0 from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7ca979894 ]
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch adds check to recognize coordinator-only query in
default pool and skip slot based admission for it.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Reviewed-on: http://gerrit.cloudera.org:8080/21340
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
(cherry picked from commit 29e418679380591ea7b8e4bdb797ccdbb0964a3d)
> Several tests timeout waiting for admission
> ---
>
> Key: IMPALA-13024
> URL: https://issues.apache.org/jira/browse/IMPALA-13024
> Project: IMPALA
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Riza Suminto
>Priority: Critical
> Fix For: Impala 4.4.0
>
>
> A bunch of seemingly unrelated tests failed with the following message:
> Example:
> query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling_aggs[protocol:
> beeswax | exec_option: {'mt_dop': 1, 'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> {code}
> ImpalaBeeswaxException: EQuery aborted:Admission for query exceeded
> timeout 6ms in pool default-pool. Queued reason: Not enough admission
> control slots available on host ... . Needed 1 slots but 18/16 are already in
> use. Additional Details: Not Applicable
> {code}
> This happened in an ASAN build. Another test also failed which may be related
> to the cause:
> custom_cluster.test_admission_controller.TestAdmissionController.test_queue_reasons_slots
>
> {code}
> Timeout: query 'e1410add778cd7b0:c40812b9' did not reach one of the
> expected states [4], last known state 5
> {code}
> test_queue_reasons_slots seems to be know flaky test: IMPALA-10338
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13015) Dataload fails due to concurrency issue with test.jceks
[
https://issues.apache.org/jira/browse/IMPALA-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843791#comment-17843791
]
ASF subversion and git services commented on IMPALA-13015:
--
Commit 5045f19b5374678c10888376955f2ff5e360ae5b in impala's branch
refs/heads/branch-4.4.0 from Abhishek Rawat
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5045f19b5 ]
IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Move 'hadoop credential' command used for creating test.jceks to
testdata/bin/create-load-data.sh. Earlier it was in bin/load-data.py
which is called in parallel and was causing failures due to race
conditions.
Testing:
- Ran JniFrontendTest#testGetSecretFromKeyStore after data loading and
test ran clean.
Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd
Reviewed-on: http://gerrit.cloudera.org:8080/21346
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
(cherry picked from commit f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08)
> Dataload fails due to concurrency issue with test.jceks
> ---
>
> Key: IMPALA-13015
> URL: https://issues.apache.org/jira/browse/IMPALA-13015
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.4.0
>Reporter: Joe McDonnell
>Assignee: Abhishek Rawat
>Priority: Major
> Labels: flaky
>
> When doing dataload locally, it fails with this error:
> {noformat}
> Traceback (most recent call last):
> File "/home/joemcdonnell/upstream/Impala/bin/load-data.py", line 523, in
>
> if __name__ == "__main__": main()
> File "/home/joemcdonnell/upstream/Impala/bin/load-data.py", line 322, in
> main
> os.remove(jceks_path)
> OSError: [Errno 2] No such file or directory:
> '/home/joemcdonnell/upstream/Impala/testdata/jceks/test.jceks'
> Background task Loading functional-query data (pid 501094) failed.
> {noformat}
> testdata/bin/create-load-data.sh calls bin/load-data.py for functional,
> TPC-H, and TPC-DS in parallel, so this logic has race conditions:
> {noformat}
> jceks_path = TESTDATA_JCEKS_DIR + "/test.jceks"
> if os.path.exists(jceks_path):
> os.remove(jceks_path){noformat}
> I don't see a specific reason for this to be in bin/load-data.py. It should
> be moved somewhere else that doesn't run in parallel. One possible location
> is to add a step in testdata/bin/create-load-data.sh
> This was introduced in
> [https://github.com/apache/impala/commit/9837637d9342a49288a13a421d4e749818da1432]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13038) Support profile tab for imported query profiles
[ https://issues.apache.org/jira/browse/IMPALA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843707#comment-17843707 ] Surya Hebbar commented on IMPALA-13038: --- [Example Query Profile JSON | ^json_profile_a34485359bfdfe1f_3ca8177b.json] [Example Query Profile Text | ^json_profile_a34485359bfdfe1f_3ca8177b.txt] > Support profile tab for imported query profiles > --- > > Key: IMPALA-13038 > URL: https://issues.apache.org/jira/browse/IMPALA-13038 > Project: IMPALA > Issue Type: New Feature >Reporter: Surya Hebbar >Assignee: Surya Hebbar >Priority: Major > Attachments: json_profile_a34485359bfdfe1f_3ca8177b.json, > json_profile_a34485359bfdfe1f_3ca8177b.txt > > > Query profile imports currently support the following tabs. > - Query Statement > - Query Timeline > - Query Text Plan > It would be helpful to support "Query Profile" tab for these imports. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13038) Support profile tab for imported query profiles
[ https://issues.apache.org/jira/browse/IMPALA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surya Hebbar updated IMPALA-13038: -- Attachment: json_profile_a34485359bfdfe1f_3ca8177b.txt json_profile_a34485359bfdfe1f_3ca8177b.json > Support profile tab for imported query profiles > --- > > Key: IMPALA-13038 > URL: https://issues.apache.org/jira/browse/IMPALA-13038 > Project: IMPALA > Issue Type: New Feature >Reporter: Surya Hebbar >Assignee: Surya Hebbar >Priority: Major > Attachments: json_profile_a34485359bfdfe1f_3ca8177b.json, > json_profile_a34485359bfdfe1f_3ca8177b.txt > > > Query profile imports currently support the following tabs. > - Query Statement > - Query Timeline > - Query Text Plan > It would be helpful to support "Query Profile" tab for these imports. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13038) Support profile tab for imported query profiles
[ https://issues.apache.org/jira/browse/IMPALA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surya Hebbar updated IMPALA-13038: -- Summary: Support profile tab for imported query profiles (was: Support Profile tab for Imported Query Profiles) > Support profile tab for imported query profiles > --- > > Key: IMPALA-13038 > URL: https://issues.apache.org/jira/browse/IMPALA-13038 > Project: IMPALA > Issue Type: New Feature >Reporter: Surya Hebbar >Assignee: Surya Hebbar >Priority: Major > > Query profile imports currently support the following tabs. > - Query Statement > - Query Timeline > - Query Text Plan > It would be helpful to support "Query Profile" tab for these imports. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
