[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 04:42:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Reviewed-on: http://gerrit.cloudera.org:8080/21064 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10309/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:08:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Code-Review+2 LGTM. Carry Michael's +1. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:07:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15322/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 20:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15321/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 20:00:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that omit some field. > typo: omit Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:55:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that omit some field. > typo: omit Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:55:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21064 to look at the new patch set (#4). Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21064/4 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that ommit some field. typo: omit -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:51:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java@709 PS2, Line 709:* Variant of toThrift() with lower detail. Intended to get a human readable output > nit: This is also returned in a user-facing API. Debugging to me implies de Done http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@404 PS2, Line 404:* slimmer than the one obtained from getCatalogObject() because binary data fields > typo: slimmer Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:35:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21064 to look at the new patch set (#3). Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21064/3 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 17:08:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java@709 PS2, Line 709:* Variant of toThrift() with lower detail. Intended for test/debugging purpose. nit: This is also returned in a user-facing API. Debugging to me implies developer audience, so maybe broaden this a little to human-readable? http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@404 PS2, Line 404:* slimer than the one obtained from getCatalogObject() because binary data fields typo: slimmer -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 17:07:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15309/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 24 Feb 2024 19:11:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21064 to look at the new patch set (#2). Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 52 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21064/2 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins
