[
https://issues.apache.org/jira/browse/IMPALA-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094359#comment-18094359
]
ASF subversion and git services commented on IMPALA-14099:
----------------------------------------------------------
Commit bb9b5fe522b53e384d0fb1d846003e360382cb65 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bb9b5fe52 ]
IMPALA-14099: Tolerate large profile log timestamps
impala-profile-tool parsed profile log timestamps directly as int64_t
before reading the rest of the line. Very large timestamp values made
stream extraction fail, so the tool rejected the whole profile even when
timestamp filters were not used.
Read the timestamp field as text and only convert it to int64_t when
--min_timestamp or --max_timestamp filtering needs numeric comparison.
This preserves filtering for normal timestamps while allowing profile
payload decoding when the timestamp is not needed.
Add regression tests for timestamp filtering. Also cover decoding an
existing public fixture after replacing its timestamp with the oversized
value from IMPALA-14099.
Testing:
- compiled impala-profile-tool in Lima VM using the exact patched source
and existing debug libraries
- manual oversized-timestamp smoke test
- bin/impala-py.test -q tests/observability/test_profile_tool.py
Generated-by: OpenAI Codex
Change-Id: Id2514011d61c4d79b501f52dc0b1c1fcf7b11b06
Reviewed-on: http://gerrit.cloudera.org:8080/24378
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> impala-profile-tool fails to parse a profile
> --------------------------------------------
>
> Key: IMPALA-14099
> URL: https://issues.apache.org/jira/browse/IMPALA-14099
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Aleksandr Efimov
> Priority: Major
> Attachments: test-profile-full.txt
>
>
> Saw this when parsing a profile log:
> {code:bash}
> $ be/build/latest/util/impala-profile-tool < profile_log.txt
> Error parsing line 1: '174830243931748302879977
> 6f497cbe31d87fce:50d6803f00000000
> eJzVWE2MI0cVLptlZzI7meyMYqe9M8PWRtEyI+2Ybrv9K+bgtdtZC//F9mQJRLLK3c+e1ra7vFXVsxlO0Yqf1YLEIgUphyARCXFESFF+biBOKByCckIcgDMgDlEkIoQIanfbY8/YnuGIL3bX+6re972q73XJkVelWy85wI7xjmnsJ7tqJqV3IK4Y6VRXh2xCNpJpOd6V/c9u6FJkNRzY2ESR5c3AY7SMpKWm0+8TdhwKupH1zdCjtTy1bdAFGPiAA7vEKBVX8pQyw7SJoOzGfYfYFrV7e0r0aKBHdYs6BjAS1Wk/G0vJsrxcKJRx63gAV3LlltZot3K3y9qGOwivge4Ik9q4Tw1YJfzY1g8ZtanDVwrQJY4lcKGzZHg/1wpgQY+MmKDlgsOIOzsUV9VoJqn0Od6Jq2omqWCH7y5rtoFbZh+2Y3IssScn9mJJrCSzqpJVMtFMKplUErIsb5T6A2IR7NWtKYiA5WKpWmre0QprfuxlYNyk9jfN4aOBj7xnrEblqBxNRZVoJqrIamxPwQ2trOWaGt7pOKZl4EQqreqqHJdTeiwud9S0nkqluzElE8+A0UnGkoaa6Mgp2H2mCuIBZfdwzjAYcL6hyNFkIpqQo4qSysbjcUWVPYq1gauZ4x0OAneOsU7trtnzS4GJbeCBRWwb2O7vApVWu1Cr78u3WqWK9o1aVdvP9YGZOvlKmfJ2zu6BBfxWvlzSqq12qaBVW6ViSWvs+7qbh2BZ+GiGzB1f2C52dQpMbVxkJs4NGI4lsCJn47FsPI0PWnnsVv9WQSvmDsqtdrFU1trFWqOSa+3Xc42XDrTWONZq5KrNXL5VqlVz5Xbrlbq2X6o2tUarXauWX3nhAup3fxP4/xd6ZdZJXD0ZdHiw9rUV79l11RcKhfJKE/jwRJYKNyClxtNxcI8MZFQlkyVASFIliYQBKjHS8dUR2J29dFvTmndzX3+6ed/ycvbBFlliCWBYkI4FmOvEgrYqy/faCVnWqcXbRicqOpasYAY6PQKGB4QJc7gzK01BmJhvvGRcSSZUWZYvjdtJ5LPQhIApehOrjf089v1EESaLNsPRU+U75eohj1Nd7lSrOW3OieY0XbfJxnihA/s/e3rcTGc00GEPj3xVkvJEEIv2cBOYuzm1AXgrRbp/erz0xsfrD3/99lMff/TLHwX+9Z3vvRf4wcMfvxeIHG6/SAXW/ZmjDseAGNii+r34rOgDZgoYhoccveNyMvisO6cCgnBBGWDdMsEWW0UQ+iGM0F1G+yeY5xpgUWKMo30QxCCCrLpiu6Zt8kMwkLTuVS1P+wPT8pQV33zy6OYH337n5i8++f3NJ7/69GbkzvWKPxvTLiaWhRVvVY514jJYz9nEOuYmH6+8nXPEIWXmt7y6j4bxjk3pYHe97u6BafcmmKz5R9Dsg2XaEOn+7N0v/vwnf/zyww8f/Tnwlw8++Vvg039+9++BP7z9+W+DkcNnPDR3On1TCDDOrnjN37qCYTXgvgNcnLDIDQaWCcZ4F5yBQQRwr4SFQnmtQR9wTI6Iabkyrx7YDHomd418382LkLRcZNQWYBuhYOTfEWnbzzbckmieWk7fds8yj94xBQ+thRGSrs/FVEzOwUX9dQlJN+bCfB0j4PycbhFDV8MSkr40hSkQQTqEQ9nkYkQsiCQ8HzRmhpD0/HzYBLPgwqQ+M3Q6aX3U9VzUZNGeXwQbswsi6YVFwGl+C1OfMHxuCtao53n09rEAHrocfv2jy0i6djY+kWcZSeGzAH/1ldO71xSUkR6UKTEmGGxOYVruWTy1ddtzEJOVuT4HM0H20vxcJ2yks4jJnbo2KzzJY2sWYHpnZqbwGVxGkqQNuzRlLzLqDHie2tw0gIERQuEgcm/ZgUfrDegCA1sHA3sLbC547UakGfjAY/RZZLGjz/Py+SZe7N5zfHuuYy/g1XNceq4/L+DMC3nyXDfO8+EC/81x3mLPLXLbYp+d57BF3prrqgV+WuikuR6a6x4kXR2FcM+NYSWEIlvSUxXoU3ZcIa+FLof/87n/CfhGW3oZmGHqIp3j3OzZWFDcNRkX/hId0InDwXthYpNjmwpMHEH3XDu6rCLSaAnXcRPZkLTqXSy9W1cIRd6SNnxVtYFL1ZXDQivhf3z4w/cDSHo2P7wQDRXfJaYYhd949/uvImmt6vTdF7p/XQqhYXOfHiwy2s+7dxkvutyo5/PUsUUIha8iabVRzzeAGKN1f/pWAElPN+r5u+4NbTT65PVrSAo36IMKEcBMYvm3nwYREAoOk54NjiajYVX9vw/enKF2psZT0uaJGsuZEjItYA7xeZQzW5I0g9Gwg/mVDwe9r+EuoK0QQui/9DrPYQ=='
> Encountered 1 parse errors{code}
> I'm able to parse the profile using Cloudera Manager:
> {noformat}
> Query (id=6f497cbe31d87fce:50d6803f00000000)
> Summary
> Session ID: e74383e3331e9419:aeaa64a55de4ad83
> Session Type: BEESWAX
> Start Time: 2025-05-26 16:41:19.631654000
> End Time: 2025-05-26 16:41:19.976615000
> Duration: 344.961ms (344961 us)
> Query Type: DDL
> Query State: FINISHED
> Impala Query State: FINISHED
> Query Status: OK
> Impala Version: impalad version 4.0.0.7.1.9.1042-1 RELEASE (build
> 5784c40307c230b48c778f21939edb626d45b07e)
> User: root
> Connected User: root
> Delegated User:
> Network Address: 10.65.50.117:33314
> Default Db: default
> Sql Statement: alter table scale_400k_500cols_db.tbl01 recover partitions
> Coordinator: quanlong-1.vpc.cloudera.com:27000
> Query Options (set by configuration):
> TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=Impala Shell
> v4.0.0.7.1.9.1042-1 (5784c40) built on Fri Apr 25 10:32:38 UTC
> 2025,DEFAULT_FILE_FORMAT=PARQUET,DEFAULT_TRANSACTIONAL_TYPE=INSERT_ONLY
> Query Options (set by configuration and planner):
> MT_DOP=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=Impala Shell
> v4.0.0.7.1.9.1042-1 (5784c40) built on Fri Apr 25 10:32:38 UTC
> 2025,DEFAULT_FILE_FORMAT=PARQUET,DEFAULT_TRANSACTIONAL_TYPE=INSERT_ONLY
> DDL Type: ALTER_TABLE
> DDL execution mode: asynchronous
> Catalog Server Operation
> Got catalog version read lock: 58.03us (58031)
> Got catalog version write lock and table write lock: 144.52us (144522)
> Got Metastore client: 9ms (9711617)
> Fetched table from Metastore: 152ms (152441833)
> Reloaded table metadata: 190ms (190898557)
> DDL finished: 191ms (191012998)
> Query Compilation
> Metadata of all 1 tables cached: 39ms (39912460)
> Analysis finished: 40ms (40263964)
> Authorization finished (noop): 40ms (40508567)
> Planning finished: 40ms (40800200)
> Query Timeline
> Query submitted: 44.37us (44370)
> Planning finished: 41ms (41633619)
> CatalogDdlRequest finished: 233ms (233902721)
> Applied catalog updates from DDL: 247ms (247352880)
> Rows available: 247ms (247520762)
> Unregister query: 344ms (344975468)
> Frontend
> Referenced Tables: scale_400k_500cols_db.tbl01
> - CatalogFetch.ColumnStats.Hits: 0 (0)
> - CatalogFetch.ColumnStats.Misses: 501 (501)
> - CatalogFetch.ColumnStats.Requests: 501 (501)
> - CatalogFetch.ColumnStats.Time: 12ms (12)
> - CatalogFetch.DatabaseList.Hits: 1 (1)
> - CatalogFetch.DatabaseList.Misses: 0 (0)
> - CatalogFetch.DatabaseList.Requests: 1 (1)
> - CatalogFetch.DatabaseList.Time: 0ms (0)
> - CatalogFetch.PartitionLists.Hits: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 0ms (0)
> - CatalogFetch.RPCs.Bytes: 52.9 KiB (54208)
> - CatalogFetch.RPCs.Requests: 4 (4)
> - CatalogFetch.RPCs.Time: 5ms (5)
> - CatalogFetch.StorageLoad.Time: 0ms (0)
> - CatalogFetch.TableList.Hits: 1 (1)
> - CatalogFetch.TableList.Misses: 1 (1)
> - CatalogFetch.TableList.Requests: 2 (2)
> - CatalogFetch.TableList.Time: 0ms (0)
> - CatalogFetch.Tables.Hits: 0 (0)
> - CatalogFetch.Tables.Misses: 1 (1)
> - CatalogFetch.Tables.Requests: 1 (1)
> - CatalogFetch.Tables.Time: 3ms (3)
> - ExecutorGroupsConsidered: 1 (1)
> Executor group 1
> Verdict: Assign to first group because query is not auto-scalable
> - MemoryMax: 8.0 EiB (9223372036854775807)
> ImpalaServer
> - CatalogOpExecTimer: 192ms (192017079)
> - ClientFetchWaitTimer: 96ms (96562506)
> - NumRowsFetched: 0 (0)
> - NumRowsFetchedFromCache: 0 (0)
> - RPCCount: 8 (8)
> - RPCReadTimer: 10.00us (10000)
> - RPCWriteTimer: 213.00us (213000)
> - RowMaterializationRate: 0 per second (0)
> - RowMaterializationTimer: 0ns (0){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
