[
https://issues.apache.org/jira/browse/HIVE-30024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zsolt Miskolczi reassigned HIVE-30024:
--------------------------------------
Assignee: Zsolt Miskolczi
> Scheduled Query auto-disable failures in HA configurations
> ----------------------------------------------------------
>
> Key: HIVE-30024
> URL: https://issues.apache.org/jira/browse/HIVE-30024
> Project: Hive
> Issue Type: Bug
> Reporter: Zsolt Miskolczi
> Assignee: Zsolt Miskolczi
> Priority: Major
> Labels: pull-request-available
>
> Auto-disable fails intermittently on HA configurations, because the query is
> not disabled after the given number of failures. Example beeline output from
> a test run where 3 failures are expected:
> {noformat}
> 'scheduled_executions.scheduled_execution_id'
> 'scheduled_executions.schedule_name'
> 'scheduled_executions.executor_query_id'
> 'scheduled_executions.state' 'scheduled_executions.start_time'
> 'scheduled_executions.end_time' 'scheduled_executions.elapsed'
> 'scheduled_executions.error_message'
> 'scheduled_executions.last_update_time'
> '13' 'sq_ucw'
> 'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0'
> 'FAILED' '2026-09-03 17:31:05' '2026-09-03
> 17:31:05' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '12' 'sq_ucw'
> 'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97'
> 'FAILED' '2026-09-03 17:29:03' '2026-09-03
> 17:29:04' '1' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '11' 'sq_ucw'
> 'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
> 'FINISHED' '2026-09-03 17:27:01' '2026-09-03
> 17:27:03' '2' 'NULL'
> 'NULL'
> '3' 'sq_ucw'
> 'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7'
> 'FAILED' '2026-09-03 17:30:04' '2026-09-03
> 17:30:04' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '2' 'sq_ucw'
> 'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55'
> 'FAILED' '2026-09-03 17:28:02' '2026-09-03
> 17:28:03' '1'
>
> 'scheduled_executions.scheduled_execution_id'
> 'scheduled_executions.schedule_name'
> 'scheduled_executions.executor_query_id'
> 'scheduled_executions.state' 'scheduled_executions.start_time'
> 'scheduled_executions.end_time' 'scheduled_executions.elapsed'
> 'scheduled_executions.error_message'
> 'scheduled_executions.last_update_time'
> '13' 'sq_ucw'
> 'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0'
> 'FAILED' '2026-09-03 17:31:05' '2026-09-03
> 17:31:05' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '3' 'sq_ucw'
> 'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7'
> 'FAILED' '2026-09-03 17:30:04' '2026-09-03
> 17:30:04' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '12' 'sq_ucw'
> 'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97'
> 'FAILED' '2026-09-03 17:29:03' '2026-09-03
> 17:29:04' '1' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '2' 'sq_ucw'
> 'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55'
> 'FAILED' '2026-09-03 17:28:02' '2026-09-03
> 17:28:03' '1'
>
> '11' 'sq_ucw'
> 'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
> 'FINISHED' '2026-09-03 17:27:01' '2026-09-03
> 17:27:03' '2' 'NULL'
> 'NULL'{noformat}
> Both the test and the actual implementation in ObjectStore expect the IDs to
> be an incrementing sequence, but if you check the above output this is not
> always the case. The above output is ordered using the ID, and start and end
> timestamps should correlate to this, but they don’t.
>
> {noformat}
> Sep 03 19:26:00 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:1) thread - renaming accordingly.
> Sep 03 19:27:01 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:11) thread - renaming accordingly.
> Sep 03 19:28:02 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:2) thread - renaming accordingly.
> Sep 03 19:29:03 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:12) thread - renaming accordingly.
> Sep 03 19:30:04 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:3) thread - renaming accordingly.
> Sep 03 19:31:05 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:13) thread - renaming accordingly.
> Sep 03 19:32:05 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:4) thread - renaming accordingly.
> Sep 03 19:33:06 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:14) thread - renaming accordingly.
> Sep 03 19:37:15 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:21) thread - renaming accordingly.
> Sep 03 19:38:17 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:22) thread - renaming accordingly.
> Sep 03 19:39:18 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:23) thread - renaming accordingly.
> Sep 03 19:40:19 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:24) thread - renaming accordingly.
> Sep 03 19:41:19 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:25) thread - renaming accordingly.
> Sep 03 19:42:20 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:26) thread - renaming accordingly.
> Sep 03 19:43:20 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:27) thread - renaming accordingly.{noformat}
> It shows that for the first scheduled query the two nodes use their own
> distinct ID ranges. Gemini theorized, the evidence and the DataNucleus
> documentation supports it, that this could be due to ID pre-allocation. From
> the DataNucleus JDO Mapping Guide (v6.0)
> :[https://www.datanucleus.org/products/accessplatform_6_0/jdo/mapping.html#value_generation]
>
> It looks there are two possible solution:
> - disable pre-allocation
> - order by based on an other property that is deterministic, instead of the
> Id itself (maybe end_time can be a proper candidate)
> Related code:
> [https://github.com/apache/hive/blob/bfe74f9b3f92188095d50cf103b86431f8aaa821/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L2956]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
