Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23436 )
Change subject: IMPALA-14447: Parallelize table loading in getMissingTables() ...................................................................... Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/23436/4/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/23436/4/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@337 PS4, Line 337: List<FeTable> tables = : (new ArrayList<>(tbls)) : .parallelStream() Something wrong seems happen in Analyzer.java once this is parallelized. I run this: ./run-tests.py query_test/test_observability.py In the coordinator, I see failed test with following traces. I20250919 15:58:21.757603 1681103 impala-server.cc:1520] 9648f3fc3a67273b:f894676500000000] Registered query query_id=9648f3fc3a67273b:f894676500000000 session_id=d548b1e160c9d65f:f9bf8d535ddf94b3 I20250919 15:58:21.757772 1681103 Frontend.java:3753] 9648f3fc3a67273b:f894676500000000] Searching for planner to use... I20250919 15:58:21.757786 1681103 Frontend.java:2407] 9648f3fc3a67273b:f894676500000000] Using Original Planner. I20250919 15:58:21.757798 1681103 Frontend.java:2435] 9648f3fc3a67273b:f894676500000000] Analyzing query: compute stats test_compute_stats_profile_8323a0c4.test_compute_stats_profile db: default I20250919 15:58:21.757817 1681103 Frontend.java:2471] 9648f3fc3a67273b:f894676500000000] The original executor group sets from executor membership snapshot: [TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:)] I20250919 15:58:21.757836 1681103 Frontend.java:2491] 9648f3fc3a67273b:f894676500000000] A total of 1 executor group sets to be considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:, max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)] I20250919 15:58:21.757867 1681103 Frontend.java:2535] 9648f3fc3a67273b:f894676500000000] Consider executor group set: TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:, max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647) with assumption of 0 cores per node. I20250919 15:58:21.758054 1681103 CatalogdMetaProvider.java:640] 9648f3fc3a67273b:f894676500000000] Request for database list: hit I20250919 15:58:21.758116 1681103 CatalogdMetaProvider.java:640] 9648f3fc3a67273b:f894676500000000] Request for table list of database default: hit I20250919 15:58:21.758128 1672056 impala-server.cc:1679] Query successfully unregistered: query_id=bd4f8fad7dfe212d:a5cb0d1000000000 I20250919 15:58:21.758241 1681103 Analyzer.java:1093] 9648f3fc3a67273b:f894676500000000] Resolving TableRef test_compute_stats_profile_8323a0c4.test_compute_stats_profile I20250919 15:58:21.758265 1681103 Analyzer.java:1551] 9648f3fc3a67273b:f894676500000000] Candidates for TABLE_REF test_compute_stats_profile_8323a0c4.test_compute_stats_profile: [] I20250919 15:58:21.758289 1681103 Analyzer.java:1599] 9648f3fc3a67273b:f894676500000000] Replace candidates with [] I20250919 15:58:21.758350 1681103 AnalysisContext.java:512] 9648f3fc3a67273b:f894676500000000] Analysis took 0 ms I20250919 15:58:21.758379 1681103 BaseAuthorizationChecker.java:114] 9648f3fc3a67273b:f894676500000000] Authorization check took 0 ms I20250919 15:58:21.758378 1672056 query-exec-mgr.cc:221] ReleaseQueryState(): deleted query_id=bd4f8fad7dfe212d:a5cb0d1000000000 I20250919 15:58:21.758452 1681103 jni-util.cc:321] 9648f3fc3a67273b:f894676500000000] org.apache.impala.common.AnalysisException: Could not resolve table reference: 'test_compute_stats_profile_8323a0c4.test_compute_stats_profile' at org.apache.impala.analysis.Analyzer.resolvePath(Analyzer.java:1609) at org.apache.impala.analysis.Analyzer.resolvePath(Analyzer.java:1541) at org.apache.impala.analysis.Analyzer.resolvePathWithMasking(Analyzer.java:1456) at org.apache.impala.analysis.Analyzer.resolveTableRef(Analyzer.java:1114) at org.apache.impala.analysis.ComputeStatsStmt.analyze(ComputeStatsStmt.java:394) at org.apache.impala.analysis.AnalysisContext$AnalysisDriverImpl.analyze(AnalysisContext.java:576) at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:496) at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2975) at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2542) at org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2411) at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2100) at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:176) I20250919 15:58:21.758458 1681103 status.cc:129] 9648f3fc3a67273b:f894676500000000] AnalysisException: Could not resolve table reference: 'test_compute_stats_profile_8323a0c4.test_compute_stats_profile' @ 0x320ab81 impala::Status::Status() @ 0x447488b impala::JniUtil::GetJniExceptionMsg() @ 0x419f3e4 impala::JniCall::Call<>() @ 0x417dc9b impala::Frontend::GetExecRequest() @ 0x3f3afab impala::QueryDriver::DoFrontendPlanning() @ 0x3f3b165 impala::QueryDriver::RunFrontendPlanner() @ 0x4260a3d impala::ImpalaServer::ExecuteInternal() @ 0x4261905 impala::ImpalaServer::Execute() @ 0x41ef1d1 impala::ImpalaServer::ExecuteStatementCommon() @ 0x41f0523 impala::ImpalaServer::ExecuteStatement() @ 0x42dde29 apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::process_ExecuteStatement() @ 0x4319b93 apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::dispatchCall() @ 0x431a138 impala::ImpalaHiveServer2ServiceProcessorT<>::dispatchCall() @ 0x310d50f apache::thrift::TDispatchProcessor::process() @ 0x3dda06d apache::thrift::server::TAcceptQueueServer::Task::run() @ 0x3df1adb impala::ThriftThread::RunRunnable() @ 0x3df2b12 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x455346c impala::Thread::SuperviseThread() @ 0x4553f71 boost::detail::thread_data<>::run() @ 0x46dd247 thread_proxy @ 0x7f4edf9e4609 start_thread @ 0x7f4edda29133 clone @ (nil) (unknown) -- To view, visit http://gerrit.cloudera.org:8080/23436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97a5165844ae846b28338d62e93a20121488d79f Gerrit-Change-Number: 23436 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com> Gerrit-Comment-Date: Fri, 19 Sep 2025 23:02:03 +0000 Gerrit-HasComments: Yes
