yihua opened a new issue, #17716:
URL: https://github.com/apache/hudi/issues/17716
### Bug Description
**What happened:**
The `TestSavepoint#testSavepoint` is flaky. It indicates that there's a bug
in `RocksDBDAO`.
```
2025-12-25T10:11:51.4787312Z [INFO] Tests run: 3, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 11.475 s - in
org.apache.hudi.client.TestPartitionTTLManagement
2025-12-25T10:11:51.4805028Z [INFO] Running
org.apache.hudi.client.TestSavepoint
2025-12-25T10:12:04.7312245Z 1478094 [qtp1995819387-128355] ERROR
org.apache.hudi.timeline.service.RequestHandler [] - Got runtime exception
servicing request
partition=2016%2F03%2F15&maxinstant=001&timelinehash=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&basepath=%2Ftmp%2Fjunit-4075102816338780888%2Fdataset
2025-12-25T10:12:04.7314410Z org.apache.hudi.exception.HoodieException:
java.util.ConcurrentModificationException
2025-12-25T10:12:04.7314987Z at
org.apache.hudi.common.util.collection.RocksDBDAO.get(RocksDBDAO.java:360)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7315650Z at
org.apache.hudi.common.util.collection.RocksDBDAO.get(RocksDBDAO.java:333)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7316357Z at
org.apache.hudi.common.table.view.RocksDbBasedFileSystemView.isPartitionAvailableInStore(RocksDbBasedFileSystemView.java:274)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7317278Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$18(AbstractTableFileSystemView.java:465)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7317938Z at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
~[?:?]
2025-12-25T10:12:04.7318592Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:463)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7319392Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestMergedFileSliceBeforeOrOnInternal(AbstractTableFileSystemView.java:1078)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7320434Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(AbstractTableFileSystemView.java:1063)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7321397Z at
org.apache.hudi.timeline.service.handlers.FileSliceHandler.getLatestMergedFileSlicesBeforeOrOn(FileSliceHandler.java:59)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7322149Z at
org.apache.hudi.timeline.service.RequestHandler.lambda$registerFileSlicesAPI$24(RequestHandler.java:380)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7322870Z at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.lambda$handle$0(RequestHandler.java:612)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7323451Z at
java.security.AccessController.doPrivileged(Native Method) ~[?:?]
2025-12-25T10:12:04.7323888Z at
javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
2025-12-25T10:12:04.7324516Z at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
~[hadoop-common-2.10.2.jar:?]
2025-12-25T10:12:04.7325077Z at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:595)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7325625Z at
io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7326138Z at
io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7326645Z at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7327133Z at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7327652Z at
io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7328197Z at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7328740Z at
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
~[?:?]
2025-12-25T10:12:04.7329202Z at
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
~[?:?]
2025-12-25T10:12:04.7329735Z at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7330284Z at
io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7330790Z at
javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
~[jakarta.servlet-api-4.0.3.jar:4.0.3]
2025-12-25T10:12:04.7331305Z at
io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7331796Z at
javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
~[jakarta.servlet-api-4.0.3.jar:4.0.3]
2025-12-25T10:12:04.7332318Z at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7332862Z at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7333445Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7333995Z at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7334830Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7335394Z at
io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7336105Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7336801Z at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7337409Z at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7337965Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7338525Z at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7339138Z at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7339719Z at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7340314Z at
org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7340890Z at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7341515Z at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7342087Z at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7342637Z at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7343221Z at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7343775Z at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7371762Z at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7372526Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7373125Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7373771Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7374715Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7375379Z at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7376049Z at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7376657Z at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7377153Z at
java.lang.Thread.run(Thread.java:829) [?:?]
2025-12-25T10:12:04.7377532Z Caused by:
java.util.ConcurrentModificationException
2025-12-25T10:12:04.7377923Z at
java.util.HashMap.computeIfAbsent(HashMap.java:1135) ~[?:?]
2025-12-25T10:12:04.7378464Z at
org.apache.hudi.common.util.collection.RocksDBDAO.getSerializerForColumnFamily(RocksDBDAO.java:520)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7380866Z at
org.apache.hudi.common.util.collection.RocksDBDAO.deserializePayload(RocksDBDAO.java:512)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7381619Z at
org.apache.hudi.common.util.collection.RocksDBDAO.get(RocksDBDAO.java:358)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7382005Z ... 56 more
2025-12-25T10:12:04.7382351Z 1478098 [Executor task launch worker for task
0.0 in stage 34.0 (TID 49)] ERROR org.apache.spark.executor.Executor [] -
Exception in task 0.0 in stage 34.0 (TID 49)
2025-12-25T10:12:04.7382756Z
org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason
phrase: Server Error
2025-12-25T10:12:04.7383362Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:294)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7384089Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(RemoteHoodieTableFileSystemView.java:358)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7385088Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.getFilesToFetchColumnStats(HoodieMetadataWriteUtils.java:555)
~[hudi-client-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7385812Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.lambda$convertMetadataToPartitionStatRecords$4f6e7b3a$1(HoodieMetadataWriteUtils.java:484)
~[hudi-client-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7386488Z at
org.apache.hudi.data.HoodieJavaRDD.lambda$mapToPair$aa72055d$1(HoodieJavaRDD.java:178)
~[classes/:?]
2025-12-25T10:12:04.7387040Z at
org.apache.spark.api.java.JavaPairRDD$.$anonfun$pairFunToScalaFun$1(JavaPairRDD.scala:1073)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7387585Z at
scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
~[scala-library-2.12.18.jar:?]
2025-12-25T10:12:04.7388043Z at
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
~[scala-library-2.12.18.jar:?]
2025-12-25T10:12:04.7388476Z at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
~[scala-library-2.12.18.jar:?]
2025-12-25T10:12:04.7388998Z at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7389588Z at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7390146Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7390701Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7391254Z at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7391771Z at
org.apache.spark.scheduler.Task.run(Task.scala:141)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7392309Z at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7392901Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
~[spark-common-utils_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7393521Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
~[spark-common-utils_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7394091Z at
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
~[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7394916Z at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
[spark-core_2.12-3.5.5.jar:3.5.5]
2025-12-25T10:12:04.7395435Z at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
2025-12-25T10:12:04.7395910Z at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
2025-12-25T10:12:04.7396453Z at
java.lang.Thread.run(Thread.java:829) [?:?]
2025-12-25T10:12:04.7396978Z Caused by:
org.apache.http.client.HttpResponseException: status code: 500, reason phrase:
Server Error
2025-12-25T10:12:04.7397531Z at
org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:70)
~[httpclient-4.5.14.jar:4.5.14]
2025-12-25T10:12:04.7398116Z at
org.apache.http.client.fluent.Response.handleResponse(Response.java:91)
~[fluent-hc-4.5.14.jar:4.5.14]
2025-12-25T10:12:04.7398625Z at
org.apache.http.client.fluent.Response.returnContent(Response.java:98)
~[fluent-hc-4.5.14.jar:4.5.14]
2025-12-25T10:12:04.7399195Z at
org.apache.hudi.timeline.TimelineServiceClient.executeRequest(TimelineServiceClient.java:64)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7399789Z at
org.apache.hudi.timeline.TimelineServiceClientBase.makeRequest(TimelineServiceClientBase.java:57)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7400503Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:169)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7401280Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:290)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7401796Z ... 22 more
2025-12-25T10:12:04.7426859Z 1478107 [task-result-getter-1] ERROR
org.apache.spark.scheduler.TaskSetManager [] - Task 0 in stage 34.0 failed 1
times; aborting job
2025-12-25T10:12:04.7517884Z 1478114 [qtp1995819387-128354] ERROR
org.apache.hudi.timeline.service.RequestHandler [] - Got runtime exception
servicing request
partition=2015%2F03%2F17&maxinstant=001&timelinehash=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&basepath=%2Ftmp%2Fjunit-4075102816338780888%2Fdataset
2025-12-25T10:12:04.7518882Z org.apache.hudi.exception.HoodieException:
java.lang.NullPointerException
2025-12-25T10:12:04.7519503Z at
org.apache.hudi.common.util.collection.RocksDBDAO.put(RocksDBDAO.java:246)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7520188Z at
org.apache.hudi.common.table.view.RocksDbBasedFileSystemView.storePartitionView(RocksDbBasedFileSystemView.java:305)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7520918Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$18(AbstractTableFileSystemView.java:476)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7521553Z at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
~[?:?]
2025-12-25T10:12:04.7522188Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:463)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7522977Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestMergedFileSliceBeforeOrOnInternal(AbstractTableFileSystemView.java:1078)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7530926Z at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(AbstractTableFileSystemView.java:1063)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7531680Z at
org.apache.hudi.timeline.service.handlers.FileSliceHandler.getLatestMergedFileSlicesBeforeOrOn(FileSliceHandler.java:59)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7532406Z at
org.apache.hudi.timeline.service.RequestHandler.lambda$registerFileSlicesAPI$24(RequestHandler.java:380)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7533079Z at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.lambda$handle$0(RequestHandler.java:612)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7533709Z at
java.security.AccessController.doPrivileged(Native Method) ~[?:?]
2025-12-25T10:12:04.7534044Z at
javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
2025-12-25T10:12:04.7534950Z at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
~[hadoop-common-2.10.2.jar:?]
2025-12-25T10:12:04.7535552Z at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:595)
~[hudi-timeline-service-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7536128Z at
io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7536641Z at
io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7537174Z at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7557822Z at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7558394Z at
io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7558972Z at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7559479Z at
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
~[?:?]
2025-12-25T10:12:04.7559989Z at
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
~[?:?]
2025-12-25T10:12:04.7560526Z at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7561063Z at
io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7561580Z at
javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
~[jakarta.servlet-api-4.0.3.jar:4.0.3]
2025-12-25T10:12:04.7562072Z at
io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7562611Z at
javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
~[jakarta.servlet-api-4.0.3.jar:4.0.3]
2025-12-25T10:12:04.7563188Z at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7563826Z at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7564750Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7565413Z at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7566075Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7566688Z at
io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
~[javalin-4.6.7.jar:4.6.7]
2025-12-25T10:12:04.7567267Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7567904Z at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7568509Z at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7569126Z at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7569731Z at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7570493Z at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7571234Z at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7571806Z at
org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7572374Z at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7572968Z at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7573540Z at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7574134Z at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[jetty-server-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7574991Z at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7575573Z at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7576100Z at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[jetty-io-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7576668Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7577267Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7577855Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7578449Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7579052Z at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7579716Z at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7580334Z at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
[jetty-util-9.4.57.v20241219.jar:9.4.57.v20241219]
2025-12-25T10:12:04.7580806Z at
java.lang.Thread.run(Thread.java:829) [?:?]
2025-12-25T10:12:04.7581127Z Caused by: java.lang.NullPointerException
2025-12-25T10:12:04.7581495Z at
org.rocksdb.RocksDB.put(RocksDB.java:857) ~[rocksdbjni-7.5.3.jar:?]
2025-12-25T10:12:04.7581996Z at
org.apache.hudi.common.util.collection.RocksDBDAO.put(RocksDBDAO.java:244)
~[hudi-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
2025-12-25T10:12:04.7582965Z ... 55 more
2025-12-25T10:12:08.6553490Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 17.157 s <<< FAILURE! - in
org.apache.hudi.client.TestSavepoint
2025-12-25T10:12:08.6555925Z [ERROR] testSavepoint{boolean,
FileSystemViewStorageType, HoodieTableType}[6] Time elapsed: 1.678 s <<<
ERROR!
2025-12-25T10:12:08.6556404Z org.apache.hudi.exception.HoodieException:
Error while completing streaming commit to metadata with instant 001
2025-12-25T10:12:08.6556956Z at
org.apache.hudi.client.StreamingMetadataWriteHandler.commitToMetadataTable(StreamingMetadataWriteHandler.java:90)
2025-12-25T10:12:08.6557504Z at
org.apache.hudi.client.SparkRDDWriteClient.writeToMetadataTable(SparkRDDWriteClient.java:165)
2025-12-25T10:12:08.6558004Z at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:314)
2025-12-25T10:12:08.6558500Z at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:270)
2025-12-25T10:12:08.6559215Z at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:148)
2025-12-25T10:12:08.6559828Z at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:94)
2025-12-25T10:12:08.6560293Z at
org.apache.hudi.client.TestSavepoint.testSavepoint(TestSavepoint.java:94)
2025-12-25T10:12:08.6560734Z at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
2025-12-25T10:12:08.6561138Z at
java.base/java.util.Optional.ifPresent(Optional.java:183)
2025-12-25T10:12:08.6561577Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2025-12-25T10:12:08.6594889Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6595500Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2025-12-25T10:12:08.6595991Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2025-12-25T10:12:08.6596492Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6596976Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2025-12-25T10:12:08.6597464Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6597961Z at
java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
2025-12-25T10:12:08.6598456Z at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2025-12-25T10:12:08.6598948Z at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2025-12-25T10:12:08.6599434Z at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2025-12-25T10:12:08.6599972Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2025-12-25T10:12:08.6600499Z at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2025-12-25T10:12:08.6600981Z at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2025-12-25T10:12:08.6601471Z at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
2025-12-25T10:12:08.6601962Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6602450Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6602933Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6603427Z at
java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
2025-12-25T10:12:08.6603908Z at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2025-12-25T10:12:08.6604599Z at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2025-12-25T10:12:08.6605103Z at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2025-12-25T10:12:08.6605605Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2025-12-25T10:12:08.6606099Z at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2025-12-25T10:12:08.6606570Z at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2025-12-25T10:12:08.6607050Z at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
2025-12-25T10:12:08.6607530Z at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
2025-12-25T10:12:08.6608005Z at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2025-12-25T10:12:08.6608484Z at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2025-12-25T10:12:08.6608972Z at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2025-12-25T10:12:08.6609649Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2025-12-25T10:12:08.6610254Z at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2025-12-25T10:12:08.6610720Z at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2025-12-25T10:12:08.6611192Z at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
2025-12-25T10:12:08.6611678Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6612151Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6612632Z at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2025-12-25T10:12:08.6613112Z at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
2025-12-25T10:12:08.6613583Z at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2025-12-25T10:12:08.6614065Z at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2025-12-25T10:12:08.6629313Z at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2025-12-25T10:12:08.6629848Z at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2025-12-25T10:12:08.6630361Z at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2025-12-25T10:12:08.6630850Z at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2025-12-25T10:12:08.6631306Z at
java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
2025-12-25T10:12:08.6631738Z at
java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
2025-12-25T10:12:08.6632458Z Caused by: org.apache.spark.SparkException: Job
aborted due to stage failure: Task 0 in stage 34.0 failed 1 times, most recent
failure: Lost task 0.0 in stage 34.0 (TID 49)
(runnervmpwfgi.mcbgwj2geepurf44gms0qyuivb.cx.internal.cloudapp.net executor
driver): org.apache.hudi.exception.HoodieRemoteException: status code: 500,
reason phrase: Server Error
2025-12-25T10:12:08.6633305Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:294)
2025-12-25T10:12:08.6633993Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(RemoteHoodieTableFileSystemView.java:358)
2025-12-25T10:12:08.6634933Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.getFilesToFetchColumnStats(HoodieMetadataWriteUtils.java:555)
2025-12-25T10:12:08.6635570Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.lambda$convertMetadataToPartitionStatRecords$4f6e7b3a$1(HoodieMetadataWriteUtils.java:484)
2025-12-25T10:12:08.6636171Z at
org.apache.hudi.data.HoodieJavaRDD.lambda$mapToPair$aa72055d$1(HoodieJavaRDD.java:178)
2025-12-25T10:12:08.6636695Z at
org.apache.spark.api.java.JavaPairRDD$.$anonfun$pairFunToScalaFun$1(JavaPairRDD.scala:1073)
2025-12-25T10:12:08.6637168Z at
scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
2025-12-25T10:12:08.6637600Z at
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
2025-12-25T10:12:08.6638040Z at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
2025-12-25T10:12:08.6638532Z at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
2025-12-25T10:12:08.6639060Z at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
2025-12-25T10:12:08.6639555Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
2025-12-25T10:12:08.6640017Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
2025-12-25T10:12:08.6640478Z at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
2025-12-25T10:12:08.6640903Z at
org.apache.spark.scheduler.Task.run(Task.scala:141)
2025-12-25T10:12:08.6641507Z at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
2025-12-25T10:12:08.6642088Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
2025-12-25T10:12:08.6642567Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
2025-12-25T10:12:08.6643022Z at
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
2025-12-25T10:12:08.6643494Z at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
2025-12-25T10:12:08.6643985Z at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2025-12-25T10:12:08.6645715Z at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2025-12-25T10:12:08.6646175Z at
java.base/java.lang.Thread.run(Thread.java:829)
2025-12-25T10:12:08.6646621Z Caused by:
org.apache.http.client.HttpResponseException: status code: 500, reason phrase:
Server Error
2025-12-25T10:12:08.6647184Z at
org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:70)
2025-12-25T10:12:08.6647755Z at
org.apache.http.client.fluent.Response.handleResponse(Response.java:91)
2025-12-25T10:12:08.6648265Z at
org.apache.http.client.fluent.Response.returnContent(Response.java:98)
2025-12-25T10:12:08.6648805Z at
org.apache.hudi.timeline.TimelineServiceClient.executeRequest(TimelineServiceClient.java:64)
2025-12-25T10:12:08.6649395Z at
org.apache.hudi.timeline.TimelineServiceClientBase.makeRequest(TimelineServiceClientBase.java:57)
2025-12-25T10:12:08.6650012Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:169)
2025-12-25T10:12:08.6650685Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:290)
2025-12-25T10:12:08.6651159Z ... 22 more
2025-12-25T10:12:08.6651277Z
2025-12-25T10:12:08.6651561Z Driver stacktrace:
2025-12-25T10:12:08.6651975Z at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
2025-12-25T10:12:08.6652513Z at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
2025-12-25T10:12:08.6653036Z at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
2025-12-25T10:12:08.6653541Z at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
2025-12-25T10:12:08.6654033Z at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
2025-12-25T10:12:08.6655789Z at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
2025-12-25T10:12:08.6656316Z at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
2025-12-25T10:12:08.6656839Z at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
2025-12-25T10:12:08.6657408Z at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
2025-12-25T10:12:08.6657890Z at
scala.Option.foreach(Option.scala:407)
2025-12-25T10:12:08.6658331Z at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
2025-12-25T10:12:08.6658841Z at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
2025-12-25T10:12:08.6659371Z at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
2025-12-25T10:12:08.6659893Z at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
2025-12-25T10:12:08.6660364Z at
org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
2025-12-25T10:12:08.6660820Z at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
2025-12-25T10:12:08.6661253Z at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2393)
2025-12-25T10:12:08.6661682Z at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2414)
2025-12-25T10:12:08.6662256Z at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2433)
2025-12-25T10:12:08.6662682Z at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2458)
2025-12-25T10:12:08.6663221Z at
org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
2025-12-25T10:12:08.6663677Z at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
2025-12-25T10:12:08.6664370Z at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
2025-12-25T10:12:08.6664825Z at
org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
2025-12-25T10:12:08.6665247Z at
org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
2025-12-25T10:12:08.6665712Z at
org.apache.spark.rdd.PairRDDFunctions.$anonfun$countByKey$1(PairRDDFunctions.scala:367)
2025-12-25T10:12:08.6666218Z at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
2025-12-25T10:12:08.6666692Z at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
2025-12-25T10:12:08.6667133Z at
org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
2025-12-25T10:12:08.6667569Z at
org.apache.spark.rdd.PairRDDFunctions.countByKey(PairRDDFunctions.scala:367)
2025-12-25T10:12:08.6668038Z at
org.apache.spark.api.java.JavaPairRDD.countByKey(JavaPairRDD.scala:314)
2025-12-25T10:12:08.6668504Z at
org.apache.hudi.data.HoodieJavaPairRDD.countByKey(HoodieJavaPairRDD.java:115)
2025-12-25T10:12:08.6669038Z at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.buildProfile(BaseSparkCommitActionExecutor.java:235)
2025-12-25T10:12:08.6669663Z at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.prepareWorkloadProfile(BaseSparkCommitActionExecutor.java:214)
2025-12-25T10:12:08.6670266Z at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:185)
2025-12-25T10:12:08.6670850Z at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:167)
2025-12-25T10:12:08.6671485Z at
org.apache.hudi.table.action.deltacommit.SparkUpsertPreppedDeltaCommitActionExecutor.execute(SparkUpsertPreppedDeltaCommitActionExecutor.java:44)
2025-12-25T10:12:08.6672128Z at
org.apache.hudi.table.HoodieSparkMergeOnReadMetadataTable.upsertPrepped(HoodieSparkMergeOnReadMetadataTable.java:61)
2025-12-25T10:12:08.6672710Z at
org.apache.hudi.client.SparkRDDMetadataWriteClient.secondaryUpsertPreppedRecords(SparkRDDMetadataWriteClient.java:136)
2025-12-25T10:12:08.6673346Z at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.secondaryWriteToMetadataTablePartitions(SparkHoodieBackedTableMetadataWriter.java:173)
2025-12-25T10:12:08.6674016Z at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.secondaryWriteToMetadataTablePartitions(SparkHoodieBackedTableMetadataWriter.java:74)
2025-12-25T10:12:08.6675902Z at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.prepareAndWriteToNonStreamingPartitions(HoodieBackedTableMetadataWriter.java:1334)
2025-12-25T10:12:08.6676593Z at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.completeStreamingCommit(HoodieBackedTableMetadataWriter.java:1321)
2025-12-25T10:12:08.6677190Z at
org.apache.hudi.client.StreamingMetadataWriteHandler.commitToMetadataTable(StreamingMetadataWriteHandler.java:88)
2025-12-25T10:12:08.6677599Z ... 54 more
2025-12-25T10:12:08.6677972Z Caused by:
org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason
phrase: Server Error
2025-12-25T10:12:08.6678565Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:294)
2025-12-25T10:12:08.6679266Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(RemoteHoodieTableFileSystemView.java:358)
2025-12-25T10:12:08.6679921Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.getFilesToFetchColumnStats(HoodieMetadataWriteUtils.java:555)
2025-12-25T10:12:08.6680598Z at
org.apache.hudi.metadata.HoodieMetadataWriteUtils.lambda$convertMetadataToPartitionStatRecords$4f6e7b3a$1(HoodieMetadataWriteUtils.java:484)
2025-12-25T10:12:08.6681371Z at
org.apache.hudi.data.HoodieJavaRDD.lambda$mapToPair$aa72055d$1(HoodieJavaRDD.java:178)
2025-12-25T10:12:08.6682074Z at
org.apache.spark.api.java.JavaPairRDD$.$anonfun$pairFunToScalaFun$1(JavaPairRDD.scala:1073)
2025-12-25T10:12:08.6682583Z at
scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
2025-12-25T10:12:08.6683025Z at
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
2025-12-25T10:12:08.6683490Z at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
2025-12-25T10:12:08.6684034Z at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
2025-12-25T10:12:08.6684825Z at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
2025-12-25T10:12:08.6685355Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
2025-12-25T10:12:08.6685884Z at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
2025-12-25T10:12:08.6686381Z at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
2025-12-25T10:12:08.6686844Z at
org.apache.spark.scheduler.Task.run(Task.scala:141)
2025-12-25T10:12:08.6687297Z at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
2025-12-25T10:12:08.6687825Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
2025-12-25T10:12:08.6688364Z at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
2025-12-25T10:12:08.6688861Z at
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
2025-12-25T10:12:08.6689327Z at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
2025-12-25T10:12:08.6689826Z at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2025-12-25T10:12:08.6690371Z at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2025-12-25T10:12:08.6690831Z at
java.base/java.lang.Thread.run(Thread.java:829)
2025-12-25T10:12:08.6691309Z Caused by:
org.apache.http.client.HttpResponseException: status code: 500, reason phrase:
Server Error
2025-12-25T10:12:08.6691863Z at
org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:70)
2025-12-25T10:12:08.6692413Z at
org.apache.http.client.fluent.Response.handleResponse(Response.java:91)
2025-12-25T10:12:08.6692886Z at
org.apache.http.client.fluent.Response.returnContent(Response.java:98)
2025-12-25T10:12:08.6693396Z at
org.apache.hudi.timeline.TimelineServiceClient.executeRequest(TimelineServiceClient.java:64)
2025-12-25T10:12:08.6693960Z at
org.apache.hudi.timeline.TimelineServiceClientBase.makeRequest(TimelineServiceClientBase.java:57)
2025-12-25T10:12:08.6696272Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:169)
2025-12-25T10:12:08.6697063Z at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlicesStreamFromParams(RemoteHoodieTableFileSystemView.java:290)
2025-12-25T10:12:08.6697582Z ... 22 more
```
**What you expected:**
The test should succeed every time.
**Steps to reproduce:**
1. Run Azure CI
### Environment
**Hudi version:** master
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
