LuciferYang opened a new pull request, #50127: URL: https://github.com/apache/spark/pull/50127
### What changes were proposed in this pull request? This pr reverts commit 971b9a40c42a1d8c050941ce8088fdfc6ac442bc. ### Why are the changes needed? A crash related to RocksDB has occurred in the macOS daily test after upgrade to 9.10.0: - https://github.com/apache/spark/actions/runs/13608353684/job/38042505327 - https://github.com/apache/spark/actions/runs/13464386260/job/37626883152 ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000137105814, pid=32170, tid=707595 # # JRE version: OpenJDK Runtime Environment Zulu21.40+17-CRaC-CA (21.0.6+7) (build 21.0.6+7-LTS) # Java VM: OpenJDK 64-Bit Server VM Zulu21.40+17-CRaC-CA (21.0.6+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64) # Problematic frame: # C [librocksdbjni16177160674801731666.jnilib+0x1d814] Java_org_rocksdb_FlushOptions_disposeInternalJni+0x360 # # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /Users/runner/work/spark/spark/sql/core/hs_err_pid32170.log 21:02:35.349 ERROR org.apache.spark.util.Utils: Aborting task org.apache.spark.SparkException: [CANNOT_WRITE_STATE_STORE.CANNOT_COMMIT] Error writing state store files for provider RocksDBStateStore[id=(op=0,part=1),dir=file:/Users/runner/work/spark/spark/sql/core/target/tmp/streaming.metadata-aa0706db-1b92-485b-b2e6-f356f91ffda6/state/0/1]. Cannot perform commit during state checkpoint. SQLSTATE: 58030 at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToCommitStateFileError(QueryExecutionErrors.scala:2203) at org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider$RocksDBStateStore.commit(RocksDBStateStoreProvider.scala:243) at org.apache.spark.sql.execution.streaming.TransformWithStateExec.$anonfun$processDataWithPartition$5(TransformWithStateExec.scala:431) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:481) at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs(statefulOperators.scala:384) at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs$(statefulOperators.scala:384) 21:02:35.351 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborting commit for partition 1 (task 6, attempt 0, stage 3.0) 21:02:35.351 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborted commit for partition 1 (task 6, attempt 0, stage 3.0) - transformWithState - streaming with rocksdb and invalid processor should fail (encoding = Avro) (without changelog checkpointing) 21:02:35.352 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.0 in stage 3.0 (TID 6) (localhost executor driver): TaskKilled (Stage cancelled: [STATEFUL_PROCESSOR_CANNOT_PERFORM_OPERATION_WITH_INVALID_HANDLE_STATE] Failed to perform stateful processor operation=get_value_state with invalid handle state=INITIALIZED. SQLSTATE: 42802) 21:02:35.372 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled. Warning: [1237.706s][warning][os] Loading hsdis library failed # # If you would like to submit a bug report, please visit: # http://www.azul.com/support/ # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
