giriprakash opened a new pull request, #57893: URL: https://github.com/apache/spark/pull/57893
### What changes were proposed in this pull request? Allow `BlockId.apply` to parse RDD block names whose RDD id is negative, e.g. `rdd_-1330910599_36`. `RDDBlockId.name` already renders negative ids with a leading minus sign. The only change is to make the RDD id regex accept an optional minus so the name round-trips through `BlockId.apply`. The split index stays non-negative, and no other block id pattern is changed. ### Why are the changes needed? `SparkContext` hands out RDD ids from a 32-bit `AtomicInteger` that never resets while the application runs. After `Int.MaxValue` RDDs, the counter wraps to negative values. Cached blocks then get names such as `rdd_-1330910599_36`. Paths that reconstruct a `BlockId` from its name — including `UpdateBlockInfo.readExternal` — failed with `UnrecognizedBlockId` because the parser only accepted digits. The application stayed up, but queries kept failing until restart. This matches the failure mode described in [SPARK-41246](https://issues.apache.org/jira/browse/SPARK-41246). ### Does this PR introduce _any_ user-facing change? Yes. Applications that previously failed with `UnrecognizedBlockId` after the RDD id counter overflowed can now parse and use those block ids. Behavior for all previously valid (non-negative) RDD block ids is unchanged. ### How was this patch tested? Added a unit test in `BlockIdSuite` that: - parses the reported failing name `rdd_-1330910599_36` - parses `rdd_-2147483648_0` (`Int.MinValue`) - confirms positive ids still round-trip and remain distinct from their negative counterparts Please run: ``` build/sbt "core/testOnly org.apache.spark.storage.BlockIdSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Agent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
