tkaymak opened a new pull request, #39952: URL: https://github.com/apache/beam/pull/39952
Follow up to the review discussion on #39906, https://github.com/apache/beam/pull/39906#discussion_r3877311875. Addresses #36841. Replaces the structured streaming specific maxRecordsPerMicroBatch with a declaration mirroring SparkPipelineOptions#getMaxRecordsPerBatch exactly, Long with default -1, so the flag carries over when users migrate from the legacy runner. Identical signatures share one PipelineOptionsFactory proxy. The new SparkStructuredStreamingPipelineOptionsTest locks that down: a value set through one interface is readable through the other, and the flag parses from args. Nothing consumes the option on master yet, the streaming source arriving with the next slice will read it and treat values below 1 as unlimited, matching the legacy semantics. Full :runners:spark:3:test and :runners:spark:4:test suites green locally on JDK 17 plus spotless, checkstyle and a live ErrorProne compile. R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
