[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

2020-10-06 Thread GitBox
iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704500066 Excellent idea to break the PR into little ones, first one merged. waiting for the next! This is an automated

[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

2020-10-06 Thread GitBox
iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704389742 I see, so it is the full switch from Read.Bounded/Unbounded to SDF by default. Can you get this one green so we can test it and then merge it, I would like to see if there is

[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

2020-10-05 Thread GitBox
iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-703727923 > @iemejia Since streaming is effectively broken due to lack of support for watermark holds. What do you think about enabling SDF for Spark and it only working in batch?

[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

2020-09-23 Thread GitBox
iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-697989304 The phenomenon of microbatches producing results early I noticed it too in the past when trying to enable the Read.Unbounded tests. I could not understand why, and I thought it