[
https://issues.apache.org/jira/browse/BEAM-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534410#comment-17534410
]
Luke Cwik commented on BEAM-14429:
----------------------------------
Can you [fix this log
statement|https://github.com/apache/beam/blob/5c21fbccec5e1e831dd0040bd7f631c050865430/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticUnboundedSource.java#L112]
so we can see the result of the initial split. All I see right now is:
{noformat}
Split into 20 bundles of sizes:
[org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a0e6e3e,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3d060183,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@413f3849,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@17e62ad8,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7ef0d984,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@73f7d5c0,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4b324687,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d2405d3,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1560cd0a,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@587443b3,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@12b3044,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4c6ca72f,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f773b35,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@25e482b3,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7d152f5b,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5469ba09,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32219d22,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@335170cf,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@676afe09,
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e4aa91f]
{noformat}
> SyntheticUnboundedSource(with SDF) produce duplicate records when split with
> DEFAULT_DESIRED_NUM_SPLITS
> -------------------------------------------------------------------------------------------------------
>
> Key: BEAM-14429
> URL: https://issues.apache.org/jira/browse/BEAM-14429
> Project: Beam
> Issue Type: Bug
> Components: io-common
> Reporter: Yichi Zhang
> Priority: P2
> Fix For: 2.39.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> With the default 20 split, the num records produced by
> Read.from(SyntheticUnboundedSource) is always larger than the numRecords
> specified. the more splits the more actual number records produced is off.
> And the Read step tends to take longer time with more splits.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L512]
> The issue is manifested with java LoadTests on dataflow runner v2.
> Initial suspicion is that duplicate source readers for the same restriction
> and checkpoint were created by multiple UnboundedSourceAsSDFWrapperFns.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
