damccorm opened a new issue, #20442:
URL: https://github.com/apache/beam/issues/20442
### Description:
`fileio.WriteToFiles` ignores the option `shards=3` given to its constructor
unless I set `max_writers_per_bundle` to `0`.
It reproduces with InteractiveRunner or DirectRunner, but does not reproduce
with DataflowRunner.
### Example:
Suppose I have the following pipeline (with interactive runner):
```
import apache_beam as beam
import apache_beam.io.fileio as fileio
import apache_beam.runners.interactive.interactive_beam
as ib
user_ids = list(map(lambda x: 'user_id' + str(x), range(0, 10000)))
with beam.Pipeline(InteractiveRunner())
as pipeline:
user_list = pipeline | 'create pcollection' >> beam.Create(user_ids)
write_sharded_csv
= user_list | 'write sharded csv files' >> fileio.WriteToFiles(
path='/tmp/data/',
shards=3,
file_naming=fileio.default_file_naming(prefix='userlist',
suffix='.csv'),
# max_writers_per_bundle=0,
)
ib.show(write_sharded_csv)
```
This pipeline is implemented to...
- Creates PCollection of strings: 'user_id1', 'user_id2', ... 'user_id10000'
- Writes the user ids to 3 local files with sharding.
The code does not work as intended. It writes whole user ids to only 1 file.
The code DOES work as intended after I added the `max_writers_per_bundle=0`
argument to the `WriteToFiles` constructor.
The code also works if I use GCP's DataflowRunner instead of
InteractiveRunner.
Is the behavior intentional or bug?
I couldn't understand why `max_writers_per_bundle` is related to the
sharding behavior. I couldn't find any documentation about this.
Imported from Jira
[BEAM-10596](https://issues.apache.org/jira/browse/BEAM-10596). Original Jira
may contain additional context.
Reported by: gecko655.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
