[
https://issues.apache.org/jira/browse/BEAM-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Guru updated BEAM-6721:
------------------------------
Description:
I go through code for TextIO. It uses WriteFiles in TypedWrite. Now,
WriteFiles has the functionality
withNumShards(ValueProvider<java.lang.Integer>) and
[withSharding|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html#withSharding-org.apache.beam.sdk.transforms.PTransform-]([PTransform|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/transforms/PTransform.html]<[PCollection|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollection.html]<[UserT|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html]>,[PCollectionView|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollectionView.html]<java.lang.Integer>>
sharding) but TextIO or its inner class TypedWrite doesn't have that. It would
really be a great to expose that functionality to user.
Are there some issues in implementation that this is not already available?
was:
I go through code for TextIO. It uses WriteFiles in TypedWrite. Now,
WriteFiles has the functionality getNumShards(ValueProvider<java.lang.Integer>)
but TypedWrite doesn't have that. It would really be a great to expose that
functionality to user.
Are there some issues in implementation that this is not already available?
> setting numshards dynamically for TextIO.write()
> -------------------------------------------------
>
> Key: BEAM-6721
> URL: https://issues.apache.org/jira/browse/BEAM-6721
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files, io-java-text
> Reporter: Aditya Guru
> Priority: Minor
>
> I go through code for TextIO. It uses WriteFiles in TypedWrite. Now,
> WriteFiles has the functionality
> withNumShards(ValueProvider<java.lang.Integer>) and
> [withSharding|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html#withSharding-org.apache.beam.sdk.transforms.PTransform-]([PTransform|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/transforms/PTransform.html]<[PCollection|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollection.html]<[UserT|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html]>,[PCollectionView|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollectionView.html]<java.lang.Integer>>
> sharding) but TextIO or its inner class TypedWrite doesn't have that. It
> would really be a great to expose that functionality to user.
> Are there some issues in implementation that this is not already available?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
