In Java, each of the file system operations take an "options" class[1] such as CreateOptions/MoveOptions. In Python, there is an explicit field as a parameter[2]. Go doesn't seem to have those options available[3].
There was originally a plan to make those options classes be like PipelineOptions where everything has a "view" and can be converted between different types. Whether you decide to continue this path or try something different, you will want to answer questions such as: * How to provide an extensible way for providing file system specific options (S3, GCS, ... all have different "options")? * How do you plumb this through from PTransforms such as FileIO/Read/...? * Which options are "filesystem" level while others are specific to an individual file? * Where do you provide hooks for people to set these options (pipeline creation time or pipeline execution time)? These design docs about filesystems may help provide additional background [4, 5, 6]. 1: https://github.com/apache/beam/tree/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs 2: https://github.com/apache/beam/blob/cb100199d5586414a642d6eaa72800c7463018ca/sdks/python/apache_beam/io/filesystem.py#L717 3: https://github.com/apache/beam/blob/cb100199d5586414a642d6eaa72800c7463018ca/sdks/go/pkg/beam/io/filesystem/filesystem.go#L60 4: https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M 5: https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8 6: http://s.apache.org/fileio-write On Mon, Sep 9, 2019 at 4:24 PM C.J. Collier <[email protected]> wrote: > Hey folks, > > I have a client requesting both content-type and content-encoding for > files. The only thing currently supported is MIME-TYPE, which is not quite > either. > > Could I get some help figuring out how best to allow arbitrary metadata to > be associated with files? > > Cheers, > > C.J. > > > https://issues.apache.org/jira/browse/BEAM-8180 > > -- > C.J. Collier | Technical Solutions Engineer | [email protected] | +1 (206) > 531-1853 <(206)%20531-1853> >
