I believe these are actually several different use cases with different
paths (Some do not exist today):
User wants to output results of a pipeline to be used in a different
pipeline:
(This is indeed a use case in my organization. Also, Spark sees this as a
use case as well with its `saveAsObjectFile`)
1. User's objects are Avro-serializable.
1. [Today] User figures out they can use AvroIO - not very user
friendly, user has to be a little more savvy.
1. [Suggestion #1] Improve documentation for this use case.
2. [Suggestion #2] User uses a more user-friendly FileObjectIO.
1. FileObjectIO, seeing the objects are
Avro-serializable, delegates to AvroIO under the hood.
2. User's objects are not necessarily Avro-serializable.
1. [Today] User writes a custom IO, from scratch, including lower level
IO operations and tests.
2. [Suggestion #3] User uses a more user-friendly FileObjectIO and
provides the proper coder.
1. User uses an existing coder, compatible with their object, for
example - SerializableCoder for serializable objects.
2. No existing coder is compatible with their object, user writes
a custom coder.
3. * Regarding writing several objects to one stream - this issue
is already acknowledged and addressed in
`IterableLikeCoder`s. Some coders
do not require delimiters (Like AvroCoder), others, like
ProtoCoder already
handle writing with delimiters. So it seems that the implicit
contract for
coders, already handles this. I did not find the appropriate
javadoc that
declares this contract.
On Mon, Feb 6, 2017 at 12:13 AM Eugene Kirpichov
<kirpic...@google.com.invalid> wrote:
> Hmm, do you have a concrete use case in mind, where all these circumstances
> come together?
> - There is a need to write data from one Beam pipeline, and read it from
> another Beam pipeline
> - These pipelines have to be kept separate, rather than merged into one
> pipeline
> - It is okay that the data can not be parsed by anything except a Beam
> pipeline using a compatible SDK version
> - The objects to be serialized are not POJOs
> - The user has chosen and was sufficiently savvy to develop a Coder for
> these objects (as opposed to representing them as POJOs and using AvroIO,
> which I think requires a rather less savvy user)
>
> It just seems like a really exotic set of circumstances to me. If the
> problem is that people don't realize that the easiest way to serialize
> their data is make it a POJO and use AvroIO, we can solve by improving
> documentation about coders.
>
> On Sun, Feb 5, 2017 at 1:41 PM Aviem Zur <aviem...@gmail.com> wrote:
>
> > AvroIO would is great for POJOs. But for use cases with more complex,
> > serializable objects, or objects which are compatible with some coder it
> > falls short.
> >
> > Also, for less savvy users to know they need to use AvroIO might be a
> > stretch.
> > Some simpler API along the the lines of ObjectFile might be more user
> > friendly (even if for optimization it uses avro under the hood for
> POJOs).
> >
> > On Sun, Feb 5, 2017, 22:00 Eugene Kirpichov <kirpic...@google.com.invalid
> >
> > wrote:
> >
> > > OK, I see what you mean; however I still think this can be solved
> without
> > > introducing a new "Beam object file" (or whatever) file format, and
> > without
> > > thereby introducing additional use cases and compatibility constraints
> on
> > > coders.
> > >
> > > I asked before in the thread why not just use AvroIO (it can serialize
> > > arbitrary POJOs using reflection); I skimmed the thread it doesn't seem
> > > like that got answered properly. I also like Dan's suggestion to use
> > AvroIO
> > > to serialize byte[] arrays and you can do whatever you want with them
> > (e.g.
> > > use another serialization library, say, Kryo, or Java serialization,
> > etc.)
> > >
> > > On Sun, Feb 5, 2017 at 11:37 AM Aviem Zur <aviem...@gmail.com> wrote:
> > >
> > > > I agree that these files will serve no use outside of Beam pipelines.
> > > >
> > > > The rationale was that you might want to have one pipeline write
> output
> > > to
> > > > files and then have a different pipeline that uses those files as
> > inputs.
> > > >
> > > > Say one team in your organization creates a pipeline and a different
> > team
> > > > utilizes those files as input for a different pipeline. The contract
> > > > between them is the file, in a Beam-readable format.
> > > > This is similar to Spark's `saveAsObjectFile`
> > https://github.com/apache/
> > > >
> > >
> >
> spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1512
> > > > <
> > >
> >
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1512
> > > >
> > > >
> > > > The merit for something like this in my eyes is to not burden the
> user
> > > with
> > > > writing a custom IO
> > > >
> > > > On Tue, Jan 31, 2017 at 10:23 PM Eugene Kirpichov
> > > > <kirpic...@google.com.invalid> wrote:
> > > >
> > > > +1 to Robert. Either this will be a Beam-specific file format (and
> then
> > > > nothing except Beam will be able to read it - which I doubt is what
> you
> > > > want), or it is an existing well-known file format and then we should
> > > just
> > > > develop an IO for it.
> > > > Note that any file format that involves encoding elements with a
> Coder
> > is
> > > > Beam-specific, because wire format of coders is Beam-specific.
> > > >
> > > > On Tue, Jan 31, 2017 at 12:20 PM Robert Bradshaw
> > > > <rober...@google.com.invalid> wrote:
> > > >
> > > > > On Tue, Jan 31, 2017 at 12:04 PM, Aviem Zur <aviem...@gmail.com>
> > > wrote:
> > > > > > +1 on what Stas said.
> > > > > > I think there is value in not having the user write a custom IO
> > for a
> > > > > > protocol they use which is not covered by Beam IOs. Plus having
> > them
> > > > deal
> > > > > > with not only the encoding but also the IO part is not ideal.
> > > > > > I think having a basic FileIO that can write to the Filesystems
> > > > supported
> > > > > > by Beam (GS/HDFS/Local/...) which you can use any coder with,
> > > including
> > > > > > your own custom coder, can be beneficial.
> > > > >
> > > > > What would the format of the file be? Just the concatenation of the
> > > > > elements encoded according to the coder? Or is there a delimiter
> > > > > needed to separate records. In which case how does one ensure the
> > > > > delimiter does not also appear in the middle of an encoded element?
> > At
> > > > > this point you're developing a file format, and might as well stick
> > > > > with one of the standard ones. https://xkcd.com/927
> > > > >
> > > > > > On Tue, Jan 31, 2017 at 7:56 PM Stas Levin <stasle...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > I believe the motivation is to have an abstraction that allows
> one
> > to
> > > > > write
> > > > > > stuff to a file in a way that is agnostic to the coder.
> > > > > > If one needs to write a non-Avro protocol to a file, and this
> > > > particular
> > > > > > protocol does not meet the assumption made by TextIO, one might
> > need
> > > to
> > > > > > duplicate the file IO related code from AvroIO.
> > > > > >
> > > > > > On Tue, Jan 31, 2017 at 6:50 PM Eugene Kirpichov
> > > > > > <kirpic...@google.com.invalid> wrote:
> > > > > >
> > > > > >> Could you clarify why it would be useful to write objects to
> files
> > > > using
> > > > > >> Beam coders, as opposed to just using e.g. AvroIO?
> > > > > >>
> > > > > >> Coders (should) make no promise as to what their wire format is,
> > so
> > > > such
> > > > > >> files could be read back only by other Beam pipelines using the
> > same
> > > > IO.
> > > > > >>
> > > > > >> On Tue, Jan 31, 2017 at 2:48 AM Aviem Zur <aviem...@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> > So If I understand the general agreement is that TextIO should
> > not
> > > > > >> support
> > > > > >> > anything but lines from files as strings.
> > > > > >> > I'll go ahead and file a ticket that says the Javadoc should
> be
> > > > > changed
> > > > > >> to
> > > > > >> > reflect this and `withCoder` method should be removed.
> > > > > >> >
> > > > > >> > Is there merit for Beam to supply an IO which does allow
> writing
> > > > > objects
> > > > > >> to
> > > > > >> > a file using Beam coders and Beam FS (To write these files to
> > > > > >> > GS/Hadoop/Local)?
> > > > > >> >
> > > > > >> > On Tue, Jan 31, 2017 at 2:28 AM Eugene Kirpichov
> > > > > >> > <kirpic...@google.com.invalid> wrote:
> > > > > >> >
> > > > > >> > P.S. Note that this point (about coders) is also mentioned in
> > the
> > > > > >> > now-being-reviewed PTransform Style Guide
> > > > > >> > https://github.com/apache/beam-site/pull/134
> > > > > >> > currently staged at
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> http://apache-beam-website-pull-requests.storage.googleapis.com/134/contribute/ptransform-style-guide/index.html#coders
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, Jan 30, 2017 at 4:25 PM Chamikara Jayalath <
> > > > > chamik...@apache.org
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > +1 to what Eugene said.
> > > > > >> > >
> > > > > >> > > I've seen a number of Python SDK users incorrectly assuming
> > that
> > > > > >> > > coder.decode() is needed when developing their own
> file-based
> > > > > sources
> > > > > >> > > (since many users usually refer to text source first).
> > Probably
> > > > > coder
> > > > > >> > > parameter should not be configurable for text source/sink
> and
> > > they
> > > > > >> should
> > > > > >> > > be updated to only read/write UTF-8 encoded strings.
> > > > > >> > >
> > > > > >> > > - Cham
> > > > > >> > >
> > > > > >> > > On Mon, Jan 30, 2017 at 3:38 PM Eugene Kirpichov
> > > > > >> > > <kirpic...@google.com.invalid> wrote:
> > > > > >> > >
> > > > > >> > > > The use of Coder in TextIO is a long standing design issue
> > > > because
> > > > > >> > coders
> > > > > >> > > > are not intended to be used for general purpose converting
> > > > things
> > > > > >> from
> > > > > >> > > and
> > > > > >> > > > to bytes, their only proper use is letting the runner
> > > > materialize
> > > > > > and
> > > > > >> > > > restore objects if the runner thinks it's necessary. IMO
> it
> > > > should
> > > > > >> have
> > > > > >> > > > been called LineIO, document that it reads lines of text
> as
> > > > > String,
> > > > > >> and
> > > > > >> > > not
> > > > > >> > > > have a withCoder parameter at all.
> > > > > >> > > >
> > > > > >> > > > The proper way to address your use case is to write a
> custom
> > > > > >> > > > FileBasedSource.
> > > > > >> > > > On Mon, Jan 30, 2017 at 2:52 AM Aviem Zur <
> > aviem...@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > The Javadoc of TextIO states:
> > > > > >> > > > >
> > > > > >> > > > > * <p>By default, {@link TextIO.Read} returns a {@link
> > > > > PCollection}
> > > > > >> of
> > > > > >> > > > > {@link String Strings},
> > > > > >> > > > > * each corresponding to one line of an input UTF-8 text
> > > file.
> > > > > To
> > > > > >> > > convert
> > > > > >> > > > > directly from the raw
> > > > > >> > > > > * bytes (split into lines delimited by '\n', '\r', or
> > > '\r\n')
> > > > > to
> > > > > >> > > another
> > > > > >> > > > > object of type {@code T},
> > > > > >> > > > > * supply a {@code Coder<T>} using {@link
> > > > > >> > > TextIO.Read#withCoder(Coder)}.
> > > > > >> > > > >
> > > > > >> > > > > However, as I stated, `withCoder` doesn't seem to have
> > > tests,
> > > > > and
> > > > > >> > > > probably
> > > > > >> > > > > won't work given the hard-coded '\n' delimiter.
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Jan 30, 2017 at 12:18 PM Jean-Baptiste Onofré <
> > > > > >> > j...@nanthrax.net
> > > > > >> > > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Aviem,
> > > > > >> > > > > >
> > > > > >> > > > > > TextIO is not designed to write/read binary file: it's
> > > pure
> > > > > > Text,
> > > > > >> > so
> > > > > >> > > > > > String.
> > > > > >> > > > > >
> > > > > >> > > > > > Regards
> > > > > >> > > > > > JB
> > > > > >> > > > > >
> > > > > >> > > > > > On 01/30/2017 09:24 AM, Aviem Zur wrote:
> > > > > >> > > > > > > Hi,
> > > > > >> > > > > > >
> > > > > >> > > > > > > While trying to use TextIO to write/read a binary
> file
> > > > > rather
> > > > > >> > than
> > > > > >> > > > > String
> > > > > >> > > > > > > lines from a textual file I ran into an issue - the
> > > > > delimiter
> > > > > >> > > TextIO
> > > > > >> > > > > uses
> > > > > >> > > > > > > seems to be hardcoded '\n'.
> > > > > >> > > > > > > See `findSeparatorBounds` -
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L1024
> > > > > >> > > > > > >
> > > > > >> > > > > > > The use case is to have a file of objects, encoded
> > into
> > > > > bytes
> > > > > >> > > using a
> > > > > >> > > > > > > coder. However, '\n' is not a good delimiter here,
> as
> > > you
> > > > > can
> > > > > >> > > > imagine.
> > > > > >> > > > > > > A similar pattern is found in Spark's
> > `saveAsObjectFile`
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1512
> > > > > >> > > > > > > where
> > > > > >> > > > > > > they use a more appropriate delimiter, to avoid such
> > > > issues.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I did not find any unit tests which use TextIO to
> read
> > > > > > anything
> > > > > >> > > other
> > > > > >> > > > > > than
> > > > > >> > > > > > > Strings.
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > --
> > > > > >> > > > > > Jean-Baptiste Onofré
> > > > > >> > > > > > jbono...@apache.org
> > > > > >> > > > > > http://blog.nanthrax.net
> > > > > >> > > > > > Talend - http://www.talend.com
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
