I want to bring this thread back up since we've had time to think about it
more and make a plan.
I think a format-specific converter will be more time consuming task than
we originally thought. It'd have to be a writer that takes another writer
as a parameter.
I think a string converter can be done as a simple transform.
I think we should start with a simple string converter and plan for a
format-specific writer.
What are your thoughts?
Thanks,
Jesse
On Thu, Nov 10, 2016 at 10:33 AM Jesse Anderson <je...@smokinghand.com>
wrote:
I was thinking about what the outputs would look like last night. I
realized that more complex formats like JSON and XML may or may not output
the data in a valid format.
Doing a direct conversion on unbounded collections would work just fine.
They're self-contained. For writing out bounded collections, that's where
we'll hit the issues. This changes the uber conversion transform into a
transform that needs to be a writer.
If a transform executes a JSON conversion on a per element basis, we'd get
this:
{
"key": "value"
}, {
"key": "value"
},
That isn't valid JSON.
The conversion transform would need to know do several things when writing
out a file. It would need to add brackets for an array. Now we have:
[
{
"key": "value"
}, {
"key": "value"
},
]
We still don't have valid JSON. We have to remove the last comma or have
the uber transform start putting in the commas, except for the last element.
[
{
"key": "value"
}, {
"key": "value"
}
]
Only by doing this do we have valid JSON.
I'd argue we'd have a similar issue with XML. Some parsers require a root
element for everything. The uber transform would have to put the root
element tags at the beginning and end of the file.
On Wed, Nov 9, 2016 at 11:36 PM Manu Zhang <owenzhang1...@gmail.com> wrote:
I would love to see a lean core and abundant Transforms at the same time.
Maybe we can look at what Confluent <https://github.com/confluentinc> does
for kafka-connect. They have official extensions support for JDBC, HDFS and
ElasticSearch under https://github.com/confluentinc. They put them along
with other community extensions on
https://www.confluent.io/product/connectors/ for visibility.
Although not a commercial company, can we have a GitHub user like
beam-community to host projects we build around beam but not suitable for
https://github.com/apache/incubator-beam. In the future, we may have
beam-algebra like http://github.com/twitter/algebird for algebra operations
and beam-ml / beam-dl for machine learning / deep learning. Also, there
will will be beam related projects elsewhere maintained by other
communities. We can put all of them on the beam-website or like spark
packages as mentioned by Amit.
My $0.02
Manu
On Thu, Nov 10, 2016 at 2:59 AM Kenneth Knowles <k...@google.com.invalid>
wrote:
On this point from Amit and Ismaël, I agree: we could benefit from a place
for miscellaneous non-core helper transformations.
We have sdks/java/extensions but it is organized as separate artifacts. I
think that is fine, considering the nature of Join and SortValues. But for
simpler transforms, Importing one artifact per tiny transform is too much
overhead. It also seems unlikely that we will have enough commonality
among
the transforms to call the artifact anything other than [some synonym for]
"miscellaneous".
I wouldn't want to take this too far - even though the SDK many
transforms*
that are not required for the model [1], I like that the SDK artifact has
everything a user might need in their "getting started" phase of use. This
user-friendliness (the user doesn't care that ParDo is core and Sum is
not)
plus the difficulty of judging which transforms go where, are probably why
we have them mostly all in one place.
Models to look at, off the top of my head, include Pig's PiggyBank and
Apex's Malhar. These have different levels of support implied. Others?
Kenn
[1] ApproximateQuantiles, ApproximateUnique, Count, Distinct, Filter,
FlatMapElements, Keys, Latest, MapElements, Max, Mean, Min, Values,
KvSwap,
Partition, Regex, Sample, Sum, Top, Values, WithKeys, WithTimestamps
* at least they are separate classes and not methods on PCollection :-)
On Wed, Nov 9, 2016 at 6:03 AM, Ismaël Mejía <ieme...@gmail.com> wrote:
Nice discussion, and thanks Jesse for bringing this subject back.
I agree 100% with Amit and the idea of having a home for those
transforms
that are not core enough to be part of the sdk, but that we all end up
re-writing somehow.
This is a needed improvement to be more developer friendly, but also as
a
reference of good practices of Beam development, and for this reason I
agree with JB that at this moment it would be better for these
transforms
to reside in the Beam repository at least for visibility reasons.
One additional question is if these transforms represent a different DSL
or
if those could be grouped with the current extensions (e.g. Join and
SortValues) into something more general that we as a community could
maintain, but well even if it is not the case, it would be really nice
to
start working on something like this.
Ismaël Mejía
On Wed, Nov 9, 2016 at 11:59 AM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
Related to spark-package, we also have Apache Bahir to host
connectors/transforms for Spark and Flink.
IMHO, right now, Beam should host this, not sure if it makes sense
directly in the core.
It reminds me the "Integration" DSL we discussed in the technical
vision
document.
Regards
JB
On 11/09/2016 11:17 AM, Amit Sela wrote:
I think Jesse has a very good point on one hand, while Luke's and
Kenneth's
worries about committing users to specific implementations is in
place.
The Spark community has a 3rd party repository for useful libraries
that
for various reasons are not a part of the Apache Spark project:
https://spark-packages.org/.
Maybe a "common-transformations" package would serve both users quick
ramp-up and ease-of-use while keeping Beam more "enabling" ?
On Tue, Nov 8, 2016 at 9:03 PM Kenneth Knowles
<k...@google.com.invalid
wrote:
It seems useful for small scale debugging / demoing to have
Dump.toString(). I think it should be named to clearly indicate its
limited
scope. Maybe other stuff could go in the Dump namespace, but
"Dump.toJson()" would be for humans to read - so it should be pretty
printed, not treated as a machine-to-machine wire format.
The broader question of representing data in JSON or XML, etc, is
already
the subject of many mature libraries which are already easy to use
with
Beam.
The more esoteric practice of implicit or semi-implicit coercions
seems
like it is also already addressed in many ways elsewhere.
Transform.via(TypeConverter) is basically the same as
MapElements.via(<lambda>) and also easy to use with Beam.
In both of the last cases, there are many reasonable approaches, and
we
shouldn't commit our users to one of them.
On Tue, Nov 8, 2016 at 10:15 AM, Lukasz Cwik
<lc...@google.com.invalid
wrote:
The suggestions you give seem good except for the the XML cases.
Might want to have the XML be a document per line similar to the
JSON
examples you have been giving.
On Tue, Nov 8, 2016 at 12:00 PM, Jesse Anderson <
je...@smokinghand.com>
wrote:
@lukasz Agreed there would have to be KV handling. I was more think
that
whatever the addition, it shouldn't just handle KV. It should
handle
Iterables, Lists, Sets, and KVs.
For JSON and XML, I wonder if we'd be able to give someone
something
general purpose enough that you would just end up writing your own
code
to
handle it anyway.
Here are some ideas on what it could look like with a method and
the
resulting string output:
*Stringify.toJSON()*
With KV:
{"key": "value"}
With Iterables:
["one", "two", "three"]
*Stringify.toXML("rootelement")*
With KV:
<rootelement key=value />
With Iterables:
<rootelement>
<item>one</item>
<item>two</item>
<item>three</item>
</rootelement>
*Stringify.toDelimited(",")*
With KV:
key,value
With Iterables:
one,two,three
Do you think that would strike a good balance between reusable
code
and
writing your own for more difficult formatting?
Thanks,
Jesse
On Tue, Nov 8, 2016 at 11:01 AM Lukasz Cwik
<lc...@google.com.invalid
wrote:
Jesse, I believe if one format gets special treatment in TextIO,
people
will then ask why doesn't JSON, XML, ... also not supported.
Also, the example that you provide is using the fact that the
input
format
is an Iterable<Item>. You had posted a question about using KV
with
TextIO.Write which wouldn't align with the proposed input format
and
still
would require to write a type conversion function, this time from
KV
to
Iterable<Item> instead of KV to string.
On Tue, Nov 8, 2016 at 9:50 AM, Jesse Anderson <
je...@smokinghand.com>
wrote:
Lukasz,
I don't think you'd need complicated logic for TextIO.Write. For
CSV
the
call would look like:
Stringify.to("", ",", "\n");
Where the arguments would be Stringify.to(prefix, delimiter,
suffix).
The code would be something like:
StringBuffer buffer = new StringBuffer(prefix);
for (Item item : list) {
buffer.append(item.toString());
if(notLast) {
buffer.append(delimiter);
}
}
buffer.append(suffix);
c.output(buffer.toString());
That would allow you to do the basic CSV, TSV, and other formats
without
complicated logic. The same sort of thing could be done for
TextIO.Write.
Thanks,
Jesse
On Tue, Nov 8, 2016 at 10:30 AM Lukasz Cwik
<lc...@google.com.invalid
wrote:
The conversion from object to string will have uses outside of
just
TextIO.Write so it seems logical that we would want to have a
ParDo
do
the
conversion.
Text file formats have a lot of variance, even if you consider
the
subset
of CSV like formats where it could have fixed width fields, or
escaping
and
quoting around other fields, or headers that should be placed at
the
top.
Having all these format conversions within TextIO.Write seems
like
a
lot
of
logic to contain in that transform which should just focus on
writing
to
files.
On Tue, Nov 8, 2016 at 8:15 AM, Jesse Anderson <
je...@smokinghand.com>
wrote:
This is a thread moved over from the user mailing list.
I think there needs to be a way to convert a PCollection<KV> to
PCollection<String> Conversion.
To do a minimal WordCount, you have to manually convert the KV
to a
String:
p
.apply(TextIO.Read.from("playing_cards.tsv"))
.apply(Regex.split("\\W+"))
.apply(Count.perElement())
* .apply(MapElements.via((KV<String, Long>
count)
->*
* count.getKey() + ":" +
count.getValue()*
* ).withOutputType(
TypeDescriptors.strings()))*
.apply(TextIO.Write.to("output/stringcounts"));
This code really should be something like:
p
.apply(TextIO.Read.from("playing_cards.tsv"))
.apply(Regex.split("\\W+"))
.apply(Count.perElement())
* .apply(ToString.stringify())*
.apply(TextIO.Write.to("output/stringcounts"));
To summarize the discussion:
- JA: Add a method to StringDelegateCoder to output any KV
or
list
- JA and DH: Add a SimpleFunction that takes an type and runs
toString()
on it:
class ToStringFn<InputT> extends SimpleFunction<InputT,
String>
{
public static String apply(InputT input) {
return input.toString();
}
}
- JB: Add a general purpose type converter like in Apache
Camel.
- JA: Add Object support to TextIO.Write that would write out
the
toString of any Object.
My thoughts:
Is converting to a PCollection<String> mostly needed when
you're
using
TextIO.Write? Will a general purpose transform only work in
certain
cases
and you'll normally have to write custom code format the strings
the
way
you want them?
IMHO, it's yes to both. I'd prefer to add Object support to
TextIO.Write
or
a SimpleFunction that takes a delimiter as an argument. Making
a
SimpleFunction that's able to specify a delimiter (and perhaps
a
prefix
and
suffix) should cover the majority of formats and cases.
Thanks,
Jesse
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
