Hi Flinkers,
so, while upgrading to the latest 0.7 snapshot (Scala API) for some
features I need, I ran into the following error (main part:
java.io.NotSerializableException: my.example.FlinkProgram):
Exception in thread "main" org.apache.flink.compiler.CompilerException:
Error translating node 'Map "org.apache.flink.api.scala.DataSet$$anon$1" :
MAP [[ GlobalProperties [partitioning=RANDOM] ]] [[ LocalProperties
[ordering=null, grouped=null, unique=null] ]]': Could not write the user
code wrapper class
org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
java.io.NotSerializableException: my.example.FlinkProgram
at
org.apache.flink.compiler.plantranslate.NepheleJobGraphGenerator.preVisit(NepheleJobGraphGenerator.java:338)
at
org.apache.flink.compiler.plantranslate.NepheleJobGraphGenerator.preVisit(NepheleJobGraphGenerator.java:95)
[many more lines of stacktrace ..]
Questions:
- Why does Flink need to serialize that class? Basically, the class only
consists of some methods, that together build my execution graph and
several data sinks. It does not even have state yet.
- How can I make it serializable? I tried extending Serializable, but that
did not work out.
- What happens, once I add state, that I do not want to serialize?
Just a quick side question, did you remove the count() method for grouped
data sets in the 0.7 release on purpose? If yes, is there a short cut for
counting a grouping, without writing the reduceGroup function by hand?
Thanks in advance
Stefan
