Re: Tagging Flink classes with InterfaceAudience and InterfaceStability

2015-01-30 Thread Henry Saputra
I like the idea. But would love to have different name for the "LimitedPrivate" to make it easier to distinguish. How about "Module" or "Package" ? - Henry On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger wrote: > I think in Hadoop they use LimitedPrivate for the different components of > the pro

Re: Community vote for Hadoop Summit result

2015-01-30 Thread Márton Balassi
The topic will be co-presented by Gyula & myself. Let's keep our fingers crossed for the other Flink talks. :) On Fri, Jan 30, 2015 at 9:50 AM, Till Rohrmann wrote: > Great news Márton. Congrats! > On Jan 30, 2015 2:41 AM, "Henry Saputra" wrote: > >> W00t! Congrats guys! >> >> On Thu, Jan 29, 2

[jira] [Created] (FLINK-1464) Added ResultTypeQueryable interface to TypeSerializerInputFormat.

2015-01-30 Thread Alexander Alexandrov (JIRA)
Alexander Alexandrov created FLINK-1464: --- Summary: Added ResultTypeQueryable interface to TypeSerializerInputFormat. Key: FLINK-1464 URL: https://issues.apache.org/jira/browse/FLINK-1464 Project

Re: Design Question in Expression API

2015-01-30 Thread Aljoscha Krettek
Yes, that's what I'm afraid about. I would also like to have the flexibility to change the underlying implementation in the future, if we want to. And no, hiding the implementation would actually make things easier, implementation wise. On Fri, Jan 30, 2015 at 7:08 PM, Robert Metzger wrote: > Hi,

Re: Design Question in Expression API

2015-01-30 Thread Robert Metzger
Hi, Am I right that you basically fear that if you are allowing users to "manually" modify DataSet's that you're loosing control of the types etc.? I think that integrating the expression API into the existing API is nicer, because it gives users more flexibility. It should also lead to a lower o

Re: ReduceGroup fails on server

2015-01-30 Thread Arvid Heise
Sorry building the 0.9 branch took longer than expected. I will follow that up on Monday. Alternatively I would be grateful for a 0.8.0 patch ;) Best, Arvid On Fri, Jan 30, 2015 at 5:09 PM, Aljoscha Krettek wrote: > Hi Arvid, > I have a fix that I hope fixes your problem: > https://github.com

Re: Timeout while requesting InputSplit

2015-01-30 Thread Till Rohrmann
I've updated the corresponding jira ticket. On Fri, Jan 30, 2015 at 5:46 PM, Till Rohrmann wrote: > I looked into the problem and the problem is a deserialization issue on > the TaskManager side. Somehow the system is not capable to send InputSplits > around whose classes are contained in the us

Re: Timeout while requesting InputSplit

2015-01-30 Thread Till Rohrmann
I looked into the problem and the problem is a deserialization issue on the TaskManager side. Somehow the system is not capable to send InputSplits around whose classes are contained in the user code jars. A similar issue was already observed by Fabian in FLINK-1438. I used his test program and the

Re: ReduceGroup fails on server

2015-01-30 Thread Aljoscha Krettek
Hi Arvid, I have a fix that I hope fixes your problem: https://github.com/aljoscha/flink/tree/serializer-factories-fix Could you try building it and running your example? Cheers, Aljoscha On Fri, Jan 30, 2015 at 3:30 PM, Aljoscha Krettek wrote: > We have a bit of a divide in how we handle TypeS

Re: ReduceGroup fails on server

2015-01-30 Thread Aljoscha Krettek
We have a bit of a divide in how we handle TypeSerializer and TypeComparator: TypeSerializer does not handle duplication but relies on outside code (RuntimeStatefulSerializerFactory) to perform the duplication. TypeComparator does duplication itself. There is also the RuntimeComparatorFactory. This

Re: ReduceGroup fails on server

2015-01-30 Thread Aljoscha Krettek
Maybe we can get rid of the serializer factories altogether. We could enhance the Serializers with a method duplicate() that does nothing for stateless serializers and does a deep copy for stateful serializers. This would also consolidate all the knowledge about stateful/stateless in one place. On

[jira] [Created] (FLINK-1463) RuntimeStatefulSerializerFactory declares ClassLoader as transient but later tries to use it

2015-01-30 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-1463: --- Summary: RuntimeStatefulSerializerFactory declares ClassLoader as transient but later tries to use it Key: FLINK-1463 URL: https://issues.apache.org/jira/browse/FLINK-1463

Re: ReduceGroup fails on server

2015-01-30 Thread Aljoscha Krettek
Of course it doesn't work. The ClassLoader is declared transient in the serializer factory, so it is Null once the factory has been serialized/deserialized once. I open a Jira issue: https://issues.apache.org/jira/browse/FLINK-1463 On Thu, Jan 29, 2015 at 9:43 PM, Stephan Ewen wrote: > Thanks for

Re: Community vote for Hadoop Summit result

2015-01-30 Thread Till Rohrmann
Great news Márton. Congrats! On Jan 30, 2015 2:41 AM, "Henry Saputra" wrote: > W00t! Congrats guys! > > On Thu, Jan 29, 2015 at 4:06 PM, Márton Balassi > wrote: > > Hi everyone, > > > > Thanks for your support for the Flink talks at the community choice for > the > > next Hadoop Summit Europe. T