Re: Why Scala Option is not a valid key?

Ok, I can't make Option comparable, so the only way that I see is to translate a key to a Comparable data structure and use it (as it was alluded to in your example above). Thank you for clarification! Thanks, Timur On Wed, Mar 30, 2016 at 9:22 AM, Chiwan Park wrote: > Hi Timur, > > Sorry for c

Re: Implicit inference of TypeInformation for join keys

Actually, there is an even easier solution (which I saw in your reply to my other question): ``` a.coGroup(b) .where(e => (e.f1, e.f2)) .equalTo(e => (e.f1, e.f2)).apply { (left, right) => 1 }.print() ``` pretty much does what I want. Explicit `apply` gives a hint that a compiler was miss

Re: Why Scala Option is not a valid key?

Hi Timur, Sorry for confusing. I meant KeySelector. `GenericType` could be used as a key type if the `T` implements `Comparable`. For example, `GenericType` could be used as a key type but `GenericType` could not. About my example in previous mail, the type of key is `Int` because the return

Re: Implicit inference of TypeInformation for join keys

Hi Timur, You have to use `createTypeInfomation` method in `org.apache.flink.api` package to create TypeInformation object for Scala-specific objects such as case classes, tuples, eithers, options. For example: ``` import org.apache.flink.api.scala._ // to import package object val a: DataSet[

Re: Why Scala Option is not a valid key?

Thank you for your answers, Chiwan! That would mean that a generic type can't be used as a key in general? This is a non-obvious limitation of Flink DSL that I didn't see in documentation. Could you please elaborate what you mean by KeyExtractor? I see that inside `where` operator an instance of K

Re: Implicit inference of TypeInformation for join keys

Thank you Chiwan! Yes, I understand that there are workarounds that don't use function argument (and thus do not require implicit arguments). I try to avoid positional and string-based keys because there is no compiler guarantees when you refactor or accidentally change the underlying case classes.

BulkIteration and BroadcastVariables

Hi all, I have a question regarding the BulkIteration and BroadcastVariables: The BulkIteration by default has one input variable and sends one variable into the next iteration, right? What if I need to collect some intermediate results in each iteration? How would I do that? For example in my c

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Yes Suneel is completely wright. If the data does not implement IOReadableWritable it is probably easier to use the TypeSerializerOutputFormat. What you need here to seralize the data is a TypeSerializer. You can obtain it the following way: val model = mlr.weightsOption.get val weightVectorTypeI

Re: Kafka Test Error

Hi! A "NoSuchMethodError" usually means that you compile and run against different versions. Make sure the version you reference in the IDE and the version on the cluster are the same. Greetings, Stephan On Wed, Mar 30, 2016 at 9:42 AM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wro

ContinuousProcessingTimeTrigger does not fire

Hi I noticed that ContinuousProcessingTimeTrigger sometimes does not fire. I asked similar question before and applied this patch. https://github.com/apache/flink/commit/607892314edee95da56f4997d85610f17a0dd470#diff-19bbcb3ea1403e483327408badfcd3f8 It looked work but still I have strange behavior

Re: Java 8 and keyBy in 1.0.0

Thanks for the quick reply looks like Chesnay Schepler already found the ticket https://issues.apache.org/jira/browse/FLINK-3138 with status: Of course if it would be great if this can be fixed soon, but this can be a bit more obvious somewh

Re: Java 8 and keyBy in 1.0.0

I will assign this issue to me and fix it soon, if that's ok? Regards, Timo On 30.03.2016 11:30, Stephan Ewen wrote: Looks like something we should fix though. Probably just needs a case distinction in the TypeExtractor. @Andrew, can you post the stack trace into the me linked issue? We'll

Re: Java 8 and keyBy in 1.0.0

Looks like something we should fix though. Probably just needs a case distinction in the TypeExtractor. @Andrew, can you post the stack trace into the me linked issue? We'll try to get at it until the next release... On Wed, Mar 30, 2016 at 10:58 AM, Chesnay Schepler wrote: > based on https://

Re: threads, parallelism and task managers

Hi Ufuk, here is our preliminary input formar implementation: https://gist.github.com/anonymous/dbf05cad2a6cc07b8aa88e74a2c23119 if you need a running project, I will have to create a test one cause I cannot share the current configuration. thanks a lot in advance! 2016-03-30 10:13 GMT+02:00

Re: Java 8 and keyBy in 1.0.0

based on https://issues.apache.org/jira/browse/FLINK-3138 this is not supported for non-static methods. On 30.03.2016 10:33, Andrew Ge Wu wrote: Hi, This is not very obvious and looks like a bug. I have a lambda expression to get key from objects in stream: *This works:* stream.keyBy(value -

Java 8 and keyBy in 1.0.0

Hi, This is not very obvious and looks like a bug. I have a lambda expression to get key from objects in stream: This works: stream.keyBy(value -> value.getId()) This does not: stream.keyBy(myClass::getId) Exception: Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1

Re: threads, parallelism and task managers

Do you have the code somewhere online? Maybe someone can have a quick look over it later. I'm pretty sure that is indeed a problem with the custom input format. – Ufuk On Tue, Mar 29, 2016 at 3:50 PM, Stefano Bortoli wrote: > Perhaps there is a misunderstanding on my side over the parallelism an

Re: Why Scala Option is not a valid key?

Hi Timur, Because Option[T] is not comparable type generally (if T is a POJO type), you cannot use Option[T] as a key type. I think you have to implement KeyExtractor to compare objects including Option[T]s. ``` case class MyKey(k1: Option[String], k2: Option[String]) val data1 = env.fromEleme

Re: Memory Leak using ProcessingTimeTimers?

Hi Aljoscha, thanks for looking into it. I have moved the discussion to the issue. Cheers, Konstantin On 27.03.2016 09:35, Aljoscha Krettek wrote: > Hi, > you are right, this is a problem. In an earlier version we were only > setting very few actual timers using the RuntimeContext because a fir

Re: Kafka Test Error

I have tested kafka 0.8.0.2 with flink 1.0.0 and it works for me. Can't talk about kafka 0.9.0.1. On Wed, Mar 30, 2016 at 12:51 PM, Zhun Shen wrote: > Hi there, > > flink version: 1.0.0 > kafka version: 0.9.0.0 > env: local > > I run the script below: > ./bin/flink run -c com.test.flink.FlinkTes

Kafka Test Error

Hi there, flink version: 1.0.0 kafka version: 0.9.0.0 env: local I run the script below: ./bin/flink run -c com.test.flink.FlinkTest test.jar --topic nginx-logs --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id myGroup --partition.assignment.strategy round robin

Re: Storm topologies compatibility and exactly-once

Thanks for the precision Maximilian. On Tue, Mar 29, 2016 at 11:27 AM, Maximilian Michels wrote: > Hi Olivier, > > Regarding the general question, please have a look at the > documentation: > https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/storm_compatibility.html > Yes, yo

Re: Implicit inference of TypeInformation for join keys

Hi Timur, You can use a composite key [1] to compare keys consisting of multiple fields. For example: ``` val a = env.fromCollection(Seq(Thing("a", "b"), Thing("c", "d"))) val b = env.fromCollection(Seq(Thing("a", "x"), Thing("z", "m"))) a.coGroup(b) .where(“f1”, “f2”) // Flink compares the va