Re: [jira] [Commented] (FLINK-1410) Integrate Flink version variables into website layout

2015-01-22 Thread Till Rohrmann
+1 On Thu, Jan 22, 2015 at 2:42 PM, Max Michels (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/FLINK-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287423#comment-14287423 ] Max Michels commented on FLINK-1410:

Re: Timeout while requesting InputSplit

2015-01-30 Thread Till Rohrmann
I've updated the corresponding jira ticket. On Fri, Jan 30, 2015 at 5:46 PM, Till Rohrmann trohrm...@apache.org wrote: I looked into the problem and the problem is a deserialization issue on the TaskManager side. Somehow the system is not capable to send InputSplits around whose classes

Re: Cluster execution - Jobmanager unreachable

2015-02-05 Thread Till Rohrmann
It looks to me that the TaskManager does not receive a ConsumerNotificationResult after having send the ScheduleOrUpdateConsumers message. This can either mean that something went wrong in ExecutionGraph.scheduleOrUpdateConsumers method or the connection was disassociated for some reasons. The

Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8

2015-02-09 Thread Till Rohrmann
Hi Andra, have you tried increasing the number of network buffers in your cluster? You can control by the configuration value: taskmanager.network.numberOfBuffers: #numberBuffers Greets, Till On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu lungu.an...@gmail.com wrote: Hello everyone, I am

Re: [VOTE] Release Apache Flink 0.8.0 (RC3)

2015-01-16 Thread Till Rohrmann
Hi, I found an issue with the yarn binaries. In flink-0.8.0-bin-hadoop2-yarn.tgz the plan visualizer does not work. The reason is that the resources folder with the javascript files is not copied to flink-dist. I'm a little bit undecided wether this is a blocker or not. It is definitely a bad

Re: Gelly is in!

2015-02-11 Thread Till Rohrmann
Awesome :-) On Wed, Feb 11, 2015 at 4:27 PM, Paris Carbone par...@kth.se wrote: Congratulations! Very exciting! Paris On 11 Feb 2015, at 15:24, Ufuk Celebi u...@apache.org wrote: Superb. :-) On 11 Feb 2015, at 15:00, Kostas Tzoumas ktzou...@apache.org wrote: Nice!!! Welcome

Re: [SUGGESTION] Push latest doc to Flink website

2015-02-19 Thread Till Rohrmann
+1 Definitely very helpful for users and developers. On Wed, Feb 18, 2015 at 5:21 PM, Stephan Ewen se...@apache.org wrote: +1 The website should have the latest stable docs and the latest snapshot docs. The snapshot docs need not be daily up to date for a start. On Wed, Feb 18, 2015 at

Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-19 Thread Till Rohrmann
Dependency conflicts were also the reason why we have to use a different Akka version for the Hadoop 2.0.0-alpha build profile. Thus, +1. On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger rmetz...@apache.org wrote: I'm also in favor of shading commonly used libraries to resolve this issue for

Re: [DISCUSS] Scala code style - explicit vs implicit code behavior

2015-02-16 Thread Till Rohrmann
+1 On Mon, Feb 16, 2015 at 3:38 PM, Aljoscha Krettek aljos...@apache.org wrote: +1 On Mon, Feb 16, 2015 at 3:18 PM, Fabian Hueske fhue...@gmail.com wrote: +1 2015-02-15 17:47 GMT+01:00 Stephan Ewen se...@apache.org: I thought about adding a wiki page for that. On Sat, Feb 14,

Re: [DISCUSS] Dedicated streaming mode and start scripts

2015-02-17 Thread Till Rohrmann
+1 On Tue, Feb 17, 2015 at 1:34 PM, Kostas Tzoumas ktzou...@apache.org wrote: +1 On Tue, Feb 17, 2015 at 12:14 PM, Márton Balassi mbala...@apache.org wrote: When it comes to the current use cases I'm for this separation. @Ufuk: As Gyula has already pointed out with the current design of

Re: Timeout while requesting InputSplit

2015-01-28 Thread Till Rohrmann
I think that the machines have lost connection. That is most likely connected to the heartbeat interval of the watch or transport failure detector. The transport failure detector should actually be set to a heartbeat interval of 1000 s and consequently it should not cause any problems. Which

Re: Timeout while requesting InputSplit

2015-01-29 Thread Till Rohrmann
Yes actually the timeouts should not really matter. However, an exception in the InputSplitAssigner should happen in the actor thread and thus cause the actor to stop. This should be logged by the supervisor. I just checked and the method InputSplitAssigner.getNextInputSplit is not supposed to

Re: Kicking off the Machine Learning Library

2015-02-01 Thread Till Rohrmann
Hi Manu, I looked into it and I'm also working on the integration. When I started my first try, Flink had the problem to not properly support interfaces and subclasses. This was relevant, because the distributed row-wise partitioned matrices can be indexed by int keys or string keys. By now, this

Re: December in Flink

2015-01-10 Thread Till Rohrmann
I like the idea of having a news list as well :-) +1 On Fri, Jan 9, 2015 at 7:36 PM, Ted Dunning ted.dunn...@gmail.com wrote: Would the user list do for now? On Fri, Jan 9, 2015 at 7:27 AM, Robert Metzger rmetz...@apache.org wrote: Our PMC Chair or a ASF member has to request a list:

Re: [VOTE] Release Apache Flink 0.8.0 (RC1)

2015-01-12 Thread Till Rohrmann
Yeah I agree with that. On Mon, Jan 12, 2015 at 11:30 AM, Ufuk Celebi u...@apache.org wrote: On Mon, Jan 12, 2015 at 11:22 AM, Stephan Ewen se...@apache.org wrote: It would be good to have the patch, but it is also a very tricky patch, so pushing it hastily may be problematic. I

Re: Cluster execution - Jobmanager unreachable

2015-02-11 Thread Till Rohrmann
I found the error. Due to some refactoring, a wrong message was sent to the JobManager in the JobManagerInfoServlet.java. I pushed a fix. Could you try it out again? On Wed, Feb 11, 2015 at 11:34 AM, Till Rohrmann till.rohrm...@gmail.com wrote: Could you check the rebasing because it seems

Re: Google Summer of Code 2015 is coming

2015-02-12 Thread Till Rohrmann
I can add a GSoC issue for the ML library. But it will be a bit vague in terms of what can be done. On Thu, Feb 12, 2015 at 3:13 PM, Márton Balassi balassi.mar...@gmail.com wrote: Hey, Please add the JIRA for the projects, the deadline for the organizations is tomorrow! Gyula has already

Re: AW: kryoException : Buffer underflow

2015-02-12 Thread Till Rohrmann
The kryo underflow should be fixed with the PR [1]. [1] https://github.com/apache/flink/pull/391 On Thu, Feb 12, 2015 at 4:10 PM, Nam-Luc Tran namluc.t...@euranova.eu wrote: Without the .returns(...) statement it yelled about type erasure. Putting.returns(Centroid25.class) did the trick.

Re: Cluster execution - Jobmanager unreachable

2015-02-05 Thread Till Rohrmann
, 2015 at 10:40 AM, Till Rohrmann trohrm...@apache.org wrote: It looks to me that the TaskManager does not receive a ConsumerNotificationResult after having send the ScheduleOrUpdateConsumers message. This can either mean that something went wrong in ExecutionGraph.scheduleOrUpdateConsumers

Re: A typo in a documentation

2015-03-18 Thread Till Rohrmann
Good catch Rui Zhu. Thanks a lot, I'll fix it. On Wed, Mar 18, 2015 at 2:21 AM, Rui Zhu rui.tyler@gmail.com wrote: Hello, I just found a typo in the document of Cluster Setup. In the HDFS Setup section of Cluster Setup, the command of starting HDFS has a typo: when we go into the

Re: [DISCUSS] Issues with heterogeneity of the code

2015-03-16 Thread Till Rohrmann
+1 for stricter Java code styles. I haven't looked into the Google Code Style but maybe we make it easier for new contributors if we apply a coding style which is somehow known. +1 for line length of 100 for Scala code. I think it makes code review on GitHub easier. For the Scala style, we could

Re: [DISCUSS] Issues with heterogeneity of the code

2015-03-16 Thread Till Rohrmann
Do we already enforce the official Scala style guide strictly? On Mon, Mar 16, 2015 at 4:57 PM, Aljoscha Krettek aljos...@apache.org wrote: I'm already always sticking to the official Scala style guide, with the exception of 100 line length. On Mar 16, 2015 3:27 PM, Till Rohrmann trohrm

Re: Could not build up connection to JobManager

2015-03-16 Thread Till Rohrmann
https://gist.github.com/viduranga/e7549ef818c6a2af73e9#file-flink-vidura-jobmanager-localhost-log On Mar 11, 2015, at 11:32 PM, Till Rohrmann trohrm...@apache.org wrote: Hi Dulaj, sorry for my late response. It looks as if the JobClient tries to connect to the JobManager

Re: Could not build up connection to JobManager

2015-03-16 Thread Till Rohrmann
tried with it is shut down as well) and also I doubly checked hosts files. I had little snitch installed but I also tried uninstalling it. Isn’t there a way around without using DNS to resolve localhost? On Mar 16, 2015, at 10:04 PM, Till Rohrmann trohrm...@apache.org wrote: It is really

Re: Introduction of new Gelly developper

2015-03-17 Thread Till Rohrmann
Hi Zhou Yi, welcome to the Flink community. Great to hear that you're gonna work on Gelly. If you have any problems getting started, then let us know. Cheers, Till On Tue, Mar 17, 2015 at 9:22 AM, Stephan Ewen se...@apache.org wrote: Hi Zhou Yi! Welcome to the Flink community. Gelly (and

Re: Restructuring the maven projects

2015-03-17 Thread Till Rohrmann
Putting the Scala and Java API into the same module means that we'll have more mixed Java/Scala projects, right? I just want to check if everyone is aware of it considering our latest experiences with these kind of modules. On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi u...@apache.org wrote: +1

Re: [DISCUSS] Documentation Java/Scala order

2015-03-09 Thread Till Rohrmann
+1 for Scala :-) On Sat, Mar 7, 2015 at 1:56 PM, Márton Balassi balassi.mar...@gmail.com wrote: I'm strongly for consistency and personally would prefer Scala as a default - thus making the shorter page the default. On Sat, Mar 7, 2015 at 1:47 PM, Stephan Ewen se...@apache.org wrote: I

Re: Could not build up connection to JobManager

2015-03-11 Thread Till Rohrmann
Hi Dulaj, sorry for my late response. It looks as if the JobClient tries to connect to the JobManager using its IPv6 instead of IPv4. Akka is really picky when it comes to remote address. If Akka binds to the FQDN, then other ActorSystem which try to connect to it using its IP address won't be

Re: Running example in IntelliJ

2015-03-06 Thread Till Rohrmann
Yes, this means that a task has finished its computation and can be removed from the TaskManager. On Fri, Mar 6, 2015 at 11:44 AM, Dulaj Viduranga vidura...@icloud.com wrote: Thank you all. IntelliJ shows Unregister task with execution ID (something)” couple of times in the output. But I guess

Re: [DISCUSS] Make a release to be announced at ApacheCon

2015-03-12 Thread Till Rohrmann
Have you run the 20 builds with the new shading code? With new shading the TaskManagerFailsITCase should no longer fail. If it still does, then we have to look into it again. On Thu, Mar 12, 2015 at 2:01 PM, Stephan Ewen se...@apache.org wrote: I am also big time skeptical. There are some

Re: [jira] [Commented] (FLINK-1106) Deprecate old Record API

2015-03-10 Thread Till Rohrmann
+1 for removal of old API On Mar 10, 2015 5:41 PM, Fabian Hueske fhue...@gmail.com wrote: And I'm +1 for removing the old API with the next release. 2015-03-10 17:38 GMT+01:00 Fabian Hueske fhue...@gmail.com: Yeah, I spotted a good amount of optimizer tests that depend on the Record API.

Re: [DISCUSS] Issues with heterogeneity of the code

2015-03-09 Thread Till Rohrmann
I also agree that we have too many different ways of doing things. A set of common rules/ways would definitely be beneficial for the project. Concerning the command line parsing: I thought that Alexander Alexandrov wanted to unify the command line parsing by replacing both tools with a better

Re: TaskManagerFailsITCase gets stuck

2015-03-25 Thread Till Rohrmann
What do the logs say? It looks as if there is some issue with the TaskManager start up because the main is in the method waitForTaskManagersToBeRegistered. This happens for the initial ForkableFlinkMiniCluster start. On Wed, Mar 25, 2015 at 1:50 PM, Ufuk Celebi u...@apache.org wrote: I saw a

Re: [VOTE] Name of Expression API Representation

2015-03-26 Thread Till Rohrmann
+Table On Thu, Mar 26, 2015 at 9:32 AM, Márton Balassi balassi.mar...@gmail.com wrote: +DataTable On Thu, Mar 26, 2015 at 9:29 AM, Markl, Volker, Prof. Dr. volker.ma...@tu-berlin.de wrote: +Table I also agree with that line of argument (think SQL ;-) ) -Ursprüngliche

Re: [DISCUSS] Add a Beta badge in the documentation to components in flink-staging

2015-03-29 Thread Till Rohrmann
+1 On Sun, Mar 29, 2015 at 5:04 PM, Chiwan Park chiwanp...@icloud.com wrote: +1 Good idea. Users can accept API changes of “flink-staging” module with “Beta badge. Regards. Chiwan Park (Sent with iPhone) On Mar 29, 2015, at 11:38 PM, Robert Metzger rmetz...@apache.org wrote: Hi,

Re: A small Project I've been working on

2015-04-01 Thread Till Rohrmann
Where can I start contributing. I've got the feeling that it will be inherently faster than everything else which is out there. On Apr 1, 2015 9:18 AM, Ufuk Celebi u...@apache.org wrote: Little side projects ftw. Very nice :-) Can you give some points on how this works internally? Is it making

Re: Make docs searchable

2015-04-01 Thread Till Rohrmann
I also like the idea. +1 On Wed, Apr 1, 2015 at 12:20 PM, Robert Metzger rmetz...@apache.org wrote: Cool. I would like to have the ability to search the docs, so +1 for this idea! On Wed, Apr 1, 2015 at 12:10 PM, Ufuk Celebi u...@apache.org wrote: Hey all, I think our documentation

Re: [DISCUSS] Make a release to be announced at ApacheCon

2015-03-27 Thread Till Rohrmann
+1 for 0.9.0-milestone-1 On Fri, Mar 27, 2015 at 3:52 PM, Kostas Tzoumas ktzou...@apache.org wrote: +1 On Fri, Mar 27, 2015 at 3:44 PM, Aljoscha Krettek aljos...@apache.org wrote: +1 for 0.9.0-M1 (or milestone-1) On Mar 27, 2015 2:45 PM, Ufuk Celebi u...@apache.org wrote: On

Re: [DISCUSS] Offer Flink with Scala 2.11

2015-03-02 Thread Till Rohrmann
+1 for Scala 2.11 On Mon, Mar 2, 2015 at 5:02 PM, Alexander Alexandrov alexander.s.alexand...@gmail.com wrote: Spark currently only provides pre-builds for 2.10 and requires custom build for 2.11. Not sure whether this is the best idea, but I can see the benefits from a project management

Re: Not Flink related: IntelliJ cannot open maven projects

2015-03-04 Thread Till Rohrmann
That is odd. Most of the committers are using the IntelliJ to develop Flink. Have you tried to delete the Flink directory and checking it out again. Make sure that all IntelliJ related files (*.iml) are deleted so that it is really a fresh import. I just cloned the flink repository and imported

Re: Could not build up connection to JobManager

2015-03-05 Thread Till Rohrmann
. # # You can also directly specify the paths to hdfs-default.xml and hdfs-site.xml # via keys 'fs.hdfs.hdfsdefault' and 'fs.hdfs.hdfssite'. # # fs.hdfs.hadoopconf: /path/to/hadoop/conf/ On Mar 5, 2015, at 2:03 PM, Till Rohrmann trohrm...@apache.org wrote: How did you start the flink

Re: [jira] [Created] (FLINK-1651) Running mvn test got stuck

2015-03-04 Thread Till Rohrmann
, 2015 at 11:02 PM, Till Rohrmann trohrm...@apache.org wrote: Is this reproducible? If so, then a stack trace of the JVM would be helpful. With the stack trace we would know which test case stalls. On Wed, Mar 4, 2015 at 9:46 PM, Henry Saputra (JIRA) j...@apache.org wrote: Henry

Re: Could not build up connection to JobManager

2015-03-04 Thread Till Rohrmann
Hi Dulaj, I looked through your commit and noticed that the JobClient might not be listening on the right network interface. Your commit seems to fix it. I just want to understand the problem properly and therefore I opened a branch with a small change. Could you try out whether this change would

Re: [jira] [Created] (FLINK-1651) Running mvn test got stuck

2015-03-04 Thread Till Rohrmann
Is this reproducible? If so, then a stack trace of the JVM would be helpful. With the stack trace we would know which test case stalls. On Wed, Mar 4, 2015 at 9:46 PM, Henry Saputra (JIRA) j...@apache.org wrote: Henry Saputra created FLINK-1651:

Re: [DISCUSS] URI NullPointerException in TestBaseUtils

2015-02-27 Thread Till Rohrmann
Catching the NullPointerException and throwing an IllegalArgumentException with a meaningful message might clarify things. Considering that it only affects the TestBaseUtils, it should not be big deal to change it. On Fri, Feb 27, 2015 at 10:30 AM, Szabó Péter nemderogator...@gmail.com wrote:

Re: Could not build up connection to JobManager

2015-02-27 Thread Till Rohrmann
It depends on how you started Flink. If you started a local cluster, then the TaskManager log is contained in the JobManager log we just don't see the respective log output in the snippet you posted. If you started a TaskManager independently, either by taskmanager.sh or by start-cluster.sh, then

Re: Could not build up connection to JobManager

2015-03-05 Thread Till Rohrmann
, Till Rohrmann trohrm...@apache.org wrote: What does the jobmanager log says? I think Stephan added some more logging output which helps us to debug this problem. On Thu, Mar 5, 2015 at 9:36 AM, Dulaj Viduranga vidura...@icloud.com wrote: Using start-locat.sh. I’m using the original

Re: Log4j configuration JUnit vs. Scalatest in IntelliJ

2015-02-25 Thread Till Rohrmann
The reason for this behaviour is the following: The log4j-test.properties is not a standard log4j properties file. It is only used if it is explicitly given to the executing JVM by -Dlog4j.configuration. The parent pom defines for the surefire plugin a corresponding argument line with this

Re: Questions about flink-streaming-examples

2015-02-25 Thread Till Rohrmann
Try to do first a mvn install on flink-parent. On Wed, Feb 25, 2015 at 4:54 PM, Max Michels m...@apache.org wrote: Hi Matthias, Did you really pull from the latest master? I just tried to compile flink-streaming-examples using mvn clean compile test-compile and it worked. Best regards,

Re: Questions about flink-streaming-examples

2015-02-26 Thread Till Rohrmann
If the streaming-examples module uses the classifier tag to add the test-core dependency then we should change it into type tag as recommended by maven [1]. Otherwise it might come to build failures if the install lifecycle is not executed. The dependency import should look like: dependency

Re: Collect of user defined classes doesn't work

2015-04-14 Thread Till Rohrmann
Could you check the definition of the collect method in the DataSet.scala file? Does it contain parentheses or not? On Tue, Apr 14, 2015 at 3:48 PM, Felix Neutatz neut...@googlemail.com wrote: I use the latest maven snapshot: dependency groupIdorg.apache.flink/groupId

Re: Merge Python API

2015-04-20 Thread Till Rohrmann
+1 On Mon, Apr 20, 2015 at 2:50 PM, Timo Walther twal...@apache.org wrote: +1 On 20.04.2015 14:49, Gyula Fóra wrote: +1 On Mon, Apr 20, 2015 at 2:41 PM, Fabian Hueske fhue...@gmail.com wrote: +1 2015-04-20 14:39 GMT+02:00 Maximilian Michels m...@apache.org: +1 Let's merge it to

Re: [DISCUSS] Change semantics of print() to eager

2015-04-28 Thread Till Rohrmann
+1 On Tue, Apr 28, 2015 at 3:19 PM, Maximilian Michels m...@apache.org wrote: +1 Very nice addition. On Tue, Apr 28, 2015 at 2:12 PM, Stephan Ewen se...@apache.org wrote: Sounds good, Max, let's to this in one fix. We can maintain a counter in the ExecutionEnvironment that tracks how

Re: Failing Builds on Travis

2015-04-28 Thread Till Rohrmann
Concerning the failed builds in the hadoop2.0.0-alpha profile I see a lot of 07:47:57,927 ERROR akka.actor.ActorSystemImpl - Uncaught fatal error from thread [flink-akka.remote.default-remote-dispatcher-7] shutting down ActorSystem [flink] java.lang.VerifyError: (class:

Re: Join with a custom predicate

2015-04-27 Thread Till Rohrmann
That's a good solution. In order to deal with ranges which overlap two intervals you have to create multiple coarse-grained join keys. One key for each interval contained in the range. Cheers, Till On Apr 26, 2015 11:22 PM, Alexander Alexandrov alexander.s.alexand...@gmail.com wrote: I thought

Re: outofmemory error. help.

2015-04-28 Thread Till Rohrmann
Hi, we would need a little bit more of background on the job you're running and the cluster setup to help you. Could you please post this information on the u...@flink.apache.org ML where this belongs to? Cheers, Till On Tue, Apr 28, 2015 at 8:45 AM, 东方不败 dashujudechunt...@163.com wrote: I am

Re: About Operator and OperatorBase

2015-04-15 Thread Till Rohrmann
I would also be in favour of making the distinction between the API and common API layer more clear by using different names. This will ease the understanding of the source code. In the wake of a possible renaming we could also get rid of the legacy code org.apache.flink.optimizer.dag.MatchNode

Re: TableAPI - Join on two keys

2015-04-17 Thread Till Rohrmann
Why not doing two separate joins, union the results and doing a distinct operation on the combined key? On Fri, Apr 17, 2015 at 9:42 AM, Aljoscha Krettek aljos...@apache.org wrote: So, the first thing is a feature of the Java API that removes duplicate fields in keys, so an equi-join on (0,0)

Re: 答复: [VOTE] Release Apache Flink 0.9.0-milestone-1 (RC1)

2015-04-09 Thread Till Rohrmann
+1 I thoroughly tested the Flink on Mac OS: - I ran the start-cluster.sh, stop-cluster.sh and start-webclient.sh scripts - Started 1 JM and 2 TM - Checked the logs and out files for exceptions and error messages - Ran the all examples using the /bin/flink - Ran the wordcount and pagerank example

Re: [VOTE] Release Apache Flink 0.9.0-milestone-1 (RC1)

2015-04-12 Thread Till Rohrmann
Thanks for your great work Ufuk :-) On Apr 12, 2015 5:27 PM, Henry Saputra henry.sapu...@gmail.com wrote: This is great news! Thanks for driving th release, Ufuk. Sorry missed verifying the release this time. - Henry On Sunday, April 12, 2015, Ufuk Celebi u...@apache.org wrote: Hey all,

Re: [jira] [Created] (FLINK-1831) runtime.taskmanager.RegistrationTests fails sporiously

2015-04-07 Thread Till Rohrmann
The error looks as if there is already another JobManager started with FAKE_JOB_MANAGER name. This might be caused by a JobManager which has not yet completely shut down. On Tue, Apr 7, 2015 at 9:52 AM, Márton Balassi (JIRA) j...@apache.org wrote: Márton Balassi created FLINK-1831:

Re: MultipleLinearRegression - Strange results

2015-06-02 Thread Till Rohrmann
Great to hear. This should no longer be a pain point once we support proper cross validation. On Tue, Jun 2, 2015 at 11:11 AM, Felix Neutatz neut...@googlemail.com wrote: Yes, grid search solved the problem :) 2015-06-02 11:07 GMT+02:00 Till Rohrmann till.rohrm...@gmail.com: The SGD

Re: MultipleLinearRegression - Strange results

2015-06-02 Thread Till Rohrmann
... 2015-06-01 20:33 GMT+10:00 Till Rohrmann trohrm...@apache.org: Since MLR uses stochastic gradient descent, you probably have to configure the step size right. SGD is very sensitive to the right step size choice. If the step size is too high, then the SGD algorithm does not converge

Re: Problem with ML pipeline

2015-06-04 Thread Till Rohrmann
I see your problem. One way to solve the problem is to implement a special PredictOperation which takes a tuple (id, vector) and returns a tuple (id, labeledVector). You can take a look at the implementation for the vector prediction operation. But we can also discuss about adding an ID field to

Re: ALS implementation

2015-06-04 Thread Till Rohrmann
(HashPartition.java:310) ... Best regards, Felix 2015-06-04 10:19 GMT+02:00 Felix Neutatz neut...@googlemail.com: Yes, I will try it again with the newest update :) 2015-06-04 10:17 GMT+02:00 Till Rohrmann till.rohrm...@gmail.com: If the first error is not fixed by Chiwans

Re: ALS implementation

2015-06-09 Thread Till Rohrmann
strategy here because one cannot get rid of the duplicate join keys. ​ On Mon, Jun 8, 2015 at 1:59 PM Till Rohrmann trohrm...@apache.org wrote: Hi Felix, I tried to reproduce the problem with the *Hash join exceeded maximum number of recursions, without reducing partitions enough to be memory

Re: Testing Apache Flink 0.9.0-rc1

2015-06-09 Thread Till Rohrmann
I also encountered a failing TaskManagerFailsWithSlotSharingITCase using Java8. I could, however, not reproduce the error a second time. The stack trace is: The JobManager should handle hard failing task manager with slot

Re: Checkstyle in IntelliJ

2015-06-09 Thread Till Rohrmann
Hi Pieter-Jan, I'm not aware of an Eclipse or IntellJ auto format profile. I think that all Flink contributors apply their style changes manually. The maven output should tell you quite precisely what's wrong and in which file the checkstyle errors occur. Moreover, applying an IDE auto format is

Re: Checkstyle in IntelliJ

2015-06-09 Thread Till Rohrmann
already been formatted, it will show up changed in IntelliJ but Git will recognize that it is in fact unmodified. That way, we would no longer touch files we've not actually modified. Regards, Pieter-Jan Van Aeken Op Dinsdag, 09/06/2015 om 11:10 schreef Till Rohrmann: Hi

Re: ALS implementation

2015-06-05 Thread Till Rohrmann
can be found here: https://github.com/FelixNeutatz/IMPRO-3.SS15/blob/8b679f1c2808a2c6d6900824409fbd47e8bed826/NullPointerException.txt Best regards, Felix 2015-06-04 19:41 GMT+02:00 Till Rohrmann till.rohrm...@gmail.com: I think it is not a problem of join hints, but rather of too

Re: Problem with ML pipeline

2015-06-06 Thread Till Rohrmann
better because it is only in some cases necessary to return the id. The special predict Operation would save this overhead. Best regards, Felix Am 04.06.2015 7:56 nachm. schrieb Till Rohrmann till.rohrm...@gmail.com : I see your problem. One way to solve the problem is to implement

Re: Problem with ML pipeline

2015-06-08 Thread Till Rohrmann
classes of the pipeline as well, in order to be able to pass the ID through the whole pipeline. Best regards, Felix Am 06.06.2015 9:46 vorm. schrieb Till Rohrmann trohrm...@apache.org : Then you only have to provide an implicit PredictOperation[SVM, (T, Int), (LabeledVector, Int

Re: Problem with ML pipeline

2015-06-08 Thread Till Rohrmann
AM, Till Rohrmann till.rohrm...@gmail.com wrote: You're right Felix. You need to provide the `FitOperation` and `PredictOperation` for the `Predictor` you want to use and the `FitOperation` and `TransformOperation` for all `Transformer`s you want to chain in front of the `Predictor

Re: ALS implementation

2015-06-08 Thread Till Rohrmann
this solves your problem. Cheers, Till ​ On Sat, Jun 6, 2015 at 1:32 AM Till Rohrmann trohrm...@apache.org wrote: I'll look into it to find the responsible join operation. On Jun 5, 2015 10:50 AM, Stephan Ewen se...@apache.org wrote: There are two different issues here: 1) Flink does figure out

Re: JIRA guidelines

2015-06-03 Thread Till Rohrmann
Hi Lokesh, there are no real guidelines for opening JIRA issues. You should select the right project to which assign the JIRA, namely Flink. Then choose an appropriate type, e.g. bug, improvement or feature. Give a descriptive summary of the problem and if necessary a further explanation in

Re: The correct location for zipWithIndex and zipWithUniqueId

2015-06-10 Thread Till Rohrmann
I agree with Theo. I think it’s a nice feature to have as part of the standard API because only few users will be aware of something like DataSetUtils. However, as a first version we can make it part of DataSetUtils. Cheers, Till ​ On Wed, Jun 10, 2015 at 11:52 AM Theodore Vasiloudis

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Till Rohrmann
Btw: I noticed that all streaming modules depend on flink-core, flink-runtime, flink-clients and flink-java. Is there a particular reason why the streaming connectors depend on flink-clients and flink-java? On Wed, Jun 10, 2015 at 3:41 PM Till Rohrmann trohrm...@apache.org wrote: I see

Re: MultipleLinearRegression - Strange results

2015-06-04 Thread Till Rohrmann
that they have to tweak this parameter. On Thu, Jun 4, 2015 at 2:54 PM, Ted Dunning ted.dunn...@gmail.com wrote: On Thu, Jun 4, 2015 at 1:26 PM, Till Rohrmann trohrm...@apache.org wrote: Maybe also the default learning rate of 0.1 is set too high. Could be. But grid search on learning rate

Re: GSoC work

2015-06-04 Thread Till Rohrmann
+1 :-) On Wed, Jun 3, 2015 at 4:53 PM, Vasiliki Kalavri vasilikikala...@gmail.com wrote: Hi Sachin, great idea to keep a blog! Thanks a lot for sharing :)) -V. On 3 June 2015 at 16:41, Sachin Goel sachingoel0...@gmail.com wrote: Hi everyone I'm maintaining a blog detailing my work

Re: Failing tests policy

2015-06-04 Thread Till Rohrmann
I'm also in favour of quickly fixing the failing test cases but I think that blocking the master is a kind of drastic measure. IMO this creates a culture of blaming someone whereas I would prefer a more proactive approach. When you see a failing test case and know that someone recently worked on

Re: ALS implementation

2015-06-04 Thread Till Rohrmann
If the first error is not fixed by Chiwans PR, then we should create a JIRA for it to not forget it. @Felix: Chiwan's PR is here [1]. Could you try to run ALS again with this version? Cheers, Till [1] https://github.com/apache/flink/pull/751 On Thu, Jun 4, 2015 at 10:10 AM, Chiwan Park

Re: MultipleLinearRegression - Strange results

2015-06-04 Thread Till Rohrmann
wrote: We should probably look into this nevertheless. Requiring full grid search for a simple algorithm like mlr sounds like overkill. Do you have written down the math of your implementation somewhere? -M - Ursprüngliche Nachricht - Von: Till Rohrmann till.rohrm

Re: Drafting the 0.9.0 release announcement

2015-06-24 Thread Till Rohrmann
We should ping the Zeppelin guys to update their Flink dependency. On Wed, Jun 24, 2015 at 2:34 PM, Maximilian Michels m...@apache.org wrote: I'm so happy we have pushed it out :) It took a while but I think we can be very pleased with the result. I will post an announcement to the user/dev

Re: Failing Builds on Travis

2015-06-24 Thread Till Rohrmann
+1 On Tue, Jun 23, 2015 at 3:16 PM, Robert Metzger rmetz...@apache.org wrote: +1 On Tue, Jun 23, 2015 at 11:31 AM, Fabian Hueske fhue...@gmail.com wrote: +1 2015-06-22 17:44 GMT+02:00 Stephan Ewen se...@apache.org: +1 On Fri, Jun 19, 2015 at 10:48 AM, Matthias J. Sax

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
? Cheers, Fabian 2015-06-19 15:08 GMT+02:00 Till Rohrmann trohrm...@apache.org: What does forever mean? Usually it's the case that you see a steep decline in performance once the system starts spilling data to disk because of the disk I/O bottleneck. The system always starts spilling to disk

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
question for today, promise :) Thanks! On Fri, Jun 19, 2015 at 2:40 PM, Till Rohrmann trohrm...@apache.org wrote: Yes, it was an issue for the milestone release. On Fri, Jun 19, 2015 at 2:18 PM Andra Lungu lungu.an...@gmail.com wrote: Yes, so I am using flink-0.9.0-milestone-1

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
Hi Andra, the problem seems to be that the deployment of some tasks takes longer than 100s. From the stack trace it looks as if you're not using the latest master. We had problems with previous version where the deployment call waited for the TM to completely download the user code jars. For

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
, Till Rohrmann trohrm...@apache.org wrote: Hi Andra, the problem seems to be that the deployment of some tasks takes longer than 100s. From the stack trace it looks as if you're not using the latest master. We had problems with previous version where the deployment call waited

Re: Apache Flink 0.9 ALS API

2015-06-15 Thread Till Rohrmann
+1 for using long for both IDs. But I don't understand what's the advantage of using a String as user ID. On Sun, Jun 14, 2015 at 6:43 PM Robert Metzger rmetz...@apache.org wrote: Hi Ronny, I accepted your previous mail to the mailing list, you got two replies:

Re: Apache Flink 0.9 ALS API

2015-06-15 Thread Till Rohrmann
+1 for longs as IDs. Not so much in favour of Strings for the user ID because the row index could also denote the actual item ID if you swap the indices. Furthermore, you can always add a transformer which assigns unique IDs to names. Cheers, Till On Sat, Jun 13, 2015 at 3:34 PM Chiwan Park

Re: Testing Apache Flink 0.9.0-rc2

2015-06-15 Thread Till Rohrmann
I might have found another release blocker. While running some cluster tests I also tried to run the `ConnectedComponents` example. However, sometimes the example couldn't be executed because the scheduler could not schedule co-located tasks, `NoResourceAvailableException`, even though it should

Re: Testing Apache Flink 0.9.0-rc2

2015-06-16 Thread Till Rohrmann
release blocker and we need to fix it. On Mon, Jun 15, 2015 at 5:04 PM, Till Rohrmann trohrm...@apache.org wrote: I might have found another release blocker. While running some cluster tests I also tried to run the `ConnectedComponents` example

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
. Furthermore, this also applies to Gelly and FlinkML. Cheers, Till On Fri, Jun 12, 2015 at 9:16 AM Till Rohrmann trohrm...@apache.org wrote: I'm currently going through the license file and I discovered some skeletons in our closet. This has to be merged as well. But I'm still working on it (we

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
What about the shaded jars? On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi u...@apache.org wrote: @Max: for the new RC. Can you make sure to set the variables correctly with regard to stable/snapshot versions in the docs?

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
are in. Plus, we need to include all Flink libraries in flink-dist. Are you going to fix that as well, Till? On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi u...@apache.org wrote: On 12 Jun 2015, at 09:45, Till Rohrmann trohrm...@apache.org wrote: Hi guys, I just noticed

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
, 2015 at 10:29 AM Till Rohrmann trohrm...@apache.org wrote: Well I think the initial idea was to keep the dist jar as small a possible and therefore we did not include the libraries. I'm not sure whether we can decide this here ad-hoc. If the community says that we shall include these libraries

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
I'm currently going through the license file and I discovered some skeletons in our closet. This has to be merged as well. But I'm still working on it (we have a lot of dependencies). Cheers, Till On Fri, Jun 12, 2015 at 12:51 AM Ufuk Celebi u...@apache.org wrote: On 12 Jun 2015, at 00:49,

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
it with the LICENSE [either we find something before the LICENSE update or we only have to review the LICENSE change] Since this is not a vote yet, it doesn't really matter, but I'm leaning towards b). On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann till.rohrm...@gmail.com wrote: What about

Re: Testing Apache Flink 0.9.0-rc1

2015-06-12 Thread Till Rohrmann
, Jun 12, 2015 at 9:44 AM, Till Rohrmann trohrm...@apache.org wrote: I've finished the legal check of the source and binary distribution. The PR with the LICENSE and NOTICE file updates can be found here [1]. What I haven't done yet is addressing the issue with the shaded dependencies

Re: The correct location for zipWithIndex and zipWithUniqueId

2015-06-12 Thread Till Rohrmann
Transformations page poining to this... What do you think? On Wed, Jun 10, 2015 at 12:33 PM, Till Rohrmann till.rohrm...@gmail.com wrote: I agree with Theo. I think it’s a nice feature to have as part of the standard API because only few users will be aware of something like DataSetUtils

  1   2   3   4   5   6   7   8   9   10   >