Re: Testing Apache Flink 0.9.0-rc1

it. On Fri, Jun 12, 2015 at 10:29 AM Till Rohrmann wrote: > Well I think the initial idea was to keep the dist jar as small a possible > and therefore we did not include the libraries. I'm not sure whether we can > decide this here ad-hoc. If the community says that we shall includ

Re: Testing Apache Flink 0.9.0-rc1

Yes you're right Ufuk. At the moment the user has to place the jars in the lib folder of Flink. If this folder is not shared then he has to do it for every node on which Flink runs. On Fri, Jun 12, 2015 at 10:42 AM Till Rohrmann wrote: > I think I found a real release blocker. Curr

Re: Testing Apache Flink 0.9.0-rc1

What about the shaded jars? On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi wrote: > @Max: for the new RC. Can you make sure to set the variables correctly > with regard to stable/snapshot versions in the docs?

Re: Testing Apache Flink 0.9.0-rc1

n update it with the > LICENSE [either we find something before the LICENSE update or we only have > to review the LICENSE change] > > Since this is not a vote yet, it doesn't really matter, but I'm leaning > towards b). > > > On Fri, Jun 12, 2015 at 11:43 AM, Ti

Listing Apache-2.0 dependencies in LICENSE file

Hi guys, I just updated our LICENSE of the binary distribution and noticed that we also list dependencies which are licensed under Apache-2.0. As far as I understand the ASF guidelines [1], this is not strictly necessary. Since it is a lot of work to keep the list up to date, I was wondering wheth

Re: Testing Apache Flink 0.9.0-rc1

t; > 2015-06-12 14:30 GMT+02:00 Maximilian Michels : > > > > > +1 Let's constitute the changes in a new release candidate. > > > > > > On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske > > wrote: > > > > > > > +1 for b) > > >

Re: Testing Apache Flink 0.9.0-rc1

much more convenient to let maven do the fat-jar > packaging. > > > On Fri, Jun 12, 2015 at 9:44 AM, Till Rohrmann > wrote: > > > I've finished the legal check of the source and binary distribution. The > PR > > with the LICENSE and NOTICE file updates can b

Re: The correct location for zipWithIndex and zipWithUniqueId

. Or there could be a link in the > > DataSet Transformations page poining to this... > > > > What do you think? > > > > On Wed, Jun 10, 2015 at 12:33 PM, Till Rohrmann > > > wrote: > > > > > I agree with Theo. I think it’s a nice feature to

Re: Listing Apache-2.0 dependencies in LICENSE file

entries from the LICENSE file. > > > > Best, > > Max > > > > On Sat, Jun 13, 2015 at 4:51 PM, Aljoscha Krettek > > wrote: > > > >> If it is not against the Apache Guidelines I would vote for removing > them. > >> I'm always in fa

Re: Apache Flink 0.9 ALS API

+1 for using long for both IDs. But I don't understand what's the advantage of using a String as user ID. On Sun, Jun 14, 2015 at 6:43 PM Robert Metzger wrote: > Hi Ronny, > > I accepted your previous mail to the mailing list, you got two replies: > > http://apache-flink-mailing-list-archive.10

Re: Apache Flink 0.9 ALS API

+1 for longs as IDs. Not so much in favour of Strings for the user ID because the row index could also denote the actual item ID if you swap the indices. Furthermore, you can always add a transformer which assigns unique IDs to names. Cheers, Till On Sat, Jun 13, 2015 at 3:34 PM Chiwan Park wro

Re: Testing Apache Flink 0.9.0-rc2

I might have found another release blocker. While running some cluster tests I also tried to run the `ConnectedComponents` example. However, sometimes the example couldn't be executed because the scheduler could not schedule co-located tasks, `NoResourceAvailableException`, even though it should ha

Re: ClassNotFound exception on a reduce group (Cluster mode)

I also tested your code with an additional sink and it is executed on a cluster without problems. Cheers, Till On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park wrote: > Hi. What flink version is running now in cluster? > I copied your code and packaging with flink quickstart archetype. There is > a

Re: Listing Apache-2.0 dependencies in LICENSE file

list them in the LICENSE file or not. Cheers, Till On Tue, Jun 16, 2015 at 7:36 AM Henry Saputra wrote: > Thanks Till, that clears up the confusion I had =) > > On Mon, Jun 15, 2015 at 1:37 AM, Till Rohrmann > wrote: > > Hi Henry, > > > > there are actually t

Re: Testing Apache Flink 0.9.0-rc2

t; You just start your cluster and then point the tool to the Flink > > > > > directory. It will then run all the examples with both builtin data > > and > > > > > external data. > > > > > > > > > > On Mon, 15 Jun 2015 at 17:15 M

Re: [VOTE] Release Apache Flink 0.9.0 (release-0.9.0-rc3)

+1 for reverting. On Thu, Jun 18, 2015 at 10:11 AM Aljoscha Krettek wrote: > +1 I also think it's the cleanest solution for now. The table API still > works, just without support for null values. > > On Thu, 18 Jun 2015 at 10:08 Maximilian Michels wrote: > > > I also vote for reverting the Tabl

Re: Flink Runtime Exception

Hi Andra, the problem seems to be that the deployment of some tasks takes longer than 100s. From the stack trace it looks as if you're not using the latest master. We had problems with previous version where the deployment call waited for the TM to completely download the user code jars. For larg

Re: Flink Runtime Exception

Yes, it was an issue for the milestone release. On Fri, Jun 19, 2015 at 2:18 PM Andra Lungu wrote: > Yes, so I am using flink-0.9.0-milestone-1. Was it a problem for this > version? > I'll just fetch the latest master if this is the case. > > On Fri, Jun 19, 2015 at 2:

Re: Flink Runtime Exception

; > That's the last question for today, promise :) > > Thanks! > > On Fri, Jun 19, 2015 at 2:40 PM, Till Rohrmann > wrote: > > > Yes, it was an issue for the milestone release. > > > > On Fri, Jun 19, 2015 at 2:18 PM Andra Lungu > wrote: > > >

Re: Flink Runtime Exception

ogram is really stuck? > > Cheers, Fabian > > 2015-06-19 15:08 GMT+02:00 Till Rohrmann : > > > What does forever mean? Usually it's the case that you see a steep > decline > > in performance once the system starts spilling data to disk because of > the > >

Re: [VOTE] Release Apache Flink 0.9.0 (release-0.9.0-rc4)

+1 - Checked LICENSE and NOTICE files - Read the readme.md - Checked signature and checksums - Built from sources - Tested all examples in cluster mode - Tested all examples in yarn mode - Tested single yarn session per job - Tested web-client job submission - Tested WordCount with HDFS input - Te

Re: Provide Hadoop pre-build Hadoop 2.4 and Hadoop 2.6 binaries

+1 for the different Hadoop versions. For the version without Hadoop binaries we should check whether the Hadoop interfaces are compatible across the different versions. But if this is the case, then also +1 for that. On Wed, Jun 24, 2015 at 12:14 PM, Stephan Ewen wrote: > big +1 from me as wel

Re: Drafting the 0.9.0 release announcement

We should ping the Zeppelin guys to update their Flink dependency. On Wed, Jun 24, 2015 at 2:34 PM, Maximilian Michels wrote: > I'm so happy we have pushed it out :) It took a while but I think we can be > very pleased with the result. > > I will post an announcement to the user/dev and the Apac

Re: Failing Builds on Travis

+1 On Tue, Jun 23, 2015 at 3:16 PM, Robert Metzger wrote: > +1 > > > > On Tue, Jun 23, 2015 at 11:31 AM, Fabian Hueske wrote: > > > +1 > > > > 2015-06-22 17:44 GMT+02:00 Stephan Ewen : > > > > > +1 > > > > > > On Fri, Jun 19, 2015 at 10:48 AM, Matthias J. Sax < > > > mj...@informatik.hu-berlin.

Re: [flink-ml] How to use ParameterMap in predict method?

Hi Chiwan, when you use the single element predict operation, you always have to implement the `getModel` method. There you have access to the resulting parameters and even to the instance to which the `PredictOperation` belongs. Within in this `getModel` method you can initialize all the informat

Re: FLINK-2066

Done On Mon, Jun 29, 2015 at 9:33 AM, Chiwan Park wrote: > We should assign FLINK-2066 to Nuno. :) > > Regards, > Chiwan Park > > > On Jun 29, 2015, at 1:21 PM, Márton Balassi > wrote: > > > > Hey, > > > > Thanks for picking up the issue. This value can be specified as > > "execution-retries.de

Re: [flink-ml] How to use ParameterMap in predict method?

> > Regards, > Chiwan Park > > > On Jun 29, 2015, at 4:43 PM, Till Rohrmann wrote: > > > > Hi Chiwan, > > > > when you use the single element predict operation, you always have to > > implement the `getModel` method. There you have access to the result

Re: [DISCUSS / VOTE] Signal name to "kill" streaming jobs

+1 for "stop" On Tue, Jun 30, 2015 at 10:26 AM, Fabian Hueske wrote: > Hi, > > thanks for starting this thread! > > I prefer "stop" over "terminate" because it sounds more graceful. A > temporary stop could be called "pause". > > Cheers, Fabian > > 2015-06-29 13:42 GMT+02:00 Matthias J. Sax >:

Re: [ml] Convergence Criterias

I think Sachin wants to provide something similar to the LossFunction but for the convergence criterion. This would mean that the user can specify a convergence calculator, for example to the optimization framework, which is used from within a iterateWithTermination call. I think this is a good id

Re: Redesigned "Features" page

I also like the new feature page. I better conveys the strong points of Flink, since it's more to the point. On Mon, Jul 6, 2015 at 6:09 PM, Stephan Ewen wrote: > Thanks Max! > > Did not even know we had a github mirror of the flink-web repo... > > On Mon, Jul 6, 2015 at 6:05 PM, Maximilian Mich

Re: [ml] Convergence Criterias

I would make the convergence criterion a parameter which is not mandatory for all Predictors. If you implement an iterative Predictor, then you can define a setConvergenceCriterion method or pass the convergence criterion to the Predictor via the ParameterMap. You can also open a JIRA issue for th

Re: Building several models in parallel

Hi Felix, this is currently not supported by FlinkML. The MultipleLinearRegression algorithm expects a DataSet and not a GroupedDataSet as input. What you can do is to extract each group from the original DataSet by using a filter operation. Once you have done this, you can train the linear model

Re: Building several models in parallel

: > Thanks for the information Till :) > > So at the moment the iteration is the only way. > > Best regards, > Felix > > 2015-07-08 10:43 GMT+02:00 Till Rohrmann : > > > Hi Felix, > > > > this is currently not supported by FlinkML. The MultipleLinearRegress

Re: Building several models in parallel

t() > > ... > > } > > Is there a more efficient way to do this? > > Thank you for your help, > > Felix > > > > > 2015-07-08 10:58 GMT+02:00 Till Rohrmann : > > > Yes it is. But you can still run the calculation in parallel because > `fit

Re: Question regarding HBase and machine-learning

Hi Lydia, I assume that your data is already in the libSVM format or you convert it into this format before you write it to a file. Otherwise, libSVM cannot read in the data. If you do this, then you can directly use this transformed DataSet as the input for the SVM algorithm, instead of writing i

Re: [DISCUSS] Unifying client code

I like the idea to have a single point of access. That would improve maintainability and makes the code easier to understand. Thus +1. On Thu, Jul 16, 2015 at 4:45 PM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > Hi, > > I just had a look into CliFrontend and Client and it seems to m

Re: Failing Test

I think the problem might be related to the way the test is constructed. The test submits a job to the JM and then tries to poll the accumulators from the JM. If it does not succeed, then the polling is retried with an decreasing pause in between. Furthermore, the task which updates the accumulator

Re: Exception during job submission in RemoteEnvironment

Hi Andreas, for me this looks like an error. The problem is that the Client does not respect the global configuration file when you execute your program from Java/IDE. There is also no way to specify a configuration file for the RemoteEnvironment, unlike for the LocalEnvironment. I think this shou

Re: Erroneous link to "Flink on Windows" guide

Thanks for spotting the broken link Samia. The correct link is [1]. I'll fix it. Cheers, Till [1] https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/local_setup.html#flink-on-windows On Thu, Jul 23, 2015 at 2:40 PM, Samia Khalid wrote: > Hi, > > I wanted to point out that the li

Re: Extending the streaming scala api with stateful functions

We have something similar for broadcast variables in FlinkML. It allows you to write ds.mapWithBcVariable(bcDS){ (dsElement, bcVar) => ... }. I like the idea to make the life of a Scala programmer a little bit less javaesque :-) ​ On Fri, Jul 24, 2015 at 5:45 PM, Stephan Ewen wrote: > This is r

Re: Travis build failure

This might be a Travis hick-up. Was it the first time this happened? Cheers, Till On Mon, Jul 27, 2015 at 9:47 PM, Sachin Goel wrote: > A recent travis build[Job 3] on my forked repo failed with the following > error: > > Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2

Re: Travis build failure

g that causes a segfault... > > > > On Tue, Jul 28, 2015 at 10:16 AM, Sachin Goel > > wrote: > > > > > Yes. I've never encountered this type of build failure. > > > > > > -- Sachin Goel > > > Computer Science, IIT Delhi > > >

Re: [Gelly] Re-organize / cleanup examples and library

Sounds reasonable to me. +1 On Tue, Jul 28, 2015 at 4:14 PM, Vasiliki Kalavri wrote: > Hello squirrels, > > adding Gelly examples has been a great way to let in new contributors and > we have had quite a big number of contributions! > However, I think it's about time we re-organize and clean the

Re: question about SlidingPreReducer.java

Hi MaGuoWei, do you mean that the branch of the if statement (line 130) is never executed? Or are you looking for an example which uses the SlidingPreReducer ? Cheers, Till ​ On Thu, Jul 30, 2015 at 11:33 AM, MaGuoWei wrote: > hi guysThere is a function updateCurrent() in this > class(SlidingP

Re: question about SlidingPreReducer.java

next element (else branch). ​ On Thu, Jul 30, 2015 at 11:54 AM, MaGuoWei wrote: > hi till > I mean the branch of the if statement is never execute. > thanks > > 发自我的 iPhone > > > 在 2015年7月30日，下午5:51，Till Rohrmann 写道： > > > > Hi MaGuoWei, > > > >

Re: Types in the Python API

Zeppelin uses py4j [1] to transfer data between a Python process and a JVM. That way they can run a Python interpreter and Java interpreter and easily share state between them. Spark also uses py4j as a bridge between Java and Python. However, I don't know for what exactly. And I also don't know wh

Re: Multiple control flows in a program

At the moment, Flink does not support the calculation of intermediate results from which you can continue your computation. When you execute jobs which share parts of its job graph, then they are recomputed. When your job contains operators with non-deterministic output, then there is no guarantee

Re: Multiple control flows in a program

SequenceIterator seems deterministic in its > operation, at least as far as how the elements are grouped together. But > surprisingly, I observed different splits now and then. > > Regards > Sachin > > -- Sachin Goel > Computer Science, IIT Delhi > m. +91-9871457685 >

Re: Multiple control flows in a program

> -- Sachin Goel > Computer Science, IIT Delhi > m. +91-9871457685 > > On Wed, Aug 12, 2015 at 1:37 PM, Till Rohrmann > wrote: > > > One branch does not occupy a single slot. A slot is usually shared by > > operators from multiple branches. Only subtasks of the same

Re: [jira](FLINK-2493) Simplify names of example program JARs

In case of merging we would have to add a `scala` tag to some of the examples because some of the examples exist in both packages. The drawback of merging both examples modules is that we'll have another mixed Java/Scala project. This might be bothering for the eclipse folks, but we already have th

Re: [jira](FLINK-2493) Simplify names of example program JARs

amples.scala". I > am also fine with dropping the subpackages like "graph" or so. Not like we > have so many examples that we need subpackages ;-) > > On Fri, Aug 14, 2015 at 11:01 AM, Till Rohrmann > wrote: > > > In case of merging we would have to add a `scala`

Re: Code style guideline for Scala

Good initiative Chiwan. +1 for a more unified code style. On Tue, Aug 18, 2015 at 10:25 AM, Chiwan Park wrote: > Okay, I’ll create a JIRA issue covered this topic. > > Regards, > Chiwan Park > > > On Aug 17, 2015, at 1:17 AM, Stephan Ewen wrote: > > > > +1 for formatting templates for Eclipse a

Re: [NOTICE] Reporting and fixing minor issues

+1, there is no point in arguing with Knuth. On Mon, Aug 17, 2015 at 1:07 AM, Henry Saputra wrote: > +1 as well. > > This is a great follow-up from my previous email about adding details > in JIRA, which also being echoed by Fabian. > > - Henry > > On Sun, Aug 16, 2015 at 3:45 PM, Fabian Hueske

Re: A Question About Execution Schedule

Hi MaGuoWei, this is not a problem. If you look at the implementation of SlotAllocationFuture.setFutureAction, you’ll see that the method is synchronized on a lock which is also used to complete the future. Furthermore, you’ll see that the slot variable is checked upon setting an action and if it’

Re: [ANNOUNCE] New Committer Chesnay Schepler

Congrats and welcome on board Chesnay :-) On Thu, Aug 20, 2015 at 11:18 AM, Robert Metzger wrote: > The Project Management Committee (PMC) for Apache Flink has asked Chesnay > Schepler to become a committer and we are pleased to announce that they > have accepted. > > Chesnay has been very invol

Re: [FAILING TEST] StateCheckpoinedITCase

I'm also in favor of JIRA, because I fear that nobody will keep the wiki page in sync. Maybe we can assign a special label for test stability to these JIRA issues. Then we can quickly find all currently instable test cases. On Fri, Aug 21, 2015 at 11:02 AM, Robert Metzger wrote: > I agree that w

Re: [DISCUSSION] Release current master as 0.9.1 (mod few changes)

+1, good solution. On Wed, Aug 26, 2015 at 3:11 PM, Márton Balassi wrote: > +1 > > On Wed, Aug 26, 2015 at 3:11 PM, Maximilian Michels > wrote: > > > We will have a proper minor release and a preview of 0.10. After all, > > a good compromise. > > > > +1 > > > > On Wed, Aug 26, 2015 at 2:57 PM,

Re: [VOTE] Release Apache Flink 0.9.1 (RC1)

+1 - Tested against Hadoop 2.7 / Scala 2.10 - Tested local-cluster and cluster (4 machine Linux cluster) execution - Executed all example test with built-in and external data using FliRTT - Tested YARN execution - Tested job submission via the web client - Tested web interface's job visualization

Re: Outer-join operator integration with DataSet API (FLINK-2576)

Hi Johann, I'd prefer 1.c, because the different join variants are semantically different and this should be IMO reflected in the API. Moreover, the `JoinHints` are used to give hints for the selection of the underlying strategy for the different join variants. For `leftOuterJoin` you could either

Re: [ANNOUNCE] Welcome Matthias Sax as new committer

Congratulations Matthias! Welcome on board :-) On Wed, Sep 2, 2015 at 2:01 PM, Robert Metzger wrote: > Congrats Matthias! > > On Wed, Sep 2, 2015 at 1:34 PM, Timo Walther wrote: > > > Congratulations Matthias! > > > > Regards, > > Timo > > > > > > On 02.09.2015 13:32, Chiwan Park wrote: > > > >

Re: Interested in implementing Machine learning Libraries as my undergraduate final project

Hi Devang, great to hear that you want to contribute to Flink. The best thing to do first is to familiarise yourself with the system. Thus, going through the documentation [1] and trying to implement a first Flink job would be a good start. Next, you should read the contribution guide [2] and pick

Re: Releasing 0.10.0-milestone1

+1 for a milestone release with the TypeInformation issues fixed. I'm working on it. On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen wrote: > Great! > > I'd like to push one more commit later today. > A fix for https://issues.apache.org/jira/browse/FLINK-2632 would also be > highly appreciated by s

Re: Releasing 0.10.0-milestone1

Fóra wrote: > This sounds good +1 from me as well :) > Till Rohrmann ezt írta (időpont: 2015. szept. > 9., > Sze, 10:40): > > > +1 for a milestone release with the TypeInformation issues fixed. I'm > > working on it. > > > > On Tue, Sep 8, 2015

Re: Flink ML source code

Hi Hanan, if you need any help implementing ANOVA, then let me know. I'd like to assist you. Cheers, Till On Thu, Sep 10, 2015 at 4:15 PM, Robert Metzger wrote: > Hi, > > you can find the Flink ML source code here: > https://github.com/apache/flink/tree/master/flink-staging/flink-ml > > On Thu

Re: Flink ML linear regression issue

Hi Alexey and Hanan, one of FlinkML’s feature is the flexible pipelining mechanism. It allows you to chain multiple transformers with a trailing predictor to form a data analysis pipeline. In order to support multiple input types, the actual program logic (matching for the type) is assembled at co

Re: Flink's Checking and uploading JAR files Issue

Hi Hanan, you're right that currently every time you submit a job to the Flink cluster, all user code jars are uploaded and overwrite possibly existing files. This is not really necessary if they don't change. Maybe we should add a check that already existing files on the JobManager are not upload

Re: Release Flink 0.10

+1 for 0.10 On Tue, Sep 29, 2015 at 12:12 PM, Stephan Ewen wrote: > +1 here as well > > On Tue, Sep 29, 2015 at 12:03 PM, Fabian Hueske wrote: > > > +1 for moving directly to 0.10. > > > > 2015-09-29 11:40 GMT+02:00 Maximilian Michels : > > > > > Hi Kostas, > > > > > > I think it makes sense to

Re: Add a module for "manual" tests

+1 for your proposal Stephan. Seems like a good idea to improve build stability as well as guarding components like the ElasticSearchSink by a test, even though it’s not regularly executed. ​ On Tue, Sep 29, 2015 at 12:34 PM, Stephan Ewen wrote: > Hi all! > > We have by now quite some tests that

Re: Pulling Streaming out of staging and project restructure

+1 for the new project structure. Getting rid of our code dump is a good thing. On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels wrote: > +1 Matthias, let's limit the overhead this has for the module maintainers. > > On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax wrote: > > I will commit s

Re: Rethink the "always copy" policy for streaming topologies

Do we know what kind of impact the non-reuse policy has? Maybe the serialization overhead is subsumed by other effects. But in general I'm ok with changing the default to non copying. We just have to document this feature properly. On Oct 2, 2015 6:31 PM, "Maximilian Michels" wrote: > +1 Good id

Re: Failing test

If there is none yet, then we do. Label it with "test-stability". I think the consensus was also to mark it as critical. Otherwise, just add the log to the JIRA. On Tue, Oct 6, 2015 at 2:57 PM, Matthias J. Sax wrote: > Hi, > > One test just failed on current master: > https://travis-ci.org/apac

Re: Flaky ScalaShellLocalStartupITCase

Yeah I'm also struggling with the test case which has some wrong assumptions about the log output. I can also open the JIRA. Working on it today. Cheers, Till On Oct 14, 2015 11:32 AM, "Ufuk Celebi" wrote: > I know that Till observed issues with the ScalaShell tests as well. If I > remember cor

Re: Extending and improving our "How to contribute" page

Thanks for leading the effort Fabian! On Fri, Oct 9, 2015 at 10:07 AM, Maximilian Michels wrote: > Very nice work, Fabian. I think we'll have to send around a reminder > from time to time and, perhaps, evaluate the new guidelines after some > period of time. It's great to have these documents no

Re: JobManager not reachable

To verify wether GC is a problem you can enable logging of memory usage of the JVM via taskmanager.debug.memory.startLogThread: true. The interval of the logging is configured via taskmanager.debug.memory.logIntervalMs. ​ On Thu, Oct 15, 2015 at 12:00 PM, Matthias J. Sax wrote: > The problem is

Re: JobManager not reachable

C is an issue. > > > > Hmmm, what else can make the failure detector kick in unexpectedly? > > > > On Thu, Oct 15, 2015 at 12:05 PM, Till Rohrmann > > wrote: > > > > > To verify wether GC is a problem you can enable logging of memory usage > > of

Re: [DISCUSS] Java code style

I like the idea to have a bit stricter code style which will increase code maintainability and makes it easier for people to go through the code. Furthermore, it will relieve us from code style comments while reviewing PRs which can be quite cumbersome. Personally, I like the Google code style. Th

Re: [DISCUSS] Java code style

o we take Google's style guide as is, including spaces > instead of tabs? I like the spaces, but that will make things hard... > > On Tue, Oct 20, 2015 at 3:01 PM, Gyula Fóra wrote: > > > +1 for both :) > > > > Till Rohrmann ezt írta (időpont: 2015. okt. 20., >

Re: Scaling Flink

Hi Greg, there is no official guide for running Flink on large clusters. As far as I know, the cluster we used for the matrix factorization was the largest cluster we've run a serious job on. Thus, it would be highly interesting to understand what made the JobManager to slow down. At some point, t

Re: [DISCUSS] Java code style

I think that the line length limitation and the space indentation are the two rules which are most controversial in the Flink community because so far it has been done completely different. Thus, they would also inflict most of the changes. However, I think that at least the line length limitation

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

I found another issue (FLINK-2894 ) while helping the mahout guys with the flink bindings for Samsara. Currently we don't allow to register default serializer for Kryo. This means it is not possible to specify a serializer for a base class and all i

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

Forget my last mail. Just found out that the Mahout guys are still running on 0.9-SNAPSHOT. ​ On Thu, Oct 22, 2015 at 2:53 PM, Till Rohrmann wrote: > I found another issue (FLINK-2894 > <https://issues.apache.org/jira/browse/FLINK-2894>) while helping the > mahout guys with the

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix, I'm not sure whether PipelinedSubpartition should be readable more than once because then it would effectively mean that we materialize the elements of the pipelined subpartition for stragglers. Therefore, I think that we should make blocking intermediate results readable more than once.

Re: [DISCUSS] updating apache-flink homebrew formula

+1 for Hadoop 2.7 and Scala 2.11 On Tue, Aug 9, 2016 at 12:25 PM, Robert Metzger wrote: > Hi, > > Thanks a lot for maintaining the flink homebrew formula. > I think its a good idea to update to Hadoop 2.7 and Scala 2.11. > > On Tue, Aug 9, 2016 at 12:59 AM, Wright, Eron wrote: > > > Hello, > >

Re: [DISCUSS] Streaming connector contributions

I agree with Stephan that the main problem is maintenance overhead for the Flink community. If we could maintain all connectors ourselves then there would not be an immediate need to out source the connectors. Thus, the solution should reduce the workload for the core project. Personally, I would

Re: [DISCUSS] API breaking change in DataStream Windows

That is a tough call but I'm personally leaning slightly towards not breaking the API and adding a note for the casting workaround. My main concern is where do we set the limit for future API breaking issues? How critical does an issue has to be to be allowed to break the API? Currently, we have 1

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

r should I mark this jira and FLINK-1713 > <https://issues.apache.org/jira/browse/FLINK-1713> as blocking for the > broadcast jira? What do you think? > > Best regards, > Felix > > 2016-08-09 17:41 GMT+07:00 Till Rohrmann : > > > Hi Felix, > > > >

Re: [VOTE] Release Apache Flink 1.1.1 (RC1)

+1 from my side as well. On Tue, Aug 9, 2016 at 9:01 PM, Stephan Ewen wrote: > +1 > > This is a crucial fix and the released sources are actually still the same, > so reduced time should be okay. > > On Tue, Aug 9, 2016 at 8:24 PM, Henry Saputra > wrote: > > > Official vote > > +1 (binding) > >

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

gt; > > result-x-checkpoint4, ...) so it may be worth addressing that soon > > anyways. > > > > > > (2) Make iterations not dependent on the special local back channel. > > > Then we can simply schedule iterations like all other things. > > > > > >

Re: N-ary stream operators - status

Hi Theo, I think currently, there is nobody actively working on side inputs. Thus, it might be possible that they don't make it in the 1.2 release. Cheers, Till On Wed, Aug 10, 2016 at 10:41 PM, Theodore Vasiloudis < theodoros.vasilou...@gmail.com> wrote: > Hello Aljoscha, > > Do you think the

Re: Some thoughts about the lower-level Flink APIs

Hi, I'm also not sure whether we should start teaching Flink by demonstrating the low-level APIs. According to my experience, people new to Flink should first learn a very basic set of primitive operations. Usually this is map, flatmap, join, windows, etc. The semantics of these operations is well

Re: [DISCUSS] FLIP-9: Trigger DSL

Cool design doc Klou. It's well described with a lot of details. I like it a lot :-) +1 for implementing the trigger DSL. Cheers, Till On Tue, Aug 16, 2016 at 3:18 PM, Kostas Kloudas wrote: > Thanks for the feedback Ufuk! > I will do that. > > > On Aug 16, 2016, at 1:41 PM, Ufuk Celebi wrote:

Re: [HEADS-UP] Dev Week 33

Good initiative @Aljoscha. - Just a convenience link for FLIP-6: The feature branch can be found here https://github.com/apache/flink/tree/flip-6. - My main task is the review of the Mesos PR of Eron ( https://github.com/apache/flink/pull/2315) so that we can continue making progress there. On W

Re: [DISCUSS] Move Redis and Flume connectors to Apache Bahir and redirect contributions there

+1 On Thu, Aug 18, 2016 at 10:21 AM, Aljoscha Krettek wrote: > +1 > > On Thu, 18 Aug 2016 at 03:54 Jark Wu wrote: > > > Good start point. +1 > > > > > > > > - Jark Wu > > > > > 在 2016年8月17日，下午11:24，Robert Metzger 写道： > > > > > > Hi, > > > > > > following the discussions in [1], the community d

Re: [FLINK-305] Code test coverage - how FLINK using it?

Hi Pavel, I think it's a good point you're raising here. The Flink community isn't using metrics like test coverage to ensure high quality code yet. I think that is one thing which we can/should improve. Unfortunately, the ASF does not allow to use codecov.io (or at least the Apache Infra team) [1

Re: [DISCUSS] FLIP-3 - Organization of Documentation

g 2, 2016 at 5:18 PM, Aljoscha Krettek > > wrote: > > > +1 > > > > > > On Tue, 2 Aug 2016 at 03:15 Till Rohrmann > wrote: > > > > > >> +1 :-) > > >> > > >> On Tue, Aug 2, 2016 at 6:09 PM, Stephan Ewen

Re: [FLINK-305] Code test coverage - how FLINK using it?

table and does not affect > build duration too much. > > 2016-08-19 12:51 GMT+03:00 Till Rohrmann : > > > Hi Pavel, > > > > I think it's a good point you're raising here. The Flink community isn't > > using metrics like test coverage to ensure hig

Re: [DISCUSS] FLIP-10: Unify Savepoints and Checkpoints

+1 for the FLIP. I like the described changes and new functionality. When looking at the public interface, I was wondering whether we should not allow the user to specify a TimeUnit for the periodic interval. I think it's nicer to be able to specify the time unit instead of converting everything

Re: Additional project downloads

I agree that it would be good to offer these kind of convenience download links. On Wed, Aug 24, 2016 at 5:25 PM, Robert Metzger wrote: > Maybe we should put a link to maven central. We could parameterize the link > so that it always links to the current release linked on our downloads > page. >

Re: Task manager processes crashing one after the other

Hi Gyula, I haven't seen this problem before. Do you have the logs of the failed TMs so that we have some more context what was going on? Cheers, Till On Thu, Aug 25, 2016 at 9:40 AM, Gyula Fóra wrote: > Hi guys, > > For quite some time now we fairly frequently experience a task manager > cras

Re: Enabling Encryption between slaves in Flink

Hi Vinay, it's not only done by enabling encryption of the Akka messages since the TaskManager to TaskManager communication utilizes Netty. There is a JIRA issue which tracks the progress on implementing secured communication within Flink: https://issues.apache.org/jira/browse/FLINK-3931. Feel fre

<    5   6   7   8   9   10   11   12   13   14   >