[jira] [Created] (FLINK-6139) Documentation for building / preparing Flink for MapR

Tzu-Li (Gordon) Tai created FLINK-6139: -- Summary: Documentation for building / preparing Flink for MapR Key: FLINK-6139 URL: https://issues.apache.org/jira/browse/FLINK-6139 Project: Flink

Re: [DISCUSS] TravisCI status on GitHub Page

I would argue for benefits of having build status. Instead of letting people go through all docs and wikis to find how Flink build system works, it guides people directly to where builds actually happen and ramps up new contributors faster. When my local tests fail during development, the homepage

[jira] [Created] (FLINK-6138) Improve UnboundedNonPartitionedProcessingOverProcessFunction

sunjincheng created FLINK-6138: -- Summary: Improve UnboundedNonPartitionedProcessingOverProcessFunction Key: FLINK-6138 URL: https://issues.apache.org/jira/browse/FLINK-6138 Project: Flink Issue

Re: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

On this guava, it looks fine when i rebuild the flink using maven 3.2.5. Doesn't get a chance to test the httpclient, but i did try to use a decompiler to check the flink-dist_2.11-1.2.0.jar, there is no package org.apache.http exposed. So i would think it's fine. So the 6125 is possibly the same

[jira] [Created] (FLINK-6137) Add custom checkstyle for flink-cep

Dawid Wysakowicz created FLINK-6137: --- Summary: Add custom checkstyle for flink-cep Key: FLINK-6137 URL: https://issues.apache.org/jira/browse/FLINK-6137 Project: Flink Issue Type: Improveme

Re: Machine Learning on Flink - Next steps

Hello all, I've updated the original Gdoc [1] to include a table with coordinators and people interested in contributing to the specific projects. With this latest additions we have many people willing to contribute to the online learning library, and 2 people who have shown interested to at least

[jira] [Created] (FLINK-6136) Separate EmbeddedNonHaServices and NonHaServices

Till Rohrmann created FLINK-6136: Summary: Separate EmbeddedNonHaServices and NonHaServices Key: FLINK-6136 URL: https://issues.apache.org/jira/browse/FLINK-6136 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-6135) Allowing adding inputs to StreamOperator

Aljoscha Krettek created FLINK-6135: --- Summary: Allowing adding inputs to StreamOperator Key: FLINK-6135 URL: https://issues.apache.org/jira/browse/FLINK-6135 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-6134) Set UUID(0L, 0L) as default leader session id

Till Rohrmann created FLINK-6134: Summary: Set UUID(0L, 0L) as default leader session id Key: FLINK-6134 URL: https://issues.apache.org/jira/browse/FLINK-6134 Project: Flink Issue Type: Impro

[jira] [Created] (FLINK-6133) fix build status in README.md

Bowen Li created FLINK-6133: --- Summary: fix build status in README.md Key: FLINK-6133 URL: https://issues.apache.org/jira/browse/FLINK-6133 Project: Flink Issue Type: Bug Components: Build

[jira] [Created] (FLINK-6132) Remove redundant code in CliFrontend.java

canbinzheng created FLINK-6132: -- Summary: Remove redundant code in CliFrontend.java Key: FLINK-6132 URL: https://issues.apache.org/jira/browse/FLINK-6132 Project: Flink Issue Type: Improvement

Re: [Disuss]Permission of checkpoint directory

Prior discussion at https://github.com/apache/flink/pull/3335 > On Mar 19, 2017, at 11:34 PM, Wangtao (WangTao) wrote: > > Hi All, > > Checkpoint directory will store user data and it is better to keep it with > minimum access right, like 700, in c

[jira] [Created] (FLINK-6131) Add side inputs for DataStream API

Aljoscha Krettek created FLINK-6131: --- Summary: Add side inputs for DataStream API Key: FLINK-6131 URL: https://issues.apache.org/jira/browse/FLINK-6131 Project: Flink Issue Type: Improvemen

[jira] [Created] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held

Ted Yu created FLINK-6130: - Summary: Consider calling resourceManager#getTerminationFuture() with lock held Key: FLINK-6130 URL: https://issues.apache.org/jira/browse/FLINK-6130 Project: Flink Issue

Re: Machine Learning on Flink - Next steps

Hi all, @Theodore: +1 for the CTR use-case. Thanks for the suggestion! @Katherin: +1 for reflecting the choices made here and contributor commitment in Gdoc. @Tao, @Ventura: It's great to here you have been working on ML on Flink :) I hope we can all aggregate our efforts somehow. It would be b

Re: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

I don't think so: https://issues.apache.org/jira/browse/FLINK-6125 >From a related issue brought to the mailing list by another user. On Mon, Mar 20, 2017 at 2:36 PM, Aljoscha Krettek wrote: > So it is resolved now? Thanks for letting us know! >> On 20 Mar 2017, at 12:55, Liangfei Su wrote: >

Re: Machine Learning on Flink - Next steps

Hello Theodore, I could lead the effort for the model serving thing, I think I have the cycles for it. I would also contact Enron and see if he wants to help in that direction. Best, Stavros On Sun, Mar 19, 2017 at 3:43 PM, Theodore Vasiloudis < theodoros.vasilou...@gmail.com> wrote: > Hello St

Re: Future of Queryable State Feature

On Mon, Mar 20, 2017 at 3:25 PM, Aljoscha Krettek wrote: > I think adding this could be as simple as adding a > makeWindowContentsQueryable(String) call to WindowedStream. In there, we > create the StateDescriptor for keeping the window contents so we can call > setQueryable() on the StateDescr

Re: Future of Queryable State Feature

I think adding this could be as simple as adding a makeWindowContentsQueryable(String) call to WindowedStream. In there, we create the StateDescriptor for keeping the window contents so we can call setQueryable() on the StateDescriptor and set the name under which it should be accessible. If yo

RE: [DISCUSS] Could we Improve tests time and stability?

Sorry Greg, I have given you wrong reference to tests of different GCs. This is correct - https://travis-ci.org/kenmy/flink/builds -Original Message- From: Greg Hogan [mailto:c...@greghogan.com] Sent: Friday, March 17, 2017 8:06 PM To: dev@flink.apache.org Subject: Re: [DISCUSS] Could we

Re: [DISCUSS] Project build time and possible restructuring

The Beam Jenkins jobs are configured inside the Beam src repo itself. For example: https://github.com/apache/beam/blob/master/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Flink.groovy For initial setup of the seed job you need admin rights on Jenkins, as described here: https://cwiki.apa

Re: [DISCUSS] Project build time and possible restructuring

We can add cluster tests using the distribution jar, and will need to do so to remove Flink’s dependency on Hadoop. The YARN and Mesos tests would still run nightly and running cluster tests should be much faster. As troublesome as TravisCI has been, a major driver for this change has been local

Re: Machine Learning on Flink - Next steps

Hello everyone, Here at DFKI, we are currently working on project that involves developing open-source Online Machine Learning algorithms on top of Flink. So far, we have simple moments, sampling (e.g.: simple reservoir sampling) and sketches (e.g., Frequent Directions) built on top of scikit-like

Re: [DISCUSS] Project build time and possible restructuring

It looks like Jetbrains TeamCity supports something in that direction: https://blog.jetbrains.com/teamcity/2012/03/incremental-building-with-maven-and-teamcity/ On Mon, Mar 20, 2017 at 2:40 PM, Timo Walther wrote: > Another solution would be to make the Travis builds more efficient. For > examp

Re: [DISCUSS] Project build time and possible restructuring

Another solution would be to make the Travis builds more efficient. For example, we could write a script that determines the modified Maven module and only run the test for this module (and maybe transitive dependencies). PRs for libraries such as Gelly, Table, CEP or connectors would not trigg

Re: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

So it is resolved now? Thanks for letting us know! > On 20 Mar 2017, at 12:55, Liangfei Su wrote: > > Well...Comments added. > > This looks due to incorrect local build of flink. > > Documentation at > https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html > shows >

[jira] [Created] (FLINK-6129) MetricQueryService does not stop query actor

Till Rohrmann created FLINK-6129: Summary: MetricQueryService does not stop query actor Key: FLINK-6129 URL: https://issues.apache.org/jira/browse/FLINK-6129 Project: Flink Issue Type: Bug

Re: [DISCUSS] TravisCI status on GitHub Page

I merged the PR and therefore obviously think it's fine. ;-) Didn't see Robert's comment in the issue though ("We once had the travis build status badge in our readme, but decided to remove it, because it often shows "Build failed" due to travis issues etc. This gives people the impression that our

RE: FW: [DISCUSS] Table API / SQL indicators for event and processing time

Hi, I am not sure if it is not about setting the timestamp within the query but you can imagine that there examples where you have different timestamps as mentioned. Take for example the case when we do a purchase online. You have: -time of purchase (when the payment was input/triggered) -time

[DISCUSS] TravisCI status on GitHub Page

We are now showing the TravisCI build status on Flink’s GitHub page. I think Robert’s comment in Jira may have gone unnoticed when the PR was committed. https://issues.apache.org/jira/browse/FLINK-6122 If not yet seeing the benefit even if bui

Re: FW: [DISCUSS] Table API / SQL indicators for event and processing time

Yes, you are right. In the current design the user cannot assign timestamp and watermarks in a table program. Operators (such as windows) might adapt the metatimestamp, if this is the case this adaption might need to be expressed in the query itself too. E.g. for a tumbling windows we could li

Re: [DISCUSS] Project build time and possible restructuring

Aljoscha, do you know how to configure jenkins? Is Apache INFRA doing that, or are the beam people doing that themselves? One downside of Jenkins is that we probably need some machines that execute the tests. A Travis container has 2 CPU cores and 4 GB main memory. We currently have 10 such contai

Re: [DISCUSS] Project build time and possible restructuring

I agress with Aljoscha that we might consider moving from Jenkins to Travis. Is there any disadvantage in using Jenkins? I think we should structure the project according to release management (e.g. more frequent releases of libraries) or other criteria (e.g. core and non-core) instead of buil

FW: [DISCUSS] Table API / SQL indicators for event and processing time

Hi, As suggested by Timo - I am forwarding this to the mailing list. Sorry for not having the conversation directly here - I initially thought it might not be of interest... @Timo - thanks for the clarification. I get the main point now which is that the rowtime is encoded within the metadata

Re: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

Well...Comments added. This looks due to incorrect local build of flink. Documentation at https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html shows clearly that """ NOTE: Maven 3.3.x can build Flink, but will not properly shade away certain dependencies. Maven 3.0

Re: Machine Learning on Flink - Next steps

Hi All, Sorry for joining this discussion late. My graduation thesis is about online learning system. I would build it on flink in the next three months. I'd like to contribute on: - Online learning On Mon, Mar 20, 2017 at 6:51 PM Katherin Eri wrote: Hello, Theodore Could you please move

Re: [DISCUSS] Project build time and possible restructuring

I prefer Jenkins to Travis by far. Working on Beam, where we have good Jenkins integration, has opened my eyes to what is possible with good CI integration. For example, look at this recent Beam PR: https://github.com/apache/beam/pull/2263 . The Jenkins

Re: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

Another finding that it looks most of these kinds of conflict comes from the dependency to hadoop/hbase (by using mvn dependency to check). It looks to me the correct way is to shade more packages from the hadoop dependencies... Thoughts here? Thanks Ralph On Mon, Mar 20, 2017 at 6:59 PM, Lian

[jira] [Created] (FLINK-6128) Optimize JVM options for improve test performance

Dmytro Shkvyra created FLINK-6128: - Summary: Optimize JVM options for improve test performance Key: FLINK-6128 URL: https://issues.apache.org/jira/browse/FLINK-6128 Project: Flink Issue Type:

Fwd: [jira] [Updated] (FLINK-6126) Yet another conflict : guava

Hi, I try to write some user function to write elastic search, my project comes with elastic search 2.3.5 (which is the same to flink connectors elastic search). But i'm seeing be exception that recorded in the below JIRA, by enabling the java -version:class, it show it's the guava dependency issu

Re: Machine Learning on Flink - Next steps

Hello, Theodore Could you please move vectors of development and their prioritized positions from *## Executive summary* to the google doc? Could you please also create some table in google doc, that is representing the selected directions and persons, who would like to drive or participate in

Re: [DISCUSS] Project build time and possible restructuring

Thank you for looking into the build times. I didn't know that the build time situation is so bad. Even with yarn, mesos, connectors and libraries removed, we are still running into the build timeout :( Aljoscha told me that the Beam community is using Jenkins for running the tests, and they are

Re: [DISCUSS] deprecated function need more detail

There is a related checkstyle rule: http://checkstyle.sourceforge.net/apidocs/com/puppycrawl/tools/checkstyle/checks/annotation/MissingDeprecatedCheck.html Added a JIRA for adding it here: https://issues.apache.org/jira/browse/FLINK-6127 We actually wrote this down in our hidden Wiki at https://c

[jira] [Created] (FLINK-6127) Add MissingDeprecatedCheck to checkstyle

Ufuk Celebi created FLINK-6127: -- Summary: Add MissingDeprecatedCheck to checkstyle Key: FLINK-6127 URL: https://issues.apache.org/jira/browse/FLINK-6127 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-6126) Yet another conflict

Su Ralph created FLINK-6126: --- Summary: Yet another conflict Key: FLINK-6126 URL: https://issues.apache.org/jira/browse/FLINK-6126 Project: Flink Issue Type: Bug Components: Build System,

Re: [DISCUSS] deprecated function need more detail

+1 I think we actually had the same discussion already a while back. Let's bring it back to everyone's awareness! On Wed, Nov 23, 2016 at 12:09 PM, Paris Carbone wrote: > +1 > > This should always be the norm, especially for user-facing code. > > While we are at it, perhaps when someone depre

Re: [DISCUSS] Project build time and possible restructuring

@Greg I am personally in favor of splitting "connectors" and "contrib" out as well. I know that @rmetzger has some reservations about the connectors, but we may be able to convince him. For the cluster tests (yarn / mesos) - in the past there were many cases where these tests caught cases that ot

RE: [DISCUSS] Could we Improve tests time and stability?

Greg, The tests falling with time out randomly. If you explore build history of apache/flink travis you can see this Оption - UseGCOverheadLimit in args of test args can hide problem with OOM, so we need remove it and add heap size in any case. If talk about GC algorithms - Evgeny Kincharov alrea

RE: [DISCUSS] Table API / SQL indicators for event and processing time

Hi, Thanks for the explanation. I take from the " FlinkTypeFactory.isRowtime() and FlinkTypeFactory.isProctime() " that it is still going to work to make this differentiation. I think this is very important and a good thing to have. For example as we will go into supporting joins, ordering/sor

Re: [DISCUSS] Table API / SQL indicators for event and processing time

Hi Radu, we differentiate rowtime and processing time fields by their field types. Both indicators extend the timestamp type. In my protoype I added the functions FlinkTypeFactory.isRowtime() and FlinkTypeFactory.isProctime() for checking this. If a time indicator has been materiatized (e.g.

RE: [DISCUSS] Table API / SQL indicators for event and processing time

Hi Timo, I have some questions regarding your implementation: " The timestamp (not an indicator anymore) becomes part of the physical row. E.g. long.cast(STRING) would require a materialization " => If we have this how are we going to make a difference between rowtime and processtime? For supp

[jira] [Created] (FLINK-6125) Commons httpclient is not shaded anymore in Flink 1.2

Robert Metzger created FLINK-6125: - Summary: Commons httpclient is not shaded anymore in Flink 1.2 Key: FLINK-6125 URL: https://issues.apache.org/jira/browse/FLINK-6125 Project: Flink Issue T

[jira] [Created] (FLINK-6124) support max/min aggregations for string type

Zhenghua Gao created FLINK-6124: --- Summary: support max/min aggregations for string type Key: FLINK-6124 URL: https://issues.apache.org/jira/browse/FLINK-6124 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-6123) Add support for the NOT pattern.

Kostas Kloudas created FLINK-6123: - Summary: Add support for the NOT pattern. Key: FLINK-6123 URL: https://issues.apache.org/jira/browse/FLINK-6123 Project: Flink Issue Type: Bug Co

Re: [DISCUSS] Table API / SQL indicators for event and processing time

Hi everyone, for the last two weeks I worked on a solution for the time indicator issue. I have implemented a prototype[1] which shows how we can express, track, and access time in a consistent way for batch and stream tables. Main changes of my current solution: - Processing and rowtime tim

[jira] [Created] (FLINK-6121) Include TravisCI build status to README.md

Bowen Li created FLINK-6121: --- Summary: Include TravisCI build status to README.md Key: FLINK-6121 URL: https://issues.apache.org/jira/browse/FLINK-6121 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-6122) Include TravisCI build status to README.md

Bowen Li created FLINK-6122: --- Summary: Include TravisCI build status to README.md Key: FLINK-6122 URL: https://issues.apache.org/jira/browse/FLINK-6122 Project: Flink Issue Type: Improvement