[jira] [Created] (FLINK-2622) Scala DataStream API does not have writeAsText method which supports WriteMode

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-2622:


 Summary: Scala DataStream API does not have writeAsText method 
which supports WriteMode
 Key: FLINK-2622
 URL: https://issues.apache.org/jira/browse/FLINK-2622
 Project: Flink
  Issue Type: Bug
  Components: Scala API, Streaming
Reporter: Till Rohrmann


The Scala DataStream API, unlike the Java DataStream API, does not support a 
{{writeAsText}} method which takes the {{WriteMode}} as a parameter. In order 
to make the two APIs consistent, it should be added to the Scala DataStream API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Maximilian Michels]

Hi Behrouz,

I would create a new sub-task under the original issue that introduce
the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525

Cheers,
Max

On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
 wrote:
> Hi Max,
>
> What you said makes sense, for "ParameterTool doesn't seem to support
> positional arguments :) but we could fix that." should we create a separate
> ticket or should it also be part of FLINK-2021 ?
>
> BR,
> Behrouz
>
>
> On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels  wrote:
>
>> Hi Behrouz,
>>
>> Thanks for starting the discussion. If I understand your question
>> correctly, you are asking if it breaks the training or other external
>> material if we convert the Flink examples to make use of the
>> ParameterTool?
>>
>> We could make the changes such that the examples will accept the same
>> parameters but use the ParameterTool internally to verify the
>> parameters and print usage information. I think most examples simply
>> use positional arguments and we could keep it that way. The only
>> problem is that the ParameterTool doesn't seem to support positional
>> arguments :) but we could fix that.
>>
>> Cheers,
>> Max
>>
>> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
>>  wrote:
>> > Hi,
>> >
>> > I had at look at this ticket FLINK-2021
>> > , there isn't much to
>> do
>> > from a technical stand point and it kinda makes sense to use the new
>> > "ParameterTool", since it is being used in most of the other part of the
>> > code base.
>> > The only question is do we really want to do it, since I'm guessing some
>> of
>> > the training materials, slides and articles are referencing these
>> examples
>> > and updating those might be a burden.
>> >
>> > Let me know what you guys think, either I can start working on it or we
>> can
>> > just resolve it for good.
>> >
>> > Cheers,
>> > Behrouz
>>

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Behrouz Derakhshan]

Will do.

Thanks,
Behrouz

On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels  wrote:

> Hi Behrouz,
>
> I would create a new sub-task under the original issue that introduce
> the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
>
> Cheers,
> Max
>
> On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
>  wrote:
> > Hi Max,
> >
> > What you said makes sense, for "ParameterTool doesn't seem to support
> > positional arguments :) but we could fix that." should we create a
> separate
> > ticket or should it also be part of FLINK-2021 ?
> >
> > BR,
> > Behrouz
> >
> >
> > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels 
> wrote:
> >
> >> Hi Behrouz,
> >>
> >> Thanks for starting the discussion. If I understand your question
> >> correctly, you are asking if it breaks the training or other external
> >> material if we convert the Flink examples to make use of the
> >> ParameterTool?
> >>
> >> We could make the changes such that the examples will accept the same
> >> parameters but use the ParameterTool internally to verify the
> >> parameters and print usage information. I think most examples simply
> >> use positional arguments and we could keep it that way. The only
> >> problem is that the ParameterTool doesn't seem to support positional
> >> arguments :) but we could fix that.
> >>
> >> Cheers,
> >> Max
> >>
> >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
> >>  wrote:
> >> > Hi,
> >> >
> >> > I had at look at this ticket FLINK-2021
> >> > , there isn't much
> to
> >> do
> >> > from a technical stand point and it kinda makes sense to use the new
> >> > "ParameterTool", since it is being used in most of the other part of
> the
> >> > code base.
> >> > The only question is do we really want to do it, since I'm guessing
> some
> >> of
> >> > the training materials, slides and articles are referencing these
> >> examples
> >> > and updating those might be a burden.
> >> >
> >> > Let me know what you guys think, either I can start working on it or
> we
> >> can
> >> > just resolve it for good.
> >> >
> >> > Cheers,
> >> > Behrouz
> >>
>

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Robert Metzger]

If you are referring to this training material (
https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java),
some of the examples are actually already using the ParameterTool.

The problem are probably websites / blogposts etc. that show how to use the
Flink examples. But I think its fine to break these. All example jars
contain the version number. If the way we pass arguments to the examples
changes between 0.9 and 0.10, that should be fine.

I think using the ParameterTool for the examples will improve the
readability of the examples a lot. Right now, all examples have a
(copy-pasted) parseParameters() method, which is doing very simplistic
parameter parsing.

The PT tool also allows to show the input parameters in the web interface.

So I'm voting for doing a breaking change and using parameters such as
"--input hdfs:/// --output hdfs:/// --iterations 15".

On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
behrouz.derakhs...@gmail.com> wrote:

> Will do.
>
> Thanks,
> Behrouz
>
> On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels 
> wrote:
>
> > Hi Behrouz,
> >
> > I would create a new sub-task under the original issue that introduce
> > the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
> >
> > Cheers,
> > Max
> >
> > On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
> >  wrote:
> > > Hi Max,
> > >
> > > What you said makes sense, for "ParameterTool doesn't seem to support
> > > positional arguments :) but we could fix that." should we create a
> > separate
> > > ticket or should it also be part of FLINK-2021 ?
> > >
> > > BR,
> > > Behrouz
> > >
> > >
> > > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels 
> > wrote:
> > >
> > >> Hi Behrouz,
> > >>
> > >> Thanks for starting the discussion. If I understand your question
> > >> correctly, you are asking if it breaks the training or other external
> > >> material if we convert the Flink examples to make use of the
> > >> ParameterTool?
> > >>
> > >> We could make the changes such that the examples will accept the same
> > >> parameters but use the ParameterTool internally to verify the
> > >> parameters and print usage information. I think most examples simply
> > >> use positional arguments and we could keep it that way. The only
> > >> problem is that the ParameterTool doesn't seem to support positional
> > >> arguments :) but we could fix that.
> > >>
> > >> Cheers,
> > >> Max
> > >>
> > >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
> > >>  wrote:
> > >> > Hi,
> > >> >
> > >> > I had at look at this ticket FLINK-2021
> > >> > , there isn't
> much
> > to
> > >> do
> > >> > from a technical stand point and it kinda makes sense to use the new
> > >> > "ParameterTool", since it is being used in most of the other part of
> > the
> > >> > code base.
> > >> > The only question is do we really want to do it, since I'm guessing
> > some
> > >> of
> > >> > the training materials, slides and articles are referencing these
> > >> examples
> > >> > and updating those might be a burden.
> > >> >
> > >> > Let me know what you guys think, either I can start working on it or
> > we
> > >> can
> > >> > just resolve it for good.
> > >> >
> > >> > Cheers,
> > >> > Behrouz
> > >>
> >
>

[jira] [Created] (FLINK-2620) StreamPartitioner is not properly initialized for shuffle and rebalance

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-2620:
--

 Summary: StreamPartitioner is not properly initialized for shuffle 
and rebalance
 Key: FLINK-2620
 URL: https://issues.apache.org/jira/browse/FLINK-2620
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Reporter: Matthias J. Sax


Using rebalance connection pattern, the round-robin distribution starts with 
the same receiver index in all parallel tasks. For high receiver dop and (very) 
small data, it might result in an inbalanced distribution. For example, 100 
source tasks, each emitting 10 record to 100 consumer tasks. Thus, only the 
first 10 consumer tasks receive data (10 records each) while the other 90 do 
not.

A possible fix would be, to compute different starting indexes for different 
producer tasks like

{noformat}
startIdx = (numReceivers / numSenders) * myIdx
{noformat}

For shuffle grouping, the random data generator is initialized with a unique 
seed for all parallel tasks. This should be changed such that each task uses a 
different seed.

To achieve both, the StreamPartitioner class should be extended with a
configuration / initialize method which is called on each parallel operator.

For a full discussion please see the mailing list archive: 
https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/browser



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2619) Some Scala Tests not being executed by Maven

[Chiwan Park (JIRA)]

Chiwan Park created FLINK-2619:
--

 Summary: Some Scala Tests not being executed by Maven
 Key: FLINK-2619
 URL: https://issues.apache.org/jira/browse/FLINK-2619
 Project: Flink
  Issue Type: Bug
  Components: Tests
Reporter: Chiwan Park
Priority: Critical


Some Scala Tests are not executed by Maven. Originally this issue are reported 
by [~StephanEwen]. I also executed {{mvn clean verify}} and found the same 
circumstance.

Original post is 
[here|http://mail-archives.apache.org/mod_mbox/flink-dev/201508.mbox/%3ccanc1h_tlgt-rrtybua6c0ypihhv8w1bwyb4sogahn9_cfck...@mail.gmail.com%3e]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Several Scala Tests not being executed by Maven

[Chiwan Park]

I just created the JIRA issue [1].

Regards,
Chiwan Park

[1] https://issues.apache.org/jira/browse/FLINK-2619

> On Sep 4, 2015, at 6:43 PM, Chiwan Park  wrote:
> 
> I also found the same circumstances. Although I add a fail test case in 
> ExecutionGraphRestartTest, `mvn clean verify` doesn’t fail. I will create an 
> issue covered this.
> 
> Regards,
> Chiwan Park
> 
> 
>> On Aug 29, 2015, at 10:13 PM, Stephan Ewen  wrote:
>> 
>> Hi!
>> 
>> I found quite a few tests that are not actually executed by Maven as part
>> of the builds. Some actually are in error.
>> 
>> ExecutionGraphRestartTest
>> TaskManagerLossFailsTasksTest
>> JobManagerRegistrationTest
>> ...
>> 
>> All of those are Scala tests with WordSpecLike traits. Seems that this
>> configuration has a problem with the Maven unit test plugin.
>> 
>> Anyone else observed something like this before?
>> 
>> Stephan
> 
> 
> 

Re: [ANNOUNCE] Welcome Matthias Sax as new committer

[Matthias J. Sax]

Thanks for the very warm welcome!

I am happy to work on Flink as a committer now :)

Furthermore, now I can truly claim: Trust me, I am a squirrel.

Looking forward to meet other committers and squirrels from the mailing
list in person at Flink Forward. Happy hacking.


-Matthias

On 09/02/2015 01:30 PM, Kostas Tzoumas wrote:
> The Project Management Committee (PMC) of Apache Flink has asked Matthias
> Sax to become a committer, and we are pleased to announce that he has
> accepted.
> 
> Matthias has been very active with Flink, and he is the original
> contributor of the Storm compatibility functionality.
> 
> Being a committer enables easier contribution to the project since there is no
> need to go via the pull request submission process. This should enable better
> productivity. Being a PMC member enables assistance with the management and
> to guide the direction of the project.
> 
> Please join me in welcoming Matthias as a new committer!
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (FLINK-2621) Add support for positional arguments to ParameterTool

[Behrouz Derakhshan (JIRA)]

Behrouz Derakhshan created FLINK-2621:
-

 Summary: Add support for positional arguments to ParameterTool
 Key: FLINK-2621
 URL: https://issues.apache.org/jira/browse/FLINK-2621
 Project: Flink
  Issue Type: Sub-task
  Components: flink-contrib
Reporter: Behrouz Derakhshan


While working on FLINK-2021, the need for positional argument support became 
needed, since after FLINK-2021 is merged, all training materials and guides 
that are pointing to old way of executing the examples will not work anymore. 

What I have in mind is something like this:
{code}
// -- Constructors 
public static ParameterTool fromArgs(String[] args) {
isPositional = true;
// check if arg type is positional
for(String arg:args){
if (arg.contains("--")){
isPositional = false;
}
}
if(isPositional){
return fromPositionalArgument(args);
}
...
// create ParameterTool for positional arguments
public static ParameterTool fromPositionalArgument(String[] args){
return new ParameterTool(args);
}

...
// underlying storage for arguments 
private String[] dataPositional;
private ParameterTool(String[] args){
this.dataPositional = args;
}
// return the positional argument based on the given index
public String get(int position){
if(!isPositional){
throw new RuntimeException("Arguments are of type of 
key, value");
}
return dataPositional[position];
}
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Updating the flink JAR with a single module

[Niklas Semmler]

Hi there,

I have a question on the setup of the development environment for flink.

Is it possible to update the big JAR file 
(flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink 
apparently uses for deployment, with a single module (flink-runtime)?


Recompiling all of the modules takes several minutes on my laptop, while 
compiling only the module is a matter of seconds.


My knowledge of maven is pretty limited, so I compile flink with

> mvn package -Dmaven.test.skip=true

This process is really slowing down my development cycle. I would 
appreciate your help :)


Best regards,
Niklas

--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752

Re: Several Scala Tests not being executed by Maven

[Chiwan Park]

I also found the same circumstances. Although I add a fail test case in 
ExecutionGraphRestartTest, `mvn clean verify` doesn’t fail. I will create an 
issue covered this.

Regards,
Chiwan Park


> On Aug 29, 2015, at 10:13 PM, Stephan Ewen  wrote:
> 
> Hi!
> 
> I found quite a few tests that are not actually executed by Maven as part
> of the builds. Some actually are in error.
> 
> ExecutionGraphRestartTest
> TaskManagerLossFailsTasksTest
> JobManagerRegistrationTest
> ...
> 
> All of those are Scala tests with WordSpecLike traits. Seems that this
> configuration has a problem with the Maven unit test plugin.
> 
> Anyone else observed something like this before?
> 
> Stephan

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Behrouz Derakhshan]

Hi Max,

What you said makes sense, for "ParameterTool doesn't seem to support
positional arguments :) but we could fix that." should we create a separate
ticket or should it also be part of FLINK-2021 ?

BR,
Behrouz


On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels  wrote:

> Hi Behrouz,
>
> Thanks for starting the discussion. If I understand your question
> correctly, you are asking if it breaks the training or other external
> material if we convert the Flink examples to make use of the
> ParameterTool?
>
> We could make the changes such that the examples will accept the same
> parameters but use the ParameterTool internally to verify the
> parameters and print usage information. I think most examples simply
> use positional arguments and we could keep it that way. The only
> problem is that the ParameterTool doesn't seem to support positional
> arguments :) but we could fix that.
>
> Cheers,
> Max
>
> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
>  wrote:
> > Hi,
> >
> > I had at look at this ticket FLINK-2021
> > , there isn't much to
> do
> > from a technical stand point and it kinda makes sense to use the new
> > "ParameterTool", since it is being used in most of the other part of the
> > code base.
> > The only question is do we really want to do it, since I'm guessing some
> of
> > the training materials, slides and articles are referencing these
> examples
> > and updating those might be a burden.
> >
> > Let me know what you guys think, either I can start working on it or we
> can
> > just resolve it for good.
> >
> > Cheers,
> > Behrouz
>

Re: Updating the flink JAR with a single module

[Fabian Hueske]

Hi Niklas,

if you only want to update individual submodules you can do:

cd flink-runtime // build this submodule
mvn -DskipTests clean install // will put the new flink-runtime jar into
the local Maven repository
cd ../flink-dist
mvn -DskipTests clean install // will fetch all required dependencies from
the local Maven repository without rebuilding them.

You need to have run mvn -DskipTests clean install before to have all
required dependencies in the local repository.

Cheers, Fabian

2015-09-04 11:46 GMT+02:00 Niklas Semmler :

> Hi there,
>
> I have a question on the setup of the development environment for flink.
>
> Is it possible to update the big JAR file
> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
> apparently uses for deployment, with a single module (flink-runtime)?
>
> Recompiling all of the modules takes several minutes on my laptop, while
> compiling only the module is a matter of seconds.
>
> My knowledge of maven is pretty limited, so I compile flink with
>
> > mvn package -Dmaven.test.skip=true
>
> This process is really slowing down my development cycle. I would
> appreciate your help :)
>
> Best regards,
> Niklas
>
> --
> PhD Student / Research Assistant
> INET, TU Berlin
> Room 4.029
> Marchstr 23
> 10587 Berlin
> Tel: +49 30 314 78752
>

Re: Updating the flink JAR with a single module

[Matthias J. Sax]

One more remark on this:

You need to be a little bit more careful. This approach works only, if
the initially changed module has no other dependent modules.

For example, flink-clients also depends on flink-runtime and thus, you
*might* need to rebuild flink-clients, too, before building flink-dist.
However, only if you did API changes in flink-runtime.

The basic message is, you need to understand the dependency tree. If you
did API changes, you need to rebuild all modules that depend on the
changed module.


-Matthias

On 09/04/2015 12:24 PM, Fabian Hueske wrote:
> Hi Niklas,
> 
> if you only want to update individual submodules you can do:
> 
> cd flink-runtime // build this submodule
> mvn -DskipTests clean install // will put the new flink-runtime jar into
> the local Maven repository
> cd ../flink-dist
> mvn -DskipTests clean install // will fetch all required dependencies from
> the local Maven repository without rebuilding them.
> 
> You need to have run mvn -DskipTests clean install before to have all
> required dependencies in the local repository.
> 
> Cheers, Fabian
> 
> 2015-09-04 11:46 GMT+02:00 Niklas Semmler :
> 
>> Hi there,
>>
>> I have a question on the setup of the development environment for flink.
>>
>> Is it possible to update the big JAR file
>> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
>> apparently uses for deployment, with a single module (flink-runtime)?
>>
>> Recompiling all of the modules takes several minutes on my laptop, while
>> compiling only the module is a matter of seconds.
>>
>> My knowledge of maven is pretty limited, so I compile flink with
>>
>>> mvn package -Dmaven.test.skip=true
>>
>> This process is really slowing down my development cycle. I would
>> appreciate your help :)
>>
>> Best regards,
>> Niklas
>>
>> --
>> PhD Student / Research Assistant
>> INET, TU Berlin
>> Room 4.029
>> Marchstr 23
>> 10587 Berlin
>> Tel: +49 30 314 78752
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Behrouz Derakhshan]

Yes, I was referring mostly to blog posts and other websites and was
wondering if breaking them is an issue or not.
I have already created a subtask to add support for positional arguments (
FLINK-2621 ), so the
examples would be backward compatible.
The problem with that is, we have to detect from the arguments to the
program, if they are positional or key/value and parse them accordingly.
But if everyone is OK with completely switching to ParameterTool and
breaking the support for the old way of executing the examples, then my job
would be also a lot easier.



On Fri, Sep 4, 2015 at 2:34 PM, Robert Metzger  wrote:

> If you are referring to this training material (
>
> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java
> ),
> some of the examples are actually already using the ParameterTool.
>
> The problem are probably websites / blogposts etc. that show how to use the
> Flink examples. But I think its fine to break these. All example jars
> contain the version number. If the way we pass arguments to the examples
> changes between 0.9 and 0.10, that should be fine.
>
> I think using the ParameterTool for the examples will improve the
> readability of the examples a lot. Right now, all examples have a
> (copy-pasted) parseParameters() method, which is doing very simplistic
> parameter parsing.
>
> The PT tool also allows to show the input parameters in the web interface.
>
> So I'm voting for doing a breaking change and using parameters such as
> "--input hdfs:/// --output hdfs:/// --iterations 15".
>
> On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
> behrouz.derakhs...@gmail.com> wrote:
>
> > Will do.
> >
> > Thanks,
> > Behrouz
> >
> > On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels 
> > wrote:
> >
> > > Hi Behrouz,
> > >
> > > I would create a new sub-task under the original issue that introduce
> > > the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
> > >
> > > Cheers,
> > > Max
> > >
> > > On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
> > >  wrote:
> > > > Hi Max,
> > > >
> > > > What you said makes sense, for "ParameterTool doesn't seem to support
> > > > positional arguments :) but we could fix that." should we create a
> > > separate
> > > > ticket or should it also be part of FLINK-2021 ?
> > > >
> > > > BR,
> > > > Behrouz
> > > >
> > > >
> > > > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels 
> > > wrote:
> > > >
> > > >> Hi Behrouz,
> > > >>
> > > >> Thanks for starting the discussion. If I understand your question
> > > >> correctly, you are asking if it breaks the training or other
> external
> > > >> material if we convert the Flink examples to make use of the
> > > >> ParameterTool?
> > > >>
> > > >> We could make the changes such that the examples will accept the
> same
> > > >> parameters but use the ParameterTool internally to verify the
> > > >> parameters and print usage information. I think most examples simply
> > > >> use positional arguments and we could keep it that way. The only
> > > >> problem is that the ParameterTool doesn't seem to support positional
> > > >> arguments :) but we could fix that.
> > > >>
> > > >> Cheers,
> > > >> Max
> > > >>
> > > >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
> > > >>  wrote:
> > > >> > Hi,
> > > >> >
> > > >> > I had at look at this ticket FLINK-2021
> > > >> > , there isn't
> > much
> > > to
> > > >> do
> > > >> > from a technical stand point and it kinda makes sense to use the
> new
> > > >> > "ParameterTool", since it is being used in most of the other part
> of
> > > the
> > > >> > code base.
> > > >> > The only question is do we really want to do it, since I'm
> guessing
> > > some
> > > >> of
> > > >> > the training materials, slides and articles are referencing these
> > > >> examples
> > > >> > and updating those might be a burden.
> > > >> >
> > > >> > Let me know what you guys think, either I can start working on it
> or
> > > we
> > > >> can
> > > >> > just resolve it for good.
> > > >> >
> > > >> > Cheers,
> > > >> > Behrouz
> > > >>
> > >
> >
>

[jira] [Created] (FLINK-2624) RabbitMQ source / sink should participate in checkpointing

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-2624:
---

 Summary: RabbitMQ source / sink should participate in checkpointing
 Key: FLINK-2624
 URL: https://issues.apache.org/jira/browse/FLINK-2624
 Project: Flink
  Issue Type: Bug
  Components: Streaming Connectors
Affects Versions: 0.10
Reporter: Stephan Ewen


The RabbitMQ connector does not offer any fault tolerance guarantees right now, 
because it does not participate in the checkpointing.

We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is 
integrated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2623) Extend API completeness check to check signatures

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-2623:


 Summary: Extend API completeness check to check signatures
 Key: FLINK-2623
 URL: https://issues.apache.org/jira/browse/FLINK-2623
 Project: Flink
  Issue Type: Improvement
Reporter: Till Rohrmann


Currently, the API completeness check only verifies that the Java and Scala API 
both have the same API methods based on their name. This, however, does not 
guarantee that both API's cover the same functionality. 

The Scala Streaming API, for example, supports the {{writeAsText}} method. 
However, it does not let you specify the {{WriteMode}}, which is supported by 
the Java Streaming API. Maybe it is possible to extend the API completeness 
check to also verify the signatures or parts of the signatures of the API 
methods to avoid a divergence in functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Updating the flink JAR with a single module

[Niklas Semmler]

Hello Fabian, Hello Matthias,

Thank you very much! I am not making API changes at the moment, so I 
should be fine with this approach.


Unfortunately the command "mvn -DskipTests clean install" in the main 
folder didn't work for me (it hanged in the middle of the operation), so 
I am using a more brute force method. I compile the flink-runtime jar 
file just as Fabian described and then use it to overwrite the content 
in the "lib/flink-dist-0.9-SNAPSHOT.jar".


It is not really a clean path, but this method has reduced my compile 
time down to 1 minute, which is a great improvement :)


Best,
Niklas

On 04.09.2015 12:59, Matthias J. Sax wrote:

One more remark on this:

You need to be a little bit more careful. This approach works only, if
the initially changed module has no other dependent modules.

For example, flink-clients also depends on flink-runtime and thus, you
*might* need to rebuild flink-clients, too, before building flink-dist.
However, only if you did API changes in flink-runtime.

The basic message is, you need to understand the dependency tree. If you
did API changes, you need to rebuild all modules that depend on the
changed module.


-Matthias

On 09/04/2015 12:24 PM, Fabian Hueske wrote:

Hi Niklas,

if you only want to update individual submodules you can do:

cd flink-runtime // build this submodule
mvn -DskipTests clean install // will put the new flink-runtime jar into
the local Maven repository
cd ../flink-dist
mvn -DskipTests clean install // will fetch all required dependencies from
the local Maven repository without rebuilding them.

You need to have run mvn -DskipTests clean install before to have all
required dependencies in the local repository.

Cheers, Fabian

2015-09-04 11:46 GMT+02:00 Niklas Semmler :


Hi there,

I have a question on the setup of the development environment for flink.

Is it possible to update the big JAR file
(flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
apparently uses for deployment, with a single module (flink-runtime)?

Recompiling all of the modules takes several minutes on my laptop, while
compiling only the module is a matter of seconds.

My knowledge of maven is pretty limited, so I compile flink with


mvn package -Dmaven.test.skip=true


This process is really slowing down my development cycle. I would
appreciate your help :)

Best regards,
Niklas

--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752







--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Maximilian Michels]

Hi Behrouz,

Thanks for starting the discussion. If I understand your question
correctly, you are asking if it breaks the training or other external
material if we convert the Flink examples to make use of the
ParameterTool?

We could make the changes such that the examples will accept the same
parameters but use the ParameterTool internally to verify the
parameters and print usage information. I think most examples simply
use positional arguments and we could keep it that way. The only
problem is that the ParameterTool doesn't seem to support positional
arguments :) but we could fix that.

Cheers,
Max

On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
 wrote:
> Hi,
>
> I had at look at this ticket FLINK-2021
> , there isn't much to do
> from a technical stand point and it kinda makes sense to use the new
> "ParameterTool", since it is being used in most of the other part of the
> code base.
> The only question is do we really want to do it, since I'm guessing some of
> the training materials, slides and articles are referencing these examples
> and updating those might be a burden.
>
> Let me know what you guys think, either I can start working on it or we can
> just resolve it for good.
>
> Cheers,
> Behrouz