Re: [Discuss] Some questions on flink-table-store micro benchmark

[Yun Tang]

Thanks for the response. I think we can also incorporate the performance 
regression monitoring with Slack in the feature, which Yanfei ever did [1].


[1] https://lists.apache.org/thread/b75j2sgzhf25298p4982fy9tzdjrvght

Best
Yun Tang

From: Jingsong Li 
Sent: Monday, February 27, 2023 18:02
To: dev@flink.apache.org 
Subject: Re: [Discuss] Some questions on flink-table-store micro benchmark

Hi Yun and Shammon,

> track the performance changes of the micro benchmark

I think we can create a github action for this. Print results everyday.

Best,
Jingsong

On Mon, Feb 27, 2023 at 5:59 PM Shammon FY  wrote:
>
> Hi jingsong
>
> Getting rid of JMH is a good idea. For the second point, how can we track
> the performance changes of the micro benchmark? What do you think?
>
> Best,
> Shammon
>
> On Mon, Feb 27, 2023 at 10:57 AM Jingsong Li  wrote:
>
> > Thanks Yun.
> >
> > Another way is we can get rid of JMH, something like Spark
> > `org.apache.spark.benchmark.Benchmark` can replace JMH.
> >
> > Best,
> > Jingsong
> >
> > On Mon, Feb 27, 2023 at 1:24 AM Yun Tang  wrote:
> > >
> > > Hi dev,
> > >
> > > I just noticed that flink-table-store had introduced the micro benchmark
> > module [1] to test the basic performance. And I have two questions here.
> > > First of all, we might not be able to keep the micro benchmark, which is
> > based on JMH, in the main repo of flink-table-store. This is because JMH is
> > under GPL license, which is not compliant with Apache-2 license. That's why
> > Flink moved the flink-benchmark out [2].
> > >
> > > Secondly, I try to run the micro benchmark locally but it seems failed,
> > I just wonder can we make the flink-table-store's micro benchmark could be
> > periodically executed just as flink-benchmarks did on
> > http://codespeed.dak8s.net:8080? Please correct me if such daily
> > benchmark has been set up.
> > > Moreover, maybe we can also consider to integrate this micro benchmark
> > notification in the Slack channel just as what flink-benchmarks did.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-29636
> > > [2] https://issues.apache.org/jira/browse/FLINK-2973
> > >
> > > Best
> > > Yun Tang
> >

Re: [Discuss] Some questions on flink-table-store micro benchmark

[Jingsong Li]

Hi Yun and Shammon,

> track the performance changes of the micro benchmark

I think we can create a github action for this. Print results everyday.

Best,
Jingsong

On Mon, Feb 27, 2023 at 5:59 PM Shammon FY  wrote:
>
> Hi jingsong
>
> Getting rid of JMH is a good idea. For the second point, how can we track
> the performance changes of the micro benchmark? What do you think?
>
> Best,
> Shammon
>
> On Mon, Feb 27, 2023 at 10:57 AM Jingsong Li  wrote:
>
> > Thanks Yun.
> >
> > Another way is we can get rid of JMH, something like Spark
> > `org.apache.spark.benchmark.Benchmark` can replace JMH.
> >
> > Best,
> > Jingsong
> >
> > On Mon, Feb 27, 2023 at 1:24 AM Yun Tang  wrote:
> > >
> > > Hi dev,
> > >
> > > I just noticed that flink-table-store had introduced the micro benchmark
> > module [1] to test the basic performance. And I have two questions here.
> > > First of all, we might not be able to keep the micro benchmark, which is
> > based on JMH, in the main repo of flink-table-store. This is because JMH is
> > under GPL license, which is not compliant with Apache-2 license. That's why
> > Flink moved the flink-benchmark out [2].
> > >
> > > Secondly, I try to run the micro benchmark locally but it seems failed,
> > I just wonder can we make the flink-table-store's micro benchmark could be
> > periodically executed just as flink-benchmarks did on
> > http://codespeed.dak8s.net:8080? Please correct me if such daily
> > benchmark has been set up.
> > > Moreover, maybe we can also consider to integrate this micro benchmark
> > notification in the Slack channel just as what flink-benchmarks did.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-29636
> > > [2] https://issues.apache.org/jira/browse/FLINK-2973
> > >
> > > Best
> > > Yun Tang
> >

Re: [Discuss] Some questions on flink-table-store micro benchmark

[Shammon FY]

Hi jingsong

Getting rid of JMH is a good idea. For the second point, how can we track
the performance changes of the micro benchmark? What do you think?

Best,
Shammon

On Mon, Feb 27, 2023 at 10:57 AM Jingsong Li  wrote:

> Thanks Yun.
>
> Another way is we can get rid of JMH, something like Spark
> `org.apache.spark.benchmark.Benchmark` can replace JMH.
>
> Best,
> Jingsong
>
> On Mon, Feb 27, 2023 at 1:24 AM Yun Tang  wrote:
> >
> > Hi dev,
> >
> > I just noticed that flink-table-store had introduced the micro benchmark
> module [1] to test the basic performance. And I have two questions here.
> > First of all, we might not be able to keep the micro benchmark, which is
> based on JMH, in the main repo of flink-table-store. This is because JMH is
> under GPL license, which is not compliant with Apache-2 license. That's why
> Flink moved the flink-benchmark out [2].
> >
> > Secondly, I try to run the micro benchmark locally but it seems failed,
> I just wonder can we make the flink-table-store's micro benchmark could be
> periodically executed just as flink-benchmarks did on
> http://codespeed.dak8s.net:8080? Please correct me if such daily
> benchmark has been set up.
> > Moreover, maybe we can also consider to integrate this micro benchmark
> notification in the Slack channel just as what flink-benchmarks did.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-29636
> > [2] https://issues.apache.org/jira/browse/FLINK-2973
> >
> > Best
> > Yun Tang
>

Re: [Discuss] Some questions on flink-table-store micro benchmark

[Jingsong Li]

Thanks Yun.

Another way is we can get rid of JMH, something like Spark
`org.apache.spark.benchmark.Benchmark` can replace JMH.

Best,
Jingsong

On Mon, Feb 27, 2023 at 1:24 AM Yun Tang  wrote:
>
> Hi dev,
>
> I just noticed that flink-table-store had introduced the micro benchmark 
> module [1] to test the basic performance. And I have two questions here.
> First of all, we might not be able to keep the micro benchmark, which is 
> based on JMH, in the main repo of flink-table-store. This is because JMH is 
> under GPL license, which is not compliant with Apache-2 license. That's why 
> Flink moved the flink-benchmark out [2].
>
> Secondly, I try to run the micro benchmark locally but it seems failed, I 
> just wonder can we make the flink-table-store's micro benchmark could be 
> periodically executed just as flink-benchmarks did on 
> http://codespeed.dak8s.net:8080? Please correct me if such daily benchmark 
> has been set up.
> Moreover, maybe we can also consider to integrate this micro benchmark 
> notification in the Slack channel just as what flink-benchmarks did.
>
> [1] https://issues.apache.org/jira/browse/FLINK-29636
> [2] https://issues.apache.org/jira/browse/FLINK-2973
>
> Best
> Yun Tang

[Discuss] Some questions on flink-table-store micro benchmark

[Yun Tang]

Hi dev,

I just noticed that flink-table-store had introduced the micro benchmark module 
[1] to test the basic performance. And I have two questions here.
First of all, we might not be able to keep the micro benchmark, which is based 
on JMH, in the main repo of flink-table-store. This is because JMH is under GPL 
license, which is not compliant with Apache-2 license. That's why Flink moved 
the flink-benchmark out [2].

Secondly, I try to run the micro benchmark locally but it seems failed, I just 
wonder can we make the flink-table-store's micro benchmark could be 
periodically executed just as flink-benchmarks did on 
http://codespeed.dak8s.net:8080? Please correct me if such daily benchmark has 
been set up.
Moreover, maybe we can also consider to integrate this micro benchmark 
notification in the Slack channel just as what flink-benchmarks did.

[1] https://issues.apache.org/jira/browse/FLINK-29636
[2] https://issues.apache.org/jira/browse/FLINK-2973

Best
Yun Tang

Re: Questions about Flink Table Store

[Caizhi Weng]

Hi Alexander!

Thanks for your interest in Flink Table Store and I'm glad to share my
thoughts with you.

Given that there is always a single writer to a stream, in what situations
> can concurrent writes ever happen to Flink Table Store?


I'm not the author of FLIP so I'm not sure what "writer" refers to in this
FLIP.

Currently, each parallelism of Flink Table Store sink will write to its own
bucket, so no concurrent write will occur at bucket level for a single job.
However a table can consist of multiple buckets (records from all buckets
will be merged when reading the table, aka the merge-on-read strategy), so
within a Flink job, multiple sink parallelisms can write into their own
buckets concurrently.

Flink Table Store also supports multiple jobs writing the same table, even
the same bucket. Conflicts will be resolved by the commit process [1],
which is an optimistic concurrency control.

If there are concurrent INSERTs into a table baked by Flink Table Store,
> how and by what component are they serialized?


Do you mean how to determine the order of INSERTs?

The order of INSERTs are determined by their sequence number. When a writer
is created, it will read current sequence number from disk. As stated
above, no concurrent writes will occur for a single job. But for multiple
jobs running at the same time, they may create INSERTs with the same
sequence number. In that case the final result might be a mix of two jobs.
This is currently OK because Flink Table Store currently only provides an
isolation level of snapshot isolation, not serializable.

Is Flink Table Store going to support ACID transactions?


It's not on our road map for the short term.

Do Flink Table Store snapshots correspond 1:1 to Flink checkpoints?


Snapshots are committed when a checkpoint is notified as complete. However
Flink's checkpoint notification is a best-effort notification, meaning that
snapshots may be skipped. Also each commit currently consists up to 2
snapshots, one for all the new level 0 files and another for all compact
changes.

Does Flink Table Store (plan to) support secondary indexes?


Yes. We have such plan but haven't decided when to implement it.

Is there an open roadmap for this Flink Table Store?


As far as I know we currently don't. Maybe Jingsong Lee will know more
about this.

[1]
https://github.com/apache/flink-table-store/blob/master/flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/FileStoreCommitImpl.java


Leonard Xu  于2022年11月8日周二 22:47写道：

> Hi, Alexander
>
> Happy to hear that you’re willing to contribute to Flink, the answer of
> these specific tech design/functionality questions may need to dig the code.
> I’m not familiar with  FTS code base, but I’v CC my colleagues who are the
> core maintainers of FTS and maybe they can give your some insights.
>
>
> Best,
> Leonard
>
> > 2022年11月7日 下午10:59，Alexander Sorokoumov 
> 写道：
> >
> > I’m Alexander from Confluent. I am new to Flink and its community. I
> would
> > like to contribute to the Flink Table Store, but am missing certain
> > details. Can someone please clarify the points mentioned below to me?
> >
> >   - Given that there is always a single writer to a stream, in what
> >   situations can concurrent writes ever happen to Flink Table Store?
> FLIP-188
> >   mentions reprocessing and snapshot generation, but I do not understand
> how
> >   these cases can lead to more than a single writer.
> >   - If there are concurrent INSERTs into a table baked by Flink Table
> >   Store, how and by what component are they serialized?
> >   - Is Flink Table Store going to support ACID transactions?
> >   - Do Flink Table Store snapshots correspond 1:1 to Flink checkpoints?
> >   - Does Flink Table Store (plan to) support secondary indexes?
> >   - Is there an open roadmap for this Flink Table Store?
> >
> > Thank you,
> > Alexander
>
>

Re: Questions about Flink Table Store

[Leonard Xu]

Hi, Alexander

Happy to hear that you’re willing to contribute to Flink, the answer of these 
specific tech design/functionality questions may need to dig the code.
I’m not familiar with  FTS code base, but I’v CC my colleagues who are the core 
maintainers of FTS and maybe they can give your some insights.


Best,
Leonard

> 2022年11月7日 下午10:59，Alexander Sorokoumov  写道：
> 
> I’m Alexander from Confluent. I am new to Flink and its community. I would
> like to contribute to the Flink Table Store, but am missing certain
> details. Can someone please clarify the points mentioned below to me?
> 
>   - Given that there is always a single writer to a stream, in what
>   situations can concurrent writes ever happen to Flink Table Store? FLIP-188
>   mentions reprocessing and snapshot generation, but I do not understand how
>   these cases can lead to more than a single writer.
>   - If there are concurrent INSERTs into a table baked by Flink Table
>   Store, how and by what component are they serialized?
>   - Is Flink Table Store going to support ACID transactions?
>   - Do Flink Table Store snapshots correspond 1:1 to Flink checkpoints?
>   - Does Flink Table Store (plan to) support secondary indexes?
>   - Is there an open roadmap for this Flink Table Store?
> 
> Thank you,
> Alexander

Questions about Flink Table Store

[Alexander Sorokoumov]

I’m Alexander from Confluent. I am new to Flink and its community. I would
like to contribute to the Flink Table Store, but am missing certain
details. Can someone please clarify the points mentioned below to me?

   - Given that there is always a single writer to a stream, in what
   situations can concurrent writes ever happen to Flink Table Store? FLIP-188
   mentions reprocessing and snapshot generation, but I do not understand how
   these cases can lead to more than a single writer.
   - If there are concurrent INSERTs into a table baked by Flink Table
   Store, how and by what component are they serialized?
   - Is Flink Table Store going to support ACID transactions?
   - Do Flink Table Store snapshots correspond 1:1 to Flink checkpoints?
   - Does Flink Table Store (plan to) support secondary indexes?
   - Is there an open roadmap for this Flink Table Store?

Thank you,
Alexander

Re: Questions on flink

[Jamie Grier]

Hi Govindarajan,

I've put some answers in-line below..

On Sat, Sep 24, 2016 at 7:32 PM, Govindarajan Srinivasaraghavan <
govindragh...@gmail.com> wrote:

> Hi,
>
> I'm working on apache flink for data streaming and I have few questions.
> Any help is greatly appreciated. Thanks.
>
> 1) Are there any restrictions on creating tumbling windows. For example,
> if I want to create a tumbling window per user id for 2 secs and let’s say
> if I have more than 10 million user id's would that be a problem. (I'm
> using keyBy user id and then creating a timeWindow for 2 secs)? How are
> these windows maintained internally in flink?
>

That should not be a problem in general.  An important question may be how
many unique keys will you see in two seconds.  This is more important than
your total key cardinality of 10 Million and probably a *much* smaller
number unless your input message rate is really high.

>
> 2) I looked at rebalance for round robin partitioning. Let’s say I have a
> cluster set up and if I have a parallelism of 1 for source and if I do a
> rebalance, will my data be shuffled across machines to improve performance?
> If so is there a specific port using which the data is transferred to other
> nodes in the cluster?
>

Yes, rebalance() does a round-robin distribution of messages to other
machines in the cluster.  There is not a specific port used for each
TaskManager to communicate on but rather an available port is assigned at
runtime.  This is the default.  You can also set this to a specific port if
you have reason and a lot depends on how you will deploy -- via YARN or as
a standalone Flink cluster.


>
> 3) Are there any limitations on state maintenance? I'm planning to
> maintain some user id related data which could grow very large. I read
> about flink using rocks db to maintain the state. Just wanted to check if
> there are any limitations on how much data can be maintained?
>

Yes, there are limits.  The total data that can be maintained today is
determined by the fact that Flink has to periodically snapshot this data
and copy it to a persistent storage system such as HDFS whether you are
using RocksDB or not.  The aggregate bandwidth required to your storage
system (like HDFS) is your total Flink state size multiplied by your Flink
checkpoint interval.


> 4) Also where is the state maintained if the amount of data is less? (I
> guess in JVM memory) If I have several machines on my cluster can every
> node get the current state version?
>

I'm not exactly sure what you're asking here.  All data is check-pointed to
a persistent store which must be accessible from each machine in the
cluster.


> 5) I need a way to send external configuration changes to flink. Lets say
> there is a new parameter that has to added or an external change which has
> to be updated inside flink's state, how can this be done?
>

The typical way to do this is to consume that configuration as a stream and
hold the configuration internally in the state of a particular user
function.


>
> Thanks
>



-- 

Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier 
ja...@data-artisans.com

Questions on flink

[Govindarajan Srinivasaraghavan]

Hi,

I'm working on apache flink for data streaming and I have few questions.
Any help is greatly appreciated. Thanks.

1) Are there any restrictions on creating tumbling windows. For example, if
I want to create a tumbling window per user id for 2 secs and let’s say if
I have more than 10 million user id's would that be a problem. (I'm using
keyBy user id and then creating a timeWindow for 2 secs)? How are these
windows maintained internally in flink?

2) I looked at rebalance for round robin partitioning. Let’s say I have a
cluster set up and if I have a parallelism of 1 for source and if I do a
rebalance, will my data be shuffled across machines to improve performance?
If so is there a specific port using which the data is transferred to other
nodes in the cluster?

3) Are there any limitations on state maintenance? I'm planning to maintain
some user id related data which could grow very large. I read about flink
using rocks db to maintain the state. Just wanted to check if there are any
limitations on how much data can be maintained?

4) Also where is the state maintained if the amount of data is less? (I
guess in JVM memory) If I have several machines on my cluster can every
node get the current state version?

5) I need a way to send external configuration changes to flink. Lets say
there is a new parameter that has to added or an external change which has
to be updated inside flink's state, how can this be done?

Thanks

Re: Questions about flink-streaming-examples

[Matthias J. Sax]

Thanks for clarifying Marton!

I was on the latest build already. However, my local maven repository
contained old jars. After removing all flink-jars from my local maven
repository it works!

Why does maven no automatically update the local repository?


-Matthias



On 02/26/2015 09:20 AM, Márton Balassi wrote:
 Dear Mathias,
 
 Thanks for reporting the issue. I have successfully built
 flink-streaming-examples with maven, you can depend on test classes, the
 following in the pom does the trick:
 
 dependency
   groupIdorg.apache.flink/groupId
   artifactIdflink-streaming-core/artifactId
   version${project.version}/version
   scopetest/scope
   classifiertests/classifier
 /dependency
 
 This tells maven that the test classes of the examples (scope) might depend
 on the tests of the core (classifier). Without explicitly stating the tests
 classifier indeed maven only uses the classes under src/main.
 
 I think the problem occurred because you only built
 flink-streaming-examples and you might have used an outdated version of the
 flink-streaming-core as a dependency installed to your local maven
 repository - which did not include the StreamingTestBase yet. The change
 that we are talking is about a week old, I wrote it.
 
 Kind regards,
 
 Marton
 
 On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax 
 mj...@informatik.hu-berlin.de wrote:
 
 Hi,

 I just build flink-streaming and avoid the problem. I guess, that the
 issue is related to the module structure and dependencies.
 flink-streaming-examples uses
 org.apache.flink.streaming.util.StreamingProgramTestBase (that is
 defined in flink-streaming-core/src/TEST

 From my understanding, maven does not support dependencies into src/test
 but only into src/main (maybe I am wrong about this).

 I would suggest, to move StreamingProgramTestBase into a new module
 flink-streaming-testUtils (into src/main) and change the dependency of
 flink-streaming-examples accordingly.

 Why the build work on the parent-project, is still a riddle to me... But
 it seems that the dependency into scr/test can be resolved for this case.


 -Matthias




 On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
 Hi,
 Were you able to solve this? It seams the examples depend on
 flink-streaming-core”. I think you have to build it before.

 On Feb 25, 2015, at 10:04 PM, Matthias J. Sax 
 mj...@informatik.hu-berlin.de wrote:

 Thanks!

 Even if I am not a building and/or maven expert, it seems to me that the
 dependencies are not configures correctly... No clue how to fix it. But
 it might be a good idea to have a look into it IMHO.

 -Matthias


 On 02/25/2015 05:07 PM, Stephan Ewen wrote:
 Hi Matthias!

 Can you try mvn clean package? It package is generally preferable to
 compile, in my opinion.

 There may be a dependency through a test jar that is not properly
 handled
 when you do not execute the package goal.

 Stephan
 Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:

 Hi Matthias,

 Did you really pull from the latest master? I just tried to compile
 flink-streaming-examples using mvn clean compile test-compile and it
 worked.

 Best regards,
 Max

 On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
 mj...@informatik.hu-berlin.de wrote:
 Hi,

 I tried to build flink-streaming-examples, but got an compilation
 error.
 If I build flink-streaming (that implicitly builds
 flink-streaming-examples), it works.

 I tried it on a fresh clone of flink/master using mvn clean compile
 test-compile

 Can anybody explain this behavior?


 -Matthias




 mjsax@T420s-dbis-mjsax
 :~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
 mvn clean compile test-compile
 [INFO] Scanning for projects...
 [WARNING]
 [WARNING] Some problems were encountered while building the
 effective
 model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique
 but found duplicate declaration of plugin
 org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
 [WARNING]
 [WARNING] It is highly recommended to fix these problems because
 they
 threaten the stability of your build.
 [WARNING]
 [WARNING] For this reason, future Maven versions might no longer
 support building such malformed projects.
 [WARNING]
 [INFO]
 [INFO]

 
 [INFO] Building flink-streaming-examples 0.9-SNAPSHOT
 [INFO]

 
 [INFO]
 [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
 flink-streaming-examples ---
 [INFO] Deleting

 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target
 [INFO]
 [INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
 flink-streaming-examples ---
 [INFO]
 [INFO]
 [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
 flink-streaming-examples ---
 [INFO]
 [INFO] --- 

Re: Questions about flink-streaming-examples

[Robert Metzger]

To update the local repository, you have to do execute the install goal.
I can recommend to always do a  mvn clean install

On Thu, Feb 26, 2015 at 10:11 AM, Matthias J. Sax 
mj...@informatik.hu-berlin.de wrote:

 Thanks for clarifying Marton!

 I was on the latest build already. However, my local maven repository
 contained old jars. After removing all flink-jars from my local maven
 repository it works!

 Why does maven no automatically update the local repository?


 -Matthias



 On 02/26/2015 09:20 AM, Márton Balassi wrote:
  Dear Mathias,
 
  Thanks for reporting the issue. I have successfully built
  flink-streaming-examples with maven, you can depend on test classes, the
  following in the pom does the trick:
 
  dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
classifiertests/classifier
  /dependency
 
  This tells maven that the test classes of the examples (scope) might
 depend
  on the tests of the core (classifier). Without explicitly stating the
 tests
  classifier indeed maven only uses the classes under src/main.
 
  I think the problem occurred because you only built
  flink-streaming-examples and you might have used an outdated version of
 the
  flink-streaming-core as a dependency installed to your local maven
  repository - which did not include the StreamingTestBase yet. The change
  that we are talking is about a week old, I wrote it.
 
  Kind regards,
 
  Marton
 
  On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax 
  mj...@informatik.hu-berlin.de wrote:
 
  Hi,
 
  I just build flink-streaming and avoid the problem. I guess, that the
  issue is related to the module structure and dependencies.
  flink-streaming-examples uses
  org.apache.flink.streaming.util.StreamingProgramTestBase (that is
  defined in flink-streaming-core/src/TEST
 
  From my understanding, maven does not support dependencies into src/test
  but only into src/main (maybe I am wrong about this).
 
  I would suggest, to move StreamingProgramTestBase into a new module
  flink-streaming-testUtils (into src/main) and change the dependency of
  flink-streaming-examples accordingly.
 
  Why the build work on the parent-project, is still a riddle to me... But
  it seems that the dependency into scr/test can be resolved for this
 case.
 
 
  -Matthias
 
 
 
 
  On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
  Hi,
  Were you able to solve this? It seams the examples depend on
  flink-streaming-core”. I think you have to build it before.
 
  On Feb 25, 2015, at 10:04 PM, Matthias J. Sax 
  mj...@informatik.hu-berlin.de wrote:
 
  Thanks!
 
  Even if I am not a building and/or maven expert, it seems to me that
 the
  dependencies are not configures correctly... No clue how to fix it.
 But
  it might be a good idea to have a look into it IMHO.
 
  -Matthias
 
 
  On 02/25/2015 05:07 PM, Stephan Ewen wrote:
  Hi Matthias!
 
  Can you try mvn clean package? It package is generally preferable
 to
  compile, in my opinion.
 
  There may be a dependency through a test jar that is not properly
  handled
  when you do not execute the package goal.
 
  Stephan
  Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
 
  Hi Matthias,
 
  Did you really pull from the latest master? I just tried to compile
  flink-streaming-examples using mvn clean compile test-compile and
 it
  worked.
 
  Best regards,
  Max
 
  On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
  mj...@informatik.hu-berlin.de wrote:
  Hi,
 
  I tried to build flink-streaming-examples, but got an compilation
  error.
  If I build flink-streaming (that implicitly builds
  flink-streaming-examples), it works.
 
  I tried it on a fresh clone of flink/master using mvn clean
 compile
  test-compile
 
  Can anybody explain this behavior?
 
 
  -Matthias
 
 
 
 
  mjsax@T420s-dbis-mjsax
 
 :~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
  mvn clean compile test-compile
  [INFO] Scanning for projects...
  [WARNING]
  [WARNING] Some problems were encountered while building the
  effective
  model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
  [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be
 unique
  but found duplicate declaration of plugin
  org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
  [WARNING]
  [WARNING] It is highly recommended to fix these problems because
  they
  threaten the stability of your build.
  [WARNING]
  [WARNING] For this reason, future Maven versions might no longer
  support building such malformed projects.
  [WARNING]
  [INFO]
  [INFO]
 
  
  [INFO] Building flink-streaming-examples 0.9-SNAPSHOT
  [INFO]
 
  
  [INFO]
  [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
  flink-streaming-examples ---
  [INFO] 

Re: Questions about flink-streaming-examples

[Till Rohrmann]

If the streaming-examples module uses the classifier tag to add the
test-core dependency then we should change it into type tag as
recommended by maven [1]. Otherwise it might come to build failures if the
install lifecycle is not executed.

The dependency import should look like:

dependency
  groupIdorg.apache.flink/groupId
  artifactIdflink-streaming-core/artifactId
  version${project.version}/version
  scopetest/scope
  typetest-jar/type
/dependency

[1] http://maven.apache.org/guides/mini/guide-attached-tests.html

On Thu, Feb 26, 2015 at 10:31 AM, Robert Metzger rmetz...@apache.org
wrote:

 To update the local repository, you have to do execute the install goal.
 I can recommend to always do a  mvn clean install

 On Thu, Feb 26, 2015 at 10:11 AM, Matthias J. Sax 
 mj...@informatik.hu-berlin.de wrote:

  Thanks for clarifying Marton!
 
  I was on the latest build already. However, my local maven repository
  contained old jars. After removing all flink-jars from my local maven
  repository it works!
 
  Why does maven no automatically update the local repository?
 
 
  -Matthias
 
 
 
  On 02/26/2015 09:20 AM, Márton Balassi wrote:
   Dear Mathias,
  
   Thanks for reporting the issue. I have successfully built
   flink-streaming-examples with maven, you can depend on test classes,
 the
   following in the pom does the trick:
  
   dependency
 groupIdorg.apache.flink/groupId
 artifactIdflink-streaming-core/artifactId
 version${project.version}/version
 scopetest/scope
 classifiertests/classifier
   /dependency
  
   This tells maven that the test classes of the examples (scope) might
  depend
   on the tests of the core (classifier). Without explicitly stating the
  tests
   classifier indeed maven only uses the classes under src/main.
  
   I think the problem occurred because you only built
   flink-streaming-examples and you might have used an outdated version of
  the
   flink-streaming-core as a dependency installed to your local maven
   repository - which did not include the StreamingTestBase yet. The
 change
   that we are talking is about a week old, I wrote it.
  
   Kind regards,
  
   Marton
  
   On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax 
   mj...@informatik.hu-berlin.de wrote:
  
   Hi,
  
   I just build flink-streaming and avoid the problem. I guess, that
 the
   issue is related to the module structure and dependencies.
   flink-streaming-examples uses
   org.apache.flink.streaming.util.StreamingProgramTestBase (that is
   defined in flink-streaming-core/src/TEST
  
   From my understanding, maven does not support dependencies into
 src/test
   but only into src/main (maybe I am wrong about this).
  
   I would suggest, to move StreamingProgramTestBase into a new module
   flink-streaming-testUtils (into src/main) and change the dependency
 of
   flink-streaming-examples accordingly.
  
   Why the build work on the parent-project, is still a riddle to me...
 But
   it seems that the dependency into scr/test can be resolved for this
  case.
  
  
   -Matthias
  
  
  
  
   On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
   Hi,
   Were you able to solve this? It seams the examples depend on
   flink-streaming-core”. I think you have to build it before.
  
   On Feb 25, 2015, at 10:04 PM, Matthias J. Sax 
   mj...@informatik.hu-berlin.de wrote:
  
   Thanks!
  
   Even if I am not a building and/or maven expert, it seems to me that
  the
   dependencies are not configures correctly... No clue how to fix it.
  But
   it might be a good idea to have a look into it IMHO.
  
   -Matthias
  
  
   On 02/25/2015 05:07 PM, Stephan Ewen wrote:
   Hi Matthias!
  
   Can you try mvn clean package? It package is generally preferable
  to
   compile, in my opinion.
  
   There may be a dependency through a test jar that is not properly
   handled
   when you do not execute the package goal.
  
   Stephan
   Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
  
   Hi Matthias,
  
   Did you really pull from the latest master? I just tried to
 compile
   flink-streaming-examples using mvn clean compile test-compile
 and
  it
   worked.
  
   Best regards,
   Max
  
   On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
   mj...@informatik.hu-berlin.de wrote:
   Hi,
  
   I tried to build flink-streaming-examples, but got an compilation
   error.
   If I build flink-streaming (that implicitly builds
   flink-streaming-examples), it works.
  
   I tried it on a fresh clone of flink/master using mvn clean
  compile
   test-compile
  
   Can anybody explain this behavior?
  
  
   -Matthias
  
  
  
  
   mjsax@T420s-dbis-mjsax
  
 
 :~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
   mvn clean compile test-compile
   [INFO] Scanning for projects...
   [WARNING]
   [WARNING] Some problems were encountered while building the
   effective
   model for
 org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
   [WARNING] 

Re: Questions about flink-streaming-examples

[Márton Balassi]

Dear Mathias,

Thanks for reporting the issue. I have successfully built
flink-streaming-examples with maven, you can depend on test classes, the
following in the pom does the trick:

dependency
  groupIdorg.apache.flink/groupId
  artifactIdflink-streaming-core/artifactId
  version${project.version}/version
  scopetest/scope
  classifiertests/classifier
/dependency

This tells maven that the test classes of the examples (scope) might depend
on the tests of the core (classifier). Without explicitly stating the tests
classifier indeed maven only uses the classes under src/main.

I think the problem occurred because you only built
flink-streaming-examples and you might have used an outdated version of the
flink-streaming-core as a dependency installed to your local maven
repository - which did not include the StreamingTestBase yet. The change
that we are talking is about a week old, I wrote it.

Kind regards,

Marton

On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax 
mj...@informatik.hu-berlin.de wrote:

 Hi,

 I just build flink-streaming and avoid the problem. I guess, that the
 issue is related to the module structure and dependencies.
 flink-streaming-examples uses
 org.apache.flink.streaming.util.StreamingProgramTestBase (that is
 defined in flink-streaming-core/src/TEST

 From my understanding, maven does not support dependencies into src/test
 but only into src/main (maybe I am wrong about this).

 I would suggest, to move StreamingProgramTestBase into a new module
 flink-streaming-testUtils (into src/main) and change the dependency of
 flink-streaming-examples accordingly.

 Why the build work on the parent-project, is still a riddle to me... But
 it seems that the dependency into scr/test can be resolved for this case.


 -Matthias




 On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
  Hi,
  Were you able to solve this? It seams the examples depend on
 flink-streaming-core”. I think you have to build it before.
 
  On Feb 25, 2015, at 10:04 PM, Matthias J. Sax 
 mj...@informatik.hu-berlin.de wrote:
 
  Thanks!
 
  Even if I am not a building and/or maven expert, it seems to me that the
  dependencies are not configures correctly... No clue how to fix it. But
  it might be a good idea to have a look into it IMHO.
 
  -Matthias
 
 
  On 02/25/2015 05:07 PM, Stephan Ewen wrote:
  Hi Matthias!
 
  Can you try mvn clean package? It package is generally preferable to
  compile, in my opinion.
 
  There may be a dependency through a test jar that is not properly
 handled
  when you do not execute the package goal.
 
  Stephan
  Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
 
  Hi Matthias,
 
  Did you really pull from the latest master? I just tried to compile
  flink-streaming-examples using mvn clean compile test-compile and it
  worked.
 
  Best regards,
  Max
 
  On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
  mj...@informatik.hu-berlin.de wrote:
  Hi,
 
  I tried to build flink-streaming-examples, but got an compilation
 error.
  If I build flink-streaming (that implicitly builds
  flink-streaming-examples), it works.
 
  I tried it on a fresh clone of flink/master using mvn clean compile
  test-compile
 
  Can anybody explain this behavior?
 
 
  -Matthias
 
 
 
 
  mjsax@T420s-dbis-mjsax
 :~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
  mvn clean compile test-compile
  [INFO] Scanning for projects...
  [WARNING]
  [WARNING] Some problems were encountered while building the
 effective
  model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
  [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique
  but found duplicate declaration of plugin
  org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
  [WARNING]
  [WARNING] It is highly recommended to fix these problems because
 they
  threaten the stability of your build.
  [WARNING]
  [WARNING] For this reason, future Maven versions might no longer
  support building such malformed projects.
  [WARNING]
  [INFO]
  [INFO]
 
 
  [INFO] Building flink-streaming-examples 0.9-SNAPSHOT
  [INFO]
 
 
  [INFO]
  [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
  flink-streaming-examples ---
  [INFO] Deleting
 
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target
  [INFO]
  [INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
  flink-streaming-examples ---
  [INFO]
  [INFO]
  [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
  flink-streaming-examples ---
  [INFO]
  [INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @
  flink-streaming-examples ---
  [INFO] Source directory:
 
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala
  added.
  [INFO]
  [INFO] --- 

Re: Questions about flink-streaming-examples

[Stephan Ewen]

Hi Matthias!

Can you try mvn clean package? It package is generally preferable to
compile, in my opinion.

There may be a dependency through a test jar that is not properly handled
when you do not execute the package goal.

Stephan
Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:

 Hi Matthias,

 Did you really pull from the latest master? I just tried to compile
 flink-streaming-examples using mvn clean compile test-compile and it
 worked.

 Best regards,
 Max

 On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
 mj...@informatik.hu-berlin.de wrote:
  Hi,
 
  I tried to build flink-streaming-examples, but got an compilation error.
  If I build flink-streaming (that implicitly builds
  flink-streaming-examples), it works.
 
  I tried it on a fresh clone of flink/master using mvn clean compile
  test-compile
 
  Can anybody explain this behavior?
 
 
  -Matthias
 
 
 
 
  mjsax@T420s-dbis-mjsax:~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
 mvn clean compile test-compile
  [INFO] Scanning for projects...
  [WARNING]
  [WARNING] Some problems were encountered while building the effective
 model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
  [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique
 but found duplicate declaration of plugin
 org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
  [WARNING]
  [WARNING] It is highly recommended to fix these problems because they
 threaten the stability of your build.
  [WARNING]
  [WARNING] For this reason, future Maven versions might no longer
 support building such malformed projects.
  [WARNING]
  [INFO]
  [INFO]
 
  [INFO] Building flink-streaming-examples 0.9-SNAPSHOT
  [INFO]
 
  [INFO]
  [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
 flink-streaming-examples ---
  [INFO] Deleting
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target
  [INFO]
  [INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
 flink-streaming-examples ---
  [INFO]
  [INFO]
  [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
 flink-streaming-examples ---
  [INFO]
  [INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @
 flink-streaming-examples ---
  [INFO] Source directory:
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala
 added.
  [INFO]
  [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
 flink-streaming-examples ---
  [INFO]
  [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
 flink-streaming-examples ---
  [INFO] Using 'UTF-8' encoding to copy filtered resources.
  [INFO] skip non existing resourceDirectory
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/resources
  [INFO] Copying 3 resources
  [INFO]
  [INFO] --- scala-maven-plugin:3.1.4:compile (scala-compile-first) @
 flink-streaming-examples ---
  [INFO]
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/java:-1:
 info: compiling
  [INFO]
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala:-1:
 info: compiling
  [INFO] Compiling 14 source files to
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target/classes
 at 1424877113889
  [INFO] compiler plugin:
 BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
  [WARNING] warning: there were 1 deprecation warning(s); re-run with
 -deprecation for details
  [WARNING] one warning found
  [INFO] prepare-compile in 0 s
  [INFO] compile in 10 s
  [INFO]
  [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
 flink-streaming-examples ---
  [INFO] Changes detected - recompiling the module!
  [INFO] Compiling 10 source files to
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target/classes
  [INFO]
  [INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
 flink-streaming-examples ---
  [INFO]
  [INFO]
  [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
 flink-streaming-examples ---
  [INFO]
  [INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @
 flink-streaming-examples ---
  [INFO] Source directory:
 /home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala
 added.
  [INFO]
  [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
 flink-streaming-examples ---
  [INFO]
  [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
 flink-streaming-examples ---
  [INFO] Using 'UTF-8' encoding to copy filtered resources.
  [INFO] skip non existing resourceDirectory