[jira] [Created] (FLINK-6201) move python example files to the examples dir

[shijinkui (JIRA)]

shijinkui created FLINK-6201:


 Summary: move python example files to the examples dir
 Key: FLINK-6201
 URL: https://issues.apache.org/jira/browse/FLINK-6201
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Reporter: shijinkui


Python example in the resource dir is not suitable. Move them to the 
examples/python dir.
```


../flink-libraries/flink-python/src/main/python/org/apache/flink/python/api
resources/python
0755

```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6200) Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL

[sunjincheng (JIRA)]

sunjincheng created FLINK-6200:
--

 Summary: Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING 
aggregation to SQL
 Key: FLINK-6200
 URL: https://issues.apache.org/jira/browse/FLINK-6200
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: sunjincheng


The goal of this issue is to add support for OVER RANGE aggregations on event 
time streams to the SQL interface.
Queries similar to the following should be supported:
SELECT 
  a, 
  SUM(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED 
PRECEDING AND CURRENT ROW) AS sumB,
  MIN(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED 
PRECEDING AND CURRENT ROW) AS minB
FROM myStream
The following restrictions should initially apply:
All OVER clauses in the same SELECT clause must be exactly the same.
The PARTITION BY clause is optional (no partitioning results in single threaded 
execution).
The ORDER BY clause may only have rowTime() as parameter. rowTime() is a 
parameterless scalar function that just indicates processing time mode.
bounded PRECEDING is not supported (see FLINK-5655)
FOLLOWING is not supported.
The restrictions will be resolved in follow up issues. If we find that some of 
the restrictions are trivial to address, we can add the functionality in this 
issue as well.
This issue includes:
Design of the DataStream operator to compute OVER ROW aggregates
Translation from Calcite's RelNode representation (LogicalProject with RexOver 
expression).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6199) Single outstanding Async I/O operation per key

[Jamie Grier (JIRA)]

Jamie Grier created FLINK-6199:
--

 Summary: Single outstanding Async I/O operation per key
 Key: FLINK-6199
 URL: https://issues.apache.org/jira/browse/FLINK-6199
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Reporter: Jamie Grier


I would like to propose we extend the Async I/O semantics a bit such that a 
user can guarantee a single outstanding async request per key.

This would allow a user to order async requests per key while still achieving 
the throughput benefits of using Async I/O in the first place.

This is essential for operations where stream order is important but we still 
need to use Async operations to interact with an external system in a 
performant way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Java library for Flink-Kudu integration

[Fabian Hueske]

Hi Ruben,

thanks for sharing this!
A Flink Kudu connector is a great contribution and Bahir seems to be the
right place for it.

Thanks, Fabian


2017-03-27 15:35 GMT+02:00 :

> Hi all,
>
> I apologize for sending the email to both accounts, but not sure where
> this topic fits better.
>
> In my team, we have been working in some PoCs and PoVs about new data
> architectures. As part of this work, we have implemented a library to
> connect Kudu and Flink. The library allows reading/writing from/to Kudu
> tablets using DataSet API and also writing to Kudu using DataStream API.
>
> You can find the code and documentation (including some examples) in
> https://github.com/rubencasado/Flink-Kudu
>
> Any comment/suggestion/contribution is very welcomed ☺
>
> We will try to publish this contribution to the Apache Bahir project.
>
> Best
>
> 
> Rubén Casado Tejedor, PhD
> > accenture digital
> Big Data Manager
> ' + 34 629 009 429
> • ruben.casado.teje...@accenture.com casado.teje...@accenture.com>
>
> 
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> 
> __
>
> www.accenture.com
>

Re: FW: [DISCUSS] Table API / SQL indicators for event and processing time

[Fabian Hueske]

Hi Timo,

I had a look at your branch. Thanks for all the refactoring and work you
put into this.

I like the proposal a lot. I think indicating the time attribute in the
schema is a good idea. I'm not sure if we should support the rowtime
expression for batch table though. For batch, any long or timestamp
attribute can be used as a time attribute. So marking a single one as time
attribute is not really necessary, IMO. If others think that this is needed
to make the batch and stream cases identical I would be OK with having it.
OTOH I would not consider the schema to be part of the query.

I think we should continue with this work as it is likely to take more time
until we can merge it into the master.
@Timo: would the next step be to open a PR for this?

Best, Fabian


2017-03-20 14:15 GMT+01:00 Radu Tudoran :

> Hi,
>
> I am not sure if it is not about setting the timestamp within the query
> but you can imagine that there examples where you have different timestamps
> as mentioned. Take for example the case when we do a purchase online. You
> have:
> -time of purchase  (when the payment was input/triggered)
> -time of executing the transaction at bank (when the payment is processed
> from account)
> -time of settlement (when the payment is executed at merchant bank - when
> money are received by the seller)
>
> In such a scenario you can imagine that over the same stream of online
> payments you might want to run different queries, each that might be driven
> by one of these times. Supporting such a scenario would mean that we have
> one input stream that enters flink engine via a table source and then in
> the query we can start running different queries:
> e.g. SELECT SUM(amount) ORDER BY rowtime(time_purchase) LIMIT 10   //you
> want the amount over your last 10 orders
> e.g. SELECT SUM(amount) ORDER BY rowtime(time_settlement) LIMIT 10 //you
> want the amount over your last 10 income
>
> Best regards,
>
>
> -Original Message-
> From: Timo Walther [mailto:twal...@apache.org]
> Sent: Monday, March 20, 2017 2:05 PM
> To: dev@flink.apache.org
> Subject: Re: FW: [DISCUSS] Table API / SQL indicators for event and
> processing time
>
> Yes, you are right. In the current design the user cannot assign
> timestamp and watermarks in a table program. Operators (such as windows)
> might adapt the metatimestamp, if this is the case this adaption might
> need to be expressed in the query itself too.
>
> E.g. for a tumbling windows we could limit the select part to
> table.select('rowtime.ceil(DAY) as 'newRowtime) (so that logical rowtime
> matches the physical metatimestamp)
>
> Do you have a good example use case that needs the assignment of rowtime
> within a query?
>
> Am 20/03/17 um 13:39 schrieb Radu Tudoran:
> > Hi,
> >
> > As suggested by Timo - I am forwarding this to the mailing list. Sorry
> for not having the conversation directly here - I initially thought it
> might not be of interest...
> >
> > @Timo - thanks for the clarification. I get the main point now which is
> that the rowtime is encoded within the  metadata of the record. I think
> this is key. My view on the matter was maybe a bit updated in the sense
> that I saw the processing pipeline as an input source (as you exemplify - a
> table scan) and from there you have a timestamp and water mark assigner
> before the processing actually starts. So by overriding the timestamp
> extractor you match the field that carries the eventtime/rowtime with the
> mechanism from flink. But as far as I understand this would not be the case
> anymore...am I right? In case the assignment of the rowtime to the metadata
> of the record is done differently - what would be the way to do it?
> >
> >
> > Dr. Radu Tudoran
> > Senior Research Engineer - Big Data Expert
> > IT R Division
> >
> >
> > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> > European Research Center
> > Riesstrasse 25, 80992 München
> >
> > E-mail: radu.tudo...@huawei.com
> > Mobile: +49 15209084330
> > Telephone: +49 891588344173
> >
> > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> > Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
> > Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
> > Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
> > Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
> > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
> > This e-mail and its attachments contain confidential information from
> HUAWEI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
> >
> >
> > -Original Message-
> > From: Timo Walther [mailto:twal...@apache.org]
> > Sent: 

[jira] [Created] (FLINK-6198) Update the documentation of the CEP library to include all the new features.

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-6198:
-

 Summary: Update the documentation of the CEP library to include 
all the new features.
 Key: FLINK-6198
 URL: https://issues.apache.org/jira/browse/FLINK-6198
 Project: Flink
  Issue Type: Bug
  Components: CEP
Affects Versions: 1.3.0
Reporter: Kostas Kloudas
 Fix For: 1.3.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6197) Add support for iterative conditions.

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-6197:
-

 Summary: Add support for iterative conditions.
 Key: FLINK-6197
 URL: https://issues.apache.org/jira/browse/FLINK-6197
 Project: Flink
  Issue Type: Bug
  Components: CEP
Affects Versions: 1.3.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.3.0


So far, the {{where}} clause only supported simple {{FilterFunction}} 
conditions. 

With this, we want to add support for conditions where the an event is accepted 
not only based on its own properties, e.g. name, as it was before, but also 
based on some statistic computed over previously accepted events in the 
pattern, e.g. if the price is higher than the average of the prices of the 
previously accepted events. 

This in combination with the recently added quantifiers will allow for a lot 
more expressive patterns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[CANCEL][VOTE] Release Apache Flink 1.2.1 (RC1)

[Robert Metzger]

I'll cancel the vote.

Once the fixes are in, I'll create a new RC.

On Mon, Mar 27, 2017 at 11:59 AM, Chesnay Schepler 
wrote:

> If possible I would like to include FLINK-6183 & FLINK-6184 as well.
>
> They fix 2 metric-related issues that could arise when a Task is cancelled
> very early. (like, right away)
>
> FLINK-6183 fixes a memory leak where the TaskMetricGroup was never closed
> FLINK-6184 fixes a NullPointerExceptions in the buffer metrics
>
> PR here: https://github.com/apache/flink/pull/3611
>
>
> On 26.03.2017 12:35, Aljoscha Krettek wrote:
>
>> I opened a PR for FLINK-6188: https://github.com/apache/flink/pull/3616 <
>> https://github.com/apache/flink/pull/3616>
>>
>> This improves the previously very sparse test coverage for
>> timestamp/watermark assigners and fixes the bug.
>>
>> On 25 Mar 2017, at 10:22, Ufuk Celebi  wrote:
>>>
>>> I agree with Aljoscha.
>>>
>>> -1 because of FLINK-6188
>>>
>>>
>>> On Sat, Mar 25, 2017 at 9:38 AM, Aljoscha Krettek 
>>> wrote:
>>>
 I filed this issue, which was observed by a user:
 https://issues.apache.org/jira/browse/FLINK-6188

 I think that’s blocking for 1.2.1.

 On 24 Mar 2017, at 18:57, Ufuk Celebi  wrote:
>
> RC1 doesn't contain Stefan's backport for the Asynchronous snapshots
> for heap-based keyed state that has been merged. Should we create RC2
> with that fix since the voting period only starts on Monday? I think
> it would only mean rerunning the scripts on your side, right?
>
> – Ufuk
>
>
> On Fri, Mar 24, 2017 at 3:05 PM, Robert Metzger 
> wrote:
>
>> Dear Flink community,
>>
>> Please vote on releasing the following candidate as Apache Flink
>> version 1.2
>> .1.
>>
>> The commit to be voted on:
>> *732e55bd* (*http://git-wip-us.apache.org
>> /repos/asf/flink/commit/732e55bd
>> *)
>>
>> Branch:
>> release-1.2.1-rc1
>>
>> The release artifacts to be voted on can be found at:
>> *http://people.apache.org/~rmetzger/flink-1.2.1-rc1/
>> *
>>
>> The release artifacts are signed with the key with fingerprint
>> D9839159:
>> http://www.apache.org/dist/flink/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapache
>> flink-1116
>>
>> -
>>
>>
>> The vote ends on Wednesday, March 29, 2017, 3pm CET.
>>
>>
>> [ ] +1 Release this package as Apache Flink 1.2.1
>> [ ] -1 Do not release this package, because ...
>>
>
>>
>

[jira] [Created] (FLINK-6196) Support dynamic schema in Table Function

[Zhuoluo Yang (JIRA)]

Zhuoluo Yang created FLINK-6196:
---

 Summary: Support dynamic schema in Table Function
 Key: FLINK-6196
 URL: https://issues.apache.org/jira/browse/FLINK-6196
 Project: Flink
  Issue Type: Improvement
Reporter: Zhuoluo Yang
Assignee: Zhuoluo Yang


In many of our use cases. We have to decide the schema of a UDTF at the run 
time. For example. udtf('c1, c2, c3') will generate three columns for a lateral 
view. 
Most systems such as calcite and hive support this feature. However, the 
current implementation of flink didn't implement the feature correctly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Question about flink client

[Till Rohrmann]

Hi Yelei,

I think you're right that we should not only rely on the response of the
JobManager but also ask the Yarn client what the status of the application
is. If the application status is something different than Succeeded, then
it should be save to shut down the cluster. If this is not the case, then
it would be great if you could file an issue to fix it.

Cheers,
Till

On Mon, Mar 27, 2017 at 9:21 AM, Yelei Feng  wrote:

> Hi Till,
>
> Thanks for your help, I merged your code into my branch and the issue is
> gone.
>
> BTW, do you have some insights for the first problem?
>
> "Once the yarn session CLI can't get cluster status(e.g. jobmanager is
> down), it will try to shutdown the cluster and cleanup related files even
> if new jobmanager will be created soon. As result,  yarn will fail to start
> a new jobmanager due to missing files on HDFS. As a workround, I can config
> `akka.lookup.timeout` to wait a bit longer,  say 60 seconds. But I'm
> wondering if it will affect other components."
>
> Thanks,
> Yelei
> 
> From: Till Rohrmann 
> Sent: Friday, March 24, 2017 8:09:29 AM
> To: dev@flink.apache.org
> Subject: Re: Question about flink client
>
> Hi Yelei,
>
> thanks for investigating the problem and pointing out the problematic
> parts. In fact, I recently stumbled across the very same problem in the
> JobClientActor and wrote a fix for it. It is already merged into the
> master. I hope that this fix solved the problem you've described.
>
> Cheers,
> Till
>
> On Wed, Mar 22, 2017 at 3:42 PM, Yelei Feng  wrote:
>
> > Hi,
> >
> >
> > I have two questions about flink client in interactive mode.
> >
> >
> > One is for yarn-session.sh,  once the session CLI can't get cluster
> stauts
> > (jobmanager is down), it will try to shutdown the cluster and cleanup
> > related files even if new jobmanager will be created soon. As result,
> yarn
> > will fail to start a new jobmanager due to missing files on HDFS. As a
> > workround, I can config `akka.lookup.timeout` to wait a bit longer,  say
> 60
> > seconds. But I'm wondering if it will affect other components.
> >
> >
> > Second is about flink cli. If cluster is down after submiting job using
> > 'flink run xx.jar',  cli hangs there only showing "New JobManager
> elected.
> > Connecting to null " instead of cleanup and close itself.
> >
> >
> > After some digging, I found the main logic is in JobClientActor. It
> > receives jobmanager status changes from two sources: zookeeper and akka
> > deathwatch. It would terminate itself once receiving message
> > `ConnectionTimeout`.
> > Client sets current leaderSessionId and unwatch previous jobmanager from
> > ZK; it receives `Teminated` of previous jobmanager from akka deathwatch
> and
> > send `ConnectionTimeout` to itself after 60s. In a great chance, they
> would
> > interfere with each other.
> >
> > Situation1:
> >
> >   1.  client get notified from zk, set leaderSessionId to null
> >   2.  client unwatch previous jobmanager
> >   3.  msg `Teminated` of previous jobmanager never got received
> >
> > Situation 2:
> >
> >   1.  msg `Teminated` of current jobmanager is received
> >   2.  schedule msg ConnectionTimeout after 60s
> >   3.  client get notified from zk, set `leaderSessionId` to null in less
> > than 60s
> >   4.  `ConnectionTimeout` will be filtered out due to different
> > `leaderSessionId`
> >
> >
> > Both of the two problems only happen in interactive mode,  not in
> detached
> > mode.  I wonder if it's issues for interactive mode, or we should only
> use
> > detached mode in production environment?
> >
> >
> > Any insight is appreciated.
> >
> >
> > Thanks,
> >
> > Yelei
> >
> >
>

[jira] [Created] (FLINK-6195) Move gelly-examples jar from opt to examples

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-6195:
-

 Summary: Move gelly-examples jar from opt to examples
 Key: FLINK-6195
 URL: https://issues.apache.org/jira/browse/FLINK-6195
 Project: Flink
  Issue Type: Sub-task
  Components: Gelly
Affects Versions: 1.3.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Trivial
 Fix For: 1.3.0


The {{opt}} directory should be reserved for Flink JARs which users may 
optionally move to {{lib}} to be loaded by the runtime. 
{{flink-gelly-examples}} is a user program so is being moved to the 
{{examples}} folder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Java library for Flink-Kudu integration

[ruben.casado.tejedor]

Hi all,

I apologize for sending the email to both accounts, but not sure where this 
topic fits better.

In my team, we have been working in some PoCs and PoVs about new data 
architectures. As part of this work, we have implemented a library to connect 
Kudu and Flink. The library allows reading/writing from/to Kudu tablets using 
DataSet API and also writing to Kudu using DataStream API.

You can find the code and documentation (including some examples) in 
https://github.com/rubencasado/Flink-Kudu

Any comment/suggestion/contribution is very welcomed ☺

We will try to publish this contribution to the Apache Bahir project.

Best


Rubén Casado Tejedor, PhD
> accenture digital
Big Data Manager
' + 34 629 009 429
• ruben.casado.teje...@accenture.com



This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited. Where allowed by local law, electronic 
communications with Accenture and its affiliates, including e-mail and instant 
messaging (including content), may be scanned by our systems for the purposes 
of information security and assessment of internal compliance with Accenture 
policy.
__

www.accenture.com

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

[Chesnay Schepler]

If possible I would like to include FLINK-6183 & FLINK-6184 as well.

They fix 2 metric-related issues that could arise when a Task is 
cancelled very early. (like, right away)


FLINK-6183 fixes a memory leak where the TaskMetricGroup was never closed
FLINK-6184 fixes a NullPointerExceptions in the buffer metrics

PR here: https://github.com/apache/flink/pull/3611

On 26.03.2017 12:35, Aljoscha Krettek wrote:

I opened a PR for FLINK-6188: https://github.com/apache/flink/pull/3616 


This improves the previously very sparse test coverage for timestamp/watermark 
assigners and fixes the bug.


On 25 Mar 2017, at 10:22, Ufuk Celebi  wrote:

I agree with Aljoscha.

-1 because of FLINK-6188


On Sat, Mar 25, 2017 at 9:38 AM, Aljoscha Krettek  wrote:

I filed this issue, which was observed by a user: 
https://issues.apache.org/jira/browse/FLINK-6188

I think that’s blocking for 1.2.1.


On 24 Mar 2017, at 18:57, Ufuk Celebi  wrote:

RC1 doesn't contain Stefan's backport for the Asynchronous snapshots
for heap-based keyed state that has been merged. Should we create RC2
with that fix since the voting period only starts on Monday? I think
it would only mean rerunning the scripts on your side, right?

– Ufuk


On Fri, Mar 24, 2017 at 3:05 PM, Robert Metzger  wrote:

Dear Flink community,

Please vote on releasing the following candidate as Apache Flink version 1.2
.1.

The commit to be voted on:
*732e55bd* (*http://git-wip-us.apache.org/repos/asf/flink/commit/732e55bd
*)

Branch:
release-1.2.1-rc1

The release artifacts to be voted on can be found at:
*http://people.apache.org/~rmetzger/flink-1.2.1-rc1/
*

The release artifacts are signed with the key with fingerprint D9839159:
http://www.apache.org/dist/flink/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapacheflink-1116

-


The vote ends on Wednesday, March 29, 2017, 3pm CET.


[ ] +1 Release this package as Apache Flink 1.2.1
[ ] -1 Do not release this package, because ...

[jira] [Created] (FLINK-6194) More broken links in docs

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-6194:


 Summary: More broken links in docs
 Key: FLINK-6194
 URL: https://issues.apache.org/jira/browse/FLINK-6194
 Project: Flink
  Issue Type: Bug
Reporter: Patrick Lucas
Assignee: Patrick Lucas


My script noticed a few broken links that made it into the docs. I'll fix them 
up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6193) Flink dist directory normalize

[shijinkui (JIRA)]

shijinkui created FLINK-6193:


 Summary: Flink dist directory normalize
 Key: FLINK-6193
 URL: https://issues.apache.org/jira/browse/FLINK-6193
 Project: Flink
  Issue Type: Improvement
  Components: Examples
Reporter: shijinkui


The Flink distribution's directory have no very clear responsibility about what 
type of files should be in which directory. For example, "opt" directories are 
mixed with library jars and example jars.

This mail here: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-dist-directory-management-td16784.html

After discuss, we determine for the distribution directory style below:
- "examples" directory only contain example jars
- "opt" directory only contain optional library jars in runtime
- "lib" directory only contain library jar that must be loaded at runtime
- "resources" directory only contain resource file used at runtime, such as web 
file



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Flink dist directory management

[shijinkui]

Hi, Fabian, Jark Wu, Greg
Thank for your feedback and support. I’ll create some sub-issue to
complete the every task of code refactor.

在 2017/3/27 下午3:38， "Fabian Hueske"  写入:

>Hi Jinkui,
>
>I agree with you, moving all examples into ./examples would be nice.
>Note, that for the DataSet and DataStream APIs, each example is packaged
>as
>a separate JAR with the main class being included in the MANIFEST file.
>
>I think we should select a few representative examples of the Table and
>Gelly examples and do the same.
>IMO, the Table API / SQL examples could also be improved a lot, but that's
>a different story.
>
>Best, Fabian
>
>2017-03-27 5:47 GMT+02:00 伍翀(云邪) :
>
>> Hi Jinkui,
>>
>> Thanks for bringing up this discussion. I’m +1 to moving table-examples
>> and belly-examples into "examples/“ to keep “opt” clean.
>>
>> Best,
>> Jark Wu
>>
>> > 在 2017年3月25日，下午3:46，shijinkui  写道：
>> >
>> > Hi, all
>> >
>> > The Flink distributionĄ¯s directory have no very clear responsibility
>> about what type of files should be in which directory.
>> >
>> > The "bin","conf","lib" directories are clear for their responsibility.
>> >
>> > But the Ą°opt" directories are mixed with library jars and example
>>jars.
>> >
>> > I think we can discuss how is reasonable for the directory. Once we
>> determined, we should follow it.
>> >
>> > IMO, directory style below is reasonable:
>> >
>> > - "examples" directory only contain example jars
>> > - "opt" directory only contain optional library jars in runtime
>> > - "lib" directory only contain library jar that must be loaded at
>>runtime
>> > - Ą°resourcesĄą directory only contain resource file used at runtime,
>> such as web file
>> >
>> > Show your opinion please.
>> >
>> > @wuchong, @fhueske @Fabian
>> >
>> > Best regards,
>> > Jinkui Shi
>> >
>> >
>> > .
>> > ŠĀŠ¤Š¤ LICENSE
>> > ŠĀŠ¤Š¤ NOTICE
>> > ŠĀŠ¤Š¤ README.txt
>> > ŠĀŠ¤Š¤ bin
>> > ŠĻ   ŠĀŠ¤Š¤ config.sh
>> > ŠĻ   ŠĀŠ¤Š¤ flink
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ conf
>> > ŠĻ   ŠĀŠ¤Š¤ flink-conf.yaml
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ examples
>> > ŠĻ   ŠĀŠ¤Š¤ batch
>> > ŠĻ   Š¸Š¤Š¤ streaming
>> > ŠĀŠ¤Š¤ lib
>> > ŠĻ   ŠĀŠ¤Š¤ flink-dist_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-python_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ log
>> > ŠĀŠ¤Š¤ opt
>> > ŠĻ   ŠĀŠ¤Š¤ flink-cep-scala_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-cep_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-examples_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-scala_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-dropwizard-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-ganglia-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-graphite-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-statsd-1.3.0.jar
>> > ŠĻ   Š¸Š¤Š¤ flink-ml_2.11-1.3.0.jar
>> > ŠĀŠ¤Š¤ resources
>> > ŠĻ   Š¸Š¤Š¤ python
>> > Š¸Š¤Š¤ tools
>> >Š¸Š¤Š¤ planVisualizer.html
>> >
>> >
>> > [1] https://github.com/apache/flink/pull/2460
>> >
>>
>>

Re: [DISCUSS] Flink dist directory management

[Fabian Hueske]

Hi Jinkui,

I agree with you, moving all examples into ./examples would be nice.
Note, that for the DataSet and DataStream APIs, each example is packaged as
a separate JAR with the main class being included in the MANIFEST file.

I think we should select a few representative examples of the Table and
Gelly examples and do the same.
IMO, the Table API / SQL examples could also be improved a lot, but that's
a different story.

Best, Fabian

2017-03-27 5:47 GMT+02:00 伍翀(云邪) :

> Hi Jinkui,
>
> Thanks for bringing up this discussion. I’m +1 to moving table-examples
> and belly-examples into "examples/“ to keep “opt” clean.
>
> Best,
> Jark Wu
>
> > 在 2017年3月25日，下午3:46，shijinkui  写道：
> >
> > Hi, all
> >
> > The Flink distributionĄ¯s directory have no very clear responsibility
> about what type of files should be in which directory.
> >
> > The "bin","conf","lib" directories are clear for their responsibility.
> >
> > But the Ą°opt" directories are mixed with library jars and example jars.
> >
> > I think we can discuss how is reasonable for the directory. Once we
> determined, we should follow it.
> >
> > IMO, directory style below is reasonable:
> >
> > - "examples" directory only contain example jars
> > - "opt" directory only contain optional library jars in runtime
> > - "lib" directory only contain library jar that must be loaded at runtime
> > - Ą°resourcesĄą directory only contain resource file used at runtime,
> such as web file
> >
> > Show your opinion please.
> >
> > @wuchong, @fhueske @Fabian
> >
> > Best regards,
> > Jinkui Shi
> >
> >
> > .
> > ŠĀŠ¤Š¤ LICENSE
> > ŠĀŠ¤Š¤ NOTICE
> > ŠĀŠ¤Š¤ README.txt
> > ŠĀŠ¤Š¤ bin
> > ŠĻ   ŠĀŠ¤Š¤ config.sh
> > ŠĻ   ŠĀŠ¤Š¤ flink
> > ŠĻ   ŠĀŠ¤Š¤ ...
> > ŠĀŠ¤Š¤ conf
> > ŠĻ   ŠĀŠ¤Š¤ flink-conf.yaml
> > ŠĻ   ŠĀŠ¤Š¤ ...
> > ŠĀŠ¤Š¤ examples
> > ŠĻ   ŠĀŠ¤Š¤ batch
> > ŠĻ   Š¸Š¤Š¤ streaming
> > ŠĀŠ¤Š¤ lib
> > ŠĻ   ŠĀŠ¤Š¤ flink-dist_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-python_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ ...
> > ŠĀŠ¤Š¤ log
> > ŠĀŠ¤Š¤ opt
> > ŠĻ   ŠĀŠ¤Š¤ flink-cep-scala_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-cep_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-examples_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-scala_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly_2.11-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-dropwizard-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-ganglia-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-graphite-1.3.0.jar
> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-statsd-1.3.0.jar
> > ŠĻ   Š¸Š¤Š¤ flink-ml_2.11-1.3.0.jar
> > ŠĀŠ¤Š¤ resources
> > ŠĻ   Š¸Š¤Š¤ python
> > Š¸Š¤Š¤ tools
> >Š¸Š¤Š¤ planVisualizer.html
> >
> >
> > [1] https://github.com/apache/flink/pull/2460
> >
>
>

[jira] [Created] (FLINK-6192) reuse zookeeer client created by CuratorFramework

[Tao Wang (JIRA)]

Tao Wang created FLINK-6192:
---

 Summary: reuse zookeeer client created by CuratorFramework
 Key: FLINK-6192
 URL: https://issues.apache.org/jira/browse/FLINK-6192
 Project: Flink
  Issue Type: Improvement
  Components: JobManager, YARN
Reporter: Tao Wang
Assignee: Tao Wang


Now in yarn mode, there're three places using zookeeper client(web monitor, 
jobmanager and resourcemanager) in ApplicationMaster/JobManager, while there're 
two in TaskManager. They create new one zookeeper client when they need them.

I believe there're more other places do the same thing, but in one JVM, one 
CuratorFramework is enough for connections to one zookeeper client, so we need 
a singleton to reuse them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Question about flink client

[Yelei Feng]

Hi Till,

Thanks for your help, I merged your code into my branch and the issue is gone.

BTW, do you have some insights for the first problem?

"Once the yarn session CLI can't get cluster status(e.g. jobmanager is down), 
it will try to shutdown the cluster and cleanup related files even if new 
jobmanager will be created soon. As result,  yarn will fail to start a new 
jobmanager due to missing files on HDFS. As a workround, I can config 
`akka.lookup.timeout` to wait a bit longer,  say 60 seconds. But I'm wondering 
if it will affect other components."

Thanks,
Yelei

From: Till Rohrmann 
Sent: Friday, March 24, 2017 8:09:29 AM
To: dev@flink.apache.org
Subject: Re: Question about flink client

Hi Yelei,

thanks for investigating the problem and pointing out the problematic
parts. In fact, I recently stumbled across the very same problem in the
JobClientActor and wrote a fix for it. It is already merged into the
master. I hope that this fix solved the problem you've described.

Cheers,
Till

On Wed, Mar 22, 2017 at 3:42 PM, Yelei Feng  wrote:

> Hi,
>
>
> I have two questions about flink client in interactive mode.
>
>
> One is for yarn-session.sh,  once the session CLI can't get cluster stauts
> (jobmanager is down), it will try to shutdown the cluster and cleanup
> related files even if new jobmanager will be created soon. As result,  yarn
> will fail to start a new jobmanager due to missing files on HDFS. As a
> workround, I can config `akka.lookup.timeout` to wait a bit longer,  say 60
> seconds. But I'm wondering if it will affect other components.
>
>
> Second is about flink cli. If cluster is down after submiting job using
> 'flink run xx.jar',  cli hangs there only showing "New JobManager elected.
> Connecting to null " instead of cleanup and close itself.
>
>
> After some digging, I found the main logic is in JobClientActor. It
> receives jobmanager status changes from two sources: zookeeper and akka
> deathwatch. It would terminate itself once receiving message
> `ConnectionTimeout`.
> Client sets current leaderSessionId and unwatch previous jobmanager from
> ZK; it receives `Teminated` of previous jobmanager from akka deathwatch and
> send `ConnectionTimeout` to itself after 60s. In a great chance, they would
> interfere with each other.
>
> Situation1:
>
>   1.  client get notified from zk, set leaderSessionId to null
>   2.  client unwatch previous jobmanager
>   3.  msg `Teminated` of previous jobmanager never got received
>
> Situation 2:
>
>   1.  msg `Teminated` of current jobmanager is received
>   2.  schedule msg ConnectionTimeout after 60s
>   3.  client get notified from zk, set `leaderSessionId` to null in less
> than 60s
>   4.  `ConnectionTimeout` will be filtered out due to different
> `leaderSessionId`
>
>
> Both of the two problems only happen in interactive mode,  not in detached
> mode.  I wonder if it's issues for interactive mode, or we should only use
> detached mode in production environment?
>
>
> Any insight is appreciated.
>
>
> Thanks,
>
> Yelei
>
>

[jira] [Created] (FLINK-6191) Make non-primitive, internal built-in serializers reconfigurable

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-6191:
--

 Summary: Make non-primitive, internal built-in serializers 
reconfigurable
 Key: FLINK-6191
 URL: https://issues.apache.org/jira/browse/FLINK-6191
 Project: Flink
  Issue Type: Sub-task
  Components: Type Serialization System
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


This sub-task follows after FLINK-6190.

For non-primitive type serializers internally created by Flink, we need to 
allow them to be reconfigurable whenever we detect a change between the 
previous and current serializer configuration.

Most notably, this is relevant for the `KryoSerializer` and `PojoSerializer` 
which are affected by the configuration, as well as composite types which can 
potentially have nested serializers (e.g. `GenericArraySerializer`).

Since not all serializers require / reconfiguration, we propose to have a 
extended abstract base class for these:

{code}
@Internal
public abstract class ReconfigurableTypeSerializer extends TypeSerializer 
{

void abstract reconfigure(SerializersConfig serializersConfig);
}
{code}

This class is also used as a tag, to check if a serializer needs to be 
reconfigured when serializer configuration change is detected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)