Make SubmittedJobGraphStore configurable

2017-07-05 Thread Chen Qin 
​Hi there,

​I would like to propose/discuss median level refactor work to make
submittedJobGraphStore configurable and extensible.

The rationale behind is to allow users offload those meta data to durable
cross dc read after write strong consistency storage and decouple with zk
quorum.
​

https://issues.apache.org/jira/browse/FLINK-7106


New configurable setting in flink.conf
​ looks like following

g​
raph
​-s
tore:
​customized/zookeeper
g​
raph
​-s
tore​.class: xx.yy.MyS3SubmittedJobGraphStore​Imp

g​
raph
​-s
tore.
​endpoint
: s3.amazonaws.com
g​
raph
​-s
tore.path.root:
​s3:/
​
/
​my root/​

Thanks,
Chen

Re: [DISCUSS] Release 1.3.2 planning

2017-07-05 Thread Tzu-Li (Gordon) Tai 
FLINK-7041 has been merged.
I’d also like to raise another blocker for 1.3.2: 
https://issues.apache.org/jira/browse/FLINK-6996.

Cheers,
Gordon
On 30 June 2017 at 12:46:07 AM, Aljoscha Krettek (aljos...@apache.org) wrote:

Gordon and I found this (in my opinion) blocking issue: 
https://issues.apache.org/jira/browse/FLINK-7041 
  

I’m trying to quickly provide a fix.  

> On 26. Jun 2017, at 15:30, Timo Walther  wrote:  
>  
> I just opened a PR which should be included in the next bug fix release for 
> the Table API:  
> https://issues.apache.org/jira/browse/FLINK-7005  
>  
> Timo  
>  
> Am 23.06.17 um 14:09 schrieb Robert Metzger:  
>> Thanks Haohui.  
>>  
>> The first main task for the release management is to come up with a  
>> timeline :)  
>> Lets just wait and see which issues get reported. There are currently no  
>> blockers set for 1.3.1 in JIRA.  
>>  
>> On Thu, Jun 22, 2017 at 6:47 PM, Haohui Mai  wrote:  
>>  
>>> Hi,  
>>>  
>>> Release management is though, I'm happy to help. Are there any timelines  
>>> you have in mind?  
>>>  
>>> Haohui  
>>> On Fri, Jun 23, 2017 at 12:01 AM Robert Metzger   
>>> wrote:  
>>>  
 Hi all,  
  
 with the 1.3.1 release on the way, we can start thinking about the 1.3.2  
 release.  
  
 We have already one issue that should go in there:  
 - https://issues.apache.org/jira/browse/FLINK-6964  
  
 If there are any other blockers, let us know here :)  
  
 I'm wondering if there's somebody from the community who's willing to  
>>> take  
 care of the release management of 1.3.2 :)  
  
>

Re: [DISCUSS] FLIP proposal for Model Serving over Flink

2017-07-05 Thread Theodore Vasiloudis 
Hello all,

I just wanted to indicate that I would be very willing to help out with
code reviews
for this project and participating in design discussions.

But I should note that I don' think I'll have time to contribute code until
I get back to Stockholm in September.

Regards,
Theodore

On Tue, Jul 4, 2017 at 9:41 AM, Andrea Spina 
wrote:

> Hi all,
> yes, we did too. We - from Radicalbit - have submitted a talk focused
> on the recently released flink-jpmml library about model serving.
> Lately, it became part of the FlinkML project.
>
> Cheers, Andrea
>
> 2017-07-04 16:14 GMT+02:00 Boris Lublinsky  >:
> > Yes,
> > I submitted a talk with Stavros on model serving
> >
> >
> > Boris Lublinsky
> > FDP Architect
> > boris.lublin...@lightbend.com
> > https://www.lightbend.com/
> >
> > On Jul 3, 2017, at 1:18 PM, Robert Metzger  wrote:
> >
> > Big +1 from my side on getting this effort started.
> >
> > Users have asked for this and I would like to see some progress there.
> > Did anybody submit a talk about the ML efforts to Flink Forward Berlin
> this
> > year?
> >
> > On Fri, Jun 30, 2017 at 6:04 PM, Fabian Hueske 
> wrote:
> >>
> >> Yes, I know that Theo is engaged in the ML efforts but wasn't sure how
> >> much
> >> he is involved in the model serving part (thought he was more into the
> >> online learning part).
> >> It would be great if Theo could help here!
> >>
> >> I just wanted to make sure that we find somebody to help bootstrapping.
> >>
> >> Cheers, Fabian
> >>
> >>
> >> 2017-06-30 17:52 GMT+02:00 Stavros Kontopoulos <
> st.kontopou...@gmail.com>:
> >>
> >> > Hi Fabian,
> >> >
> >> > However, we should keep in mind that we need a committer to bootstrap
> >> > the
> >> > > new module.
> >> >
> >> >
> >> > Absolutely I thought Theodore Vassiloudis could help, as an initial
> >> > committer.
> >> > Is this known? He is part of the effort btw.
> >> >
> >> > Best,
> >> > Stavros
> >> >
> >> > On Fri, Jun 30, 2017 at 6:42 PM, Fabian Hueske 
> >> > wrote:
> >> >
> >> > > Thanks Stavros (and everybody else involved) for starting this
> effort
> >> > > and
> >> > > bringing the discussion back to the mailing list.
> >> > >
> >> > > As I said before, a model serving module/component would be a great
> >> > feature
> >> > > for Flink.
> >> > > I see the biggest advantage for such a module in the integration
> with
> >> > > the
> >> > > other APIs and libraries, such as DataStream, CEP, SQL.
> >> > >
> >> > > A FLIP would be a great way to continue your efforts and work on a
> >> > > design
> >> > > for the component.
> >> > >
> >> > > However, we should keep in mind that we need a committer to
> bootstrap
> >> > > the
> >> > > new module.
> >> > > As people are contributing to the model serving module, the number
> of
> >> > > committers should hopefully grow after some time.
> >> > >
> >> > > Best, Fabian
> >> > >
> >> > > 2017-06-30 10:58 GMT+02:00 Stavros Kontopoulos
> >> > >  >> > >:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > After coordinating with Theodore Vasiloudis and the guys behind
> the
> >> > Flink
> >> > > > Model Serving effort (Eron, Radicalbit people, Boris, Bas (ING)),
> we
> >> > > > propose to start working on the model serving over Flink in a more
> >> > > official
> >> > > > way.
> >> > > > That translates to capturing design details in a FLIP document.
> >> > > >
> >> > > > Please let's discuss and vote whether you think this FLIP would be
> >> > > viable.
> >> > > >
> >> > > > Model Serving as a Flink component might involve a lot of work and
> >> > > > we
> >> > > need
> >> > > > to commit to support it in future Flink releases.
> >> > > >
> >> > > > In the mean time a lot of people have joined Flink ml slack
> channel
> >> > > > (
> >> > > > https://flinkml.slack.com, https://flinkml-invites.herokuapp.com/
> )
> >> > and I
> >> > > > think its time to try get them gradually on board.
> >> > > >
> >> > > > So far we have several efforts hosted here:
> >> > > > https://github.com/FlinkML
> >> > > >
> >> > > > Related documents for what we are doing:
> >> > > >
> >> > > > Flink ML roadmap
> >> > > >
> >> > > > https://docs.google.com/document/d/1afQbvZBTV15qF3vobVWUjxQc49h3U
> >> > > > d06MIRhahtJ6dw/edit
> >> > > >
> >> > > > Flink MS
> >> > > >
> >> > > > https://docs.google.com/document/d/1CjWL9aLxPrKytKxUF5c3ohs0ickp0
> >> > > > fdEXPsPYPEywsE/edit#
> >> > > >
> >> > > > PS. I will work on the last document the next few days to
> >> > > > consolidate
> >> > > > effort results to some extend and break work down.
> >> > > > Our target is to provide a generic API based on some plugin
> >> > architecture
> >> > > to
> >> > > > serve different popular models/pipelines along with custom ones
> over
> >> > > flink.
> >> > > >
> >> > > > Best,
> >> > > > Stavros
> >> > > >
> >> > >
> >> >
> >
> >
> >
>

[jira] [Created] (FLINK-7109) HadoopUtils#getHadoopConfiguration should not load global configuration

2017-07-05 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-7109:


 Summary: HadoopUtils#getHadoopConfiguration should not load global 
configuration
 Key: FLINK-7109
 URL: https://issues.apache.org/jira/browse/FLINK-7109
 Project: Flink
  Issue Type: Improvement
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 1.4.0


The {{HadoopUtils#getHadoopConfiguration}} should not load the global 
configuration via {{GlobalConfiguration.loadConfiguration}}. Instead we should 
pass a valid Flink configuration to the static method. Otherwise, we will load 
the global configuration whenever {{HadoopUtils#getHadoopConfiguration}} is 
called.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7108) Implement Session cluster entry point

2017-07-05 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-7108:


 Summary: Implement Session cluster entry point
 Key: FLINK-7108
 URL: https://issues.apache.org/jira/browse/FLINK-7108
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management, YARN
Reporter: Till Rohrmann
Assignee: Till Rohrmann


Implement a Yarn session cluster entry point.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7107) Start Yarn session via start-up script

2017-07-05 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-7107:


 Summary: Start Yarn session via start-up script
 Key: FLINK-7107
 URL: https://issues.apache.org/jira/browse/FLINK-7107
 Project: Flink
  Issue Type: Sub-task
  Components: Startup Shell Scripts
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor


The {{yarn-session.sh}} shell script should be adapted to be able to also start 
a Flip-6 Yarn session cluster ({{YarnSessionClusterEntrypoint}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7106) Make SubmittedJobGraphStore configurable

2017-07-05 Thread Chen Qin (JIRA) 
Chen Qin created FLINK-7106:
---

 Summary: Make SubmittedJobGraphStore configurable
 Key: FLINK-7106
 URL: https://issues.apache.org/jira/browse/FLINK-7106
 Project: Flink
  Issue Type: Improvement
  Components: flink-contrib, Local Runtime
Reporter: Chen Qin


Current SubmittedJobGraphStore is hardcoded to store in zookeeper if user 
choose HAMode. The goal of this task is to allow user build their own 
implementation and pass configuration from flink.conf and define how/where 
those information stores. (e.g rocksdb statebackend)

P.S I think this would be interesting to see how flink in HA mode can fallback 
to Standalone when zk suffers temp outages.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7105) Make ActorSystem creation per default non-daemonic

2017-07-05 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-7105:


 Summary: Make ActorSystem creation per default non-daemonic
 Key: FLINK-7105
 URL: https://issues.apache.org/jira/browse/FLINK-7105
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination
Reporter: Till Rohrmann
Assignee: Till Rohrmann


At the moment, we create all {{ActorSystems}} with the setting {{daemonic=on}}. 
This has the consequence that we have to wait in the main thread on the 
{{ActorSystem's}} termination. By making the {{ActorSystems}} non-daemonic, we 
could get rid of this artifact. Especially since we have the {{ProcessReapers}} 
which terminate the process once a registered actor terminates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Task assignment

2017-07-05 Thread Chesnay Schepler 

Hello Andrey,

you now have contributor permissions and can assign issues to yourself.

Regards,
Chesnay Schepler

On 05.07.2017 12:13, Andrey Zhylka wrote:

Hello!

I would like to help with task 
https://issues.apache.org/jira/browse/FLINK-4303. Please assign it to me, my 
jira account is andrey_zhylka.

Best regards,
Andrey Zhylka
Software Engineer

Office: +375 17 389 0100 x 23597   Email: 
andrey_zhy...@epam.com
Minsk, Belarus (GMT+3)   epam.com

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) 
to which it is addressed and contains information that is legally privileged 
and confidential. If you are not the intended recipient, or the person 
responsible for delivering the message to the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited. All unintended recipients are obliged to 
delete this message and destroy any printed copies.

Task assignment

2017-07-05 Thread Andrey Zhylka 
Hello!

I would like to help with task 
https://issues.apache.org/jira/browse/FLINK-4303. Please assign it to me, my 
jira account is andrey_zhylka.

Best regards,
Andrey Zhylka
Software Engineer

Office: +375 17 389 0100 x 23597   
Email: andrey_zhy...@epam.com
Minsk, Belarus (GMT+3)   epam.com

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) 
to which it is addressed and contains information that is legally privileged 
and confidential. If you are not the intended recipient, or the person 
responsible for delivering the message to the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited. All unintended recipients are obliged to 
delete this message and destroy any printed copies.