[jira] [Created] (FLINK-8315) Use AWSCredentialsProvider to get AWS credentials

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-8315:
-

 Summary: Use AWSCredentialsProvider to get AWS credentials
 Key: FLINK-8315
 URL: https://issues.apache.org/jira/browse/FLINK-8315
 Project: Flink
  Issue Type: New Feature
  Components: Kinesis Connector
Reporter: Ivan Mushketyk
Priority: Minor


Instead of providing credentials like:

{code:java}
Properties consumerConfig = new Properties();
consumerConfig.put(ConsumerConfigConstants.AWS_REGION, "us-east-1");
consumerConfig.put(ConsumerConfigConstants.AWS_ACCESS_KEY_ID, "...");
consumerConfig.put(ConsumerConfigConstants.AWS_SECRET_ACCESS_KEY, 
"...");
consumerConfig.put(ConsumerConfigConstants.STREAM_INITIAL_POSITION, 
"TRIM_HORIZON");
{code}

Kinesis connector could use the AWSCredentialsProvider interface.

I can contribute it myself, but I wonder if this feature makes sense to you.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8314) Add support for Kinesis Firehose

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-8314:
-

 Summary: Add support for Kinesis Firehose
 Key: FLINK-8314
 URL: https://issues.apache.org/jira/browse/FLINK-8314
 Project: Flink
  Issue Type: New Feature
  Components: Kinesis Connector
Reporter: Ivan Mushketyk
Priority: Minor


[Kinesis 
Firehose|https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html]
 is another product in Kinesis Family that allows zero-ops scalable stream for 
delivering streaming data to S3, RedShift, Elasticsearch, etc. It has a 
different 
[API|http://docs.aws.amazon.com/firehose/latest/APIReference/Welcome.html].

I can contribute this myself, but I wonder if this feature makes sense to you.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Flink CEP development is stalling

[Ivan Mushketyk]

Hi Stephan!

Thank you for your answer.

I appreciate your efforts and I know how busy you and other Flink
committers are.
I am looking forward for the upcoming discussion of Flink modules!

Best regards,
Ivan.

On Mon, 16 Jan 2017 at 11:32 Stephan Ewen <se...@apache.org> wrote:

> Hi Ivan!
>
> Thank you for bringing this thread up. I agree, we need to do something
> about how some modules are currently handled.
>
> The CEP library definitely needs more active committers. Adding new
> committers will be necessary, I think, but as you mentioned, it needs at
> least one (better more) experienced committers that can help the new
> committers to get into the process and the technical matter. Otherwise we
> cannot keep up a good quality.
>
> How important the involvement of experienced committers is even for
> something that seems more or less self-contained (like the CEP library),
> has already been visible in some previous pull requests to the CEP library
> - those were not compatible with the overall design or with the strategy
> for making streaming applications rescalable. It is hard for new committers
> to be aware of all that, hence the need for some experienced committers to
> help.
>
> Currently, the community is pushing hard on the 1.2 release: testing, docs,
> fixes, usability. I expect that to take not too much longer.
> After that, we will kick off some threads discussing about community and
> project structure. That should involve how to deal with projects like the
> CEP library, and also with the sheer size of the project and code base in
> general.
>
> Greetings,
> Stephan
>
>
> On Sun, Jan 15, 2017 at 11:28 PM, Ivan Mushketyk <ivan.mushke...@gmail.com
> >
> wrote:
>
> > Hi Alexey,
> >
> > I agree with you. Most contributors are overloaded, but PRs for other
> > sub-projects are reviewed much faster. In my experience, in most cases,
> you
> > can get a first review for a PR in less than a week and it's usually
> merged
> > within a month or less.
> > Flink CEP is a notable exception. I believe the main reason for that is
> > that there is only one core committer who currently can review Flink CEP
> > PRs (Till) and he is very busy with other work.
> >
> > Best regards,
> > Ivan.
> >
> > On Sun, 15 Jan 2017 at 19:22 Alexey Demin <diomi...@gmail.com> wrote:
> >
> > > Hi Ivan,
> > >
> > > I think problem not only with CEP project.
> > > Main contributors overloaded and simple fixes frequently are staying as
> > PR
> > > without merge.
> > >
> > > You can see how amount of open PR increasing over time.
> > >
> > > Thanks,
> > > Alexey Diomin
> > >
> > >
> > > 2017-01-15 17:18 GMT+04:00 Ivan Mushketyk <ivan.mushke...@gmail.com>:
> > >
> > > > Hi Dmitry,
> > > >
> > > > Your contributions are welcomed, but right now the most critical
> issue
> > is
> > > > that CEP project does not have an experienced Flink contributor who
> can
> > > > review and approve new pull requests.
> > > >
> > > > I hope that Flink community will promptly resolve the issue, so feel
> > free
> > > > to take select a JIRA issue and work on it.
> > > >
> > > > Best regards,
> > > > Ivan.
> > > >
> > > > On Sat, 14 Jan 2017 at 12:29 Dmitry Vorobiov <
> 2belikespr...@gmail.com>
> > > > wrote:
> > > >
> > > > I would be interested to contribute to CEP. I am following Flink
> > project,
> > > > but haven't contributed yet. Next 2 weeks I am a bit busy with my
> work,
> > > but
> > > > then I llbe happy to dig into it. I used to work in IoT so event
> > > processing
> > > > is a close topic for me.
> > > >
> > > > Dmitry.
> > > > On Fri, 13 Jan 2017 at 14:58, Ivan Mushketyk <
> ivan.mushke...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Till,
> > > > >
> > > > > Thank you for your reply.
> > > > >
> > > > > I wonder if the following will work.
> > > > > What if you can find a Flink committer/committers that will review
> > and
> > > > > iterate on CEP PRs before you review them. They don't need to know
> > all
> > > > CEP
> > > > > internals, but they will help to eradicate most of the issues.
> > > > > Then you will have to review PRs only when most of the issue

Re: Flink CEP development is stalling

[Ivan Mushketyk]

Hi Alexey,

I agree with you. Most contributors are overloaded, but PRs for other
sub-projects are reviewed much faster. In my experience, in most cases, you
can get a first review for a PR in less than a week and it's usually merged
within a month or less.
Flink CEP is a notable exception. I believe the main reason for that is
that there is only one core committer who currently can review Flink CEP
PRs (Till) and he is very busy with other work.

Best regards,
Ivan.

On Sun, 15 Jan 2017 at 19:22 Alexey Demin <diomi...@gmail.com> wrote:

> Hi Ivan,
>
> I think problem not only with CEP project.
> Main contributors overloaded and simple fixes frequently are staying as PR
> without merge.
>
> You can see how amount of open PR increasing over time.
>
> Thanks,
> Alexey Diomin
>
>
> 2017-01-15 17:18 GMT+04:00 Ivan Mushketyk <ivan.mushke...@gmail.com>:
>
> > Hi Dmitry,
> >
> > Your contributions are welcomed, but right now the most critical issue is
> > that CEP project does not have an experienced Flink contributor who can
> > review and approve new pull requests.
> >
> > I hope that Flink community will promptly resolve the issue, so feel free
> > to take select a JIRA issue and work on it.
> >
> > Best regards,
> > Ivan.
> >
> > On Sat, 14 Jan 2017 at 12:29 Dmitry Vorobiov <2belikespr...@gmail.com>
> > wrote:
> >
> > I would be interested to contribute to CEP. I am following Flink project,
> > but haven't contributed yet. Next 2 weeks I am a bit busy with my work,
> but
> > then I llbe happy to dig into it. I used to work in IoT so event
> processing
> > is a close topic for me.
> >
> > Dmitry.
> > On Fri, 13 Jan 2017 at 14:58, Ivan Mushketyk <ivan.mushke...@gmail.com>
> > wrote:
> >
> > > Hi Till,
> > >
> > > Thank you for your reply.
> > >
> > > I wonder if the following will work.
> > > What if you can find a Flink committer/committers that will review and
> > > iterate on CEP PRs before you review them. They don't need to know all
> > CEP
> > > internals, but they will help to eradicate most of the issues.
> > > Then you will have to review PRs only when most of the issues are fixed
> > and
> > > to make a final decision about whether to merge a PR or not. In this
> > case,
> > > you probably won't need to spend much time on reviewing CEP PRs. As an
> > > additional bonus, after some time these new CEP reviewers will learn
> > enough
> > > about CEP to review them by themselves without your input.
> > >
> > > What do you think about this?
> > >
> > > Best regards,
> > > Ivan.
> > >
> > > On Fri, 13 Jan 2017 at 11:28 Till Rohrmann <trohrm...@apache.org>
> wrote:
> > >
> > > Hi Ivan,
> > >
> > > first of all let me apologise for the bad experience you've had with
> > > opening CEP PRs in the past.
> > >
> > > The general problem as you've said is that there is nobody who reviews
> > the
> > > open PRs. I used to do this in the but at the moment I hardly find time
> > due
> > > to other commitments.
> > >
> > > I think the way to mitigate the problem is to attract more contributors
> > and
> > > committers who are willing to spend time on PR reviews and finally
> (this
> > > applies only to committers) to commit the PRs. I can try to reach out
> to
> > > other committers to make them aware of the CEP library.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Jan 12, 2017 at 9:15 AM, <ruben.casado.teje...@accenture.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I have some clientes interested in CEP features
> > > >
> > > > El 11/1/17 16:23, "Ivan Mushketyk" <ivan.mushke...@gmail.com>
> > escribió:
> > > >
> > > > Hi Flink devs,
> > > >
> > > > Roughly half a year ago I implemented several PRs for
> > > > Flink-CEP[1][2][3]
> > > > but since then there were no progress with the reviews. What is
> > > > frustrating
> > > > about this situation is that Flink customers are asking for
> > features
> > > in
> > > > these PRs. For example customers commented on [1] and [2] asking
> > for
> > > > these
> > > > features. During a presentation about CEP during Flink Forward
> 2016
> > > > somebody as

Re: Flink CEP development is stalling

[Ivan Mushketyk]

Hi Dmitry,

Your contributions are welcomed, but right now the most critical issue is
that CEP project does not have an experienced Flink contributor who can
review and approve new pull requests.

I hope that Flink community will promptly resolve the issue, so feel free
to take select a JIRA issue and work on it.

Best regards,
Ivan.

On Sat, 14 Jan 2017 at 12:29 Dmitry Vorobiov <2belikespr...@gmail.com>
wrote:

I would be interested to contribute to CEP. I am following Flink project,
but haven't contributed yet. Next 2 weeks I am a bit busy with my work, but
then I llbe happy to dig into it. I used to work in IoT so event processing
is a close topic for me.

Dmitry.
On Fri, 13 Jan 2017 at 14:58, Ivan Mushketyk <ivan.mushke...@gmail.com>
wrote:

> Hi Till,
>
> Thank you for your reply.
>
> I wonder if the following will work.
> What if you can find a Flink committer/committers that will review and
> iterate on CEP PRs before you review them. They don't need to know all CEP
> internals, but they will help to eradicate most of the issues.
> Then you will have to review PRs only when most of the issues are fixed
and
> to make a final decision about whether to merge a PR or not. In this case,
> you probably won't need to spend much time on reviewing CEP PRs. As an
> additional bonus, after some time these new CEP reviewers will learn
enough
> about CEP to review them by themselves without your input.
>
> What do you think about this?
>
> Best regards,
> Ivan.
>
> On Fri, 13 Jan 2017 at 11:28 Till Rohrmann <trohrm...@apache.org> wrote:
>
> Hi Ivan,
>
> first of all let me apologise for the bad experience you've had with
> opening CEP PRs in the past.
>
> The general problem as you've said is that there is nobody who reviews the
> open PRs. I used to do this in the but at the moment I hardly find time
due
> to other commitments.
>
> I think the way to mitigate the problem is to attract more contributors
and
> committers who are willing to spend time on PR reviews and finally (this
> applies only to committers) to commit the PRs. I can try to reach out to
> other committers to make them aware of the CEP library.
>
> Cheers,
> Till
>
> On Thu, Jan 12, 2017 at 9:15 AM, <ruben.casado.teje...@accenture.com>
> wrote:
>
> > +1
> >
> > I have some clientes interested in CEP features
> >
> > El 11/1/17 16:23, "Ivan Mushketyk" <ivan.mushke...@gmail.com> escribió:
> >
> > Hi Flink devs,
> >
> > Roughly half a year ago I implemented several PRs for
> > Flink-CEP[1][2][3]
> > but since then there were no progress with the reviews. What is
> > frustrating
> > about this situation is that Flink customers are asking for features
> in
> > these PRs. For example customers commented on [1] and [2] asking for
> > these
> > features. During a presentation about CEP during Flink Forward 2016
> > somebody asked[4] for a feature that is implemented in [1]. Another
> CEP
> > feature that was requested in this Stack Overflow post[5] was
> > implemented
> > by PR [2].
> >
> > I also started conversations regarding following JIRA issues:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > apache.org_jira_browse_FLINK-2D4641=DgIBaQ=eIGjsITfXP_y-
> > DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> > a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> > Ikfcqrek_iJbQ=ml7ZfW_GN8zahdrUAEQrRD3KTBbvr6RbPpqTg71Fhao=
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > apache.org_jira_browse_FLINK-2D3414=DgIBaQ=eIGjsITfXP_y-
> > DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> > a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> > Ikfcqrek_iJbQ=uBNY1zlRRLXhvJQlguoBh4qXZVPRtHxmlavRCLD2UGE=
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > apache.org_jira_browse_FLINK-2D3320=DgIBaQ=eIGjsITfXP_y-
> > DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> > a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> > Ikfcqrek_iJbQ=7lPpQ1mPiTRr7qz8mX6GK0Qq9CWWeXcg8Bf3h43uJ08=  (wrote
to
> > Till about this
> > one)
> >
> > and I would like to work on them, but it seems pointless if nobody
is
> > going
> > to review new PRs.
> >
> > I wrote to Till(who is the only Flink CEP reviewer at the moment)
but
> > it
> > seems that he is very busy and cannot help with these PRs. On the
> other
> > hand Flink CEP has got some attention and customers are asking for
> new
> > fe

Re: Flink CEP development is stalling

[Ivan Mushketyk]

Hi Till,

Thank you for your reply.

I wonder if the following will work.
What if you can find a Flink committer/committers that will review and
iterate on CEP PRs before you review them. They don't need to know all CEP
internals, but they will help to eradicate most of the issues.
Then you will have to review PRs only when most of the issues are fixed and
to make a final decision about whether to merge a PR or not. In this case,
you probably won't need to spend much time on reviewing CEP PRs. As an
additional bonus, after some time these new CEP reviewers will learn enough
about CEP to review them by themselves without your input.

What do you think about this?

Best regards,
Ivan.

On Fri, 13 Jan 2017 at 11:28 Till Rohrmann <trohrm...@apache.org> wrote:

Hi Ivan,

first of all let me apologise for the bad experience you've had with
opening CEP PRs in the past.

The general problem as you've said is that there is nobody who reviews the
open PRs. I used to do this in the but at the moment I hardly find time due
to other commitments.

I think the way to mitigate the problem is to attract more contributors and
committers who are willing to spend time on PR reviews and finally (this
applies only to committers) to commit the PRs. I can try to reach out to
other committers to make them aware of the CEP library.

Cheers,
Till

On Thu, Jan 12, 2017 at 9:15 AM, <ruben.casado.teje...@accenture.com> wrote:

> +1
>
> I have some clientes interested in CEP features
>
> El 11/1/17 16:23, "Ivan Mushketyk" <ivan.mushke...@gmail.com> escribió:
>
> Hi Flink devs,
>
> Roughly half a year ago I implemented several PRs for
> Flink-CEP[1][2][3]
> but since then there were no progress with the reviews. What is
> frustrating
> about this situation is that Flink customers are asking for features
in
> these PRs. For example customers commented on [1] and [2] asking for
> these
> features. During a presentation about CEP during Flink Forward 2016
> somebody asked[4] for a feature that is implemented in [1]. Another
CEP
> feature that was requested in this Stack Overflow post[5] was
> implemented
> by PR [2].
>
> I also started conversations regarding following JIRA issues:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_FLINK-2D4641=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=ml7ZfW_GN8zahdrUAEQrRD3KTBbvr6RbPpqTg71Fhao=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_FLINK-2D3414=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=uBNY1zlRRLXhvJQlguoBh4qXZVPRtHxmlavRCLD2UGE=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_FLINK-2D3320=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=7lPpQ1mPiTRr7qz8mX6GK0Qq9CWWeXcg8Bf3h43uJ08=  (wrote to
> Till about this
> one)
>
> and I would like to work on them, but it seems pointless if nobody is
> going
> to review new PRs.
>
> I wrote to Till(who is the only Flink CEP reviewer at the moment) but
> it
> seems that he is very busy and cannot help with these PRs. On the
other
> hand Flink CEP has got some attention and customers are asking for new
> features.
>
> Is there any way for community to make progress with Flink CEP?
> Are there other core committers that can review Flink CEP PRs?
>
> Best regards,
> Ivan.
>
>
>
> [1] - https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_flink_pull_2361=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=dNLjBUb6wz8125hkAlm3tEzxLnZNfzeRrkFPEFJR2qM=
> [2] - https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_flink_pull_2367=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=tpKgiz_VFKYGXCn26GJolxFfSvRwqaMSpOPYFqG3ZYI=
> [3] - https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_flink_pull_2396=DgIBaQ=eIGjsITfXP_y-
> DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=brkRAgrW3LbdVDOiRLzI7SFUIWBL5a
> a2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc=OvPHe08A8IQPc6PIPmNkgZWKDWmAMA
> Ikfcqrek_iJbQ=28ZX8-B728xOrkxZ1DxhnOZfaZOB30fQQxWzJV5lrXY=
> [4] - https://

Flink CEP development is stalling

[Ivan Mushketyk]

Hi Flink devs,

Roughly half a year ago I implemented several PRs for Flink-CEP[1][2][3]
but since then there were no progress with the reviews. What is frustrating
about this situation is that Flink customers are asking for features in
these PRs. For example customers commented on [1] and [2] asking for these
features. During a presentation about CEP during Flink Forward 2016
somebody asked[4] for a feature that is implemented in [1]. Another CEP
feature that was requested in this Stack Overflow post[5] was implemented
by PR [2].

I also started conversations regarding following JIRA issues:
https://issues.apache.org/jira/browse/FLINK-4641
https://issues.apache.org/jira/browse/FLINK-3414
https://issues.apache.org/jira/browse/FLINK-3320 (wrote to Till about this
one)

and I would like to work on them, but it seems pointless if nobody is going
to review new PRs.

I wrote to Till(who is the only Flink CEP reviewer at the moment) but it
seems that he is very busy and cannot help with these PRs. On the other
hand Flink CEP has got some attention and customers are asking for new
features.

Is there any way for community to make progress with Flink CEP?
Are there other core committers that can review Flink CEP PRs?

Best regards,
Ivan.



[1] - https://github.com/apache/flink/pull/2361
[2] - https://github.com/apache/flink/pull/2367
[3] - https://github.com/apache/flink/pull/2396
[4] - https://youtu.be/vws5bv3XdD8?t=35m26s
[5] -
http://stackoverflow.com/questions/38225286/ho-can-i-do-a-lazy-match-with-flink-cep

[jira] [Created] (FLINK-5362) Implement methods to access BipartiteGraph properties

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5362:
-

 Summary: Implement methods to access BipartiteGraph properties
 Key: FLINK-5362
 URL: https://issues.apache.org/jira/browse/FLINK-5362
 Project: Flink
  Issue Type: Improvement
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5360) Fix arguments names in WindowedStream

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5360:
-

 Summary: Fix arguments names in WindowedStream
 Key: FLINK-5360
 URL: https://issues.apache.org/jira/browse/FLINK-5360
 Project: Flink
  Issue Type: Bug
Reporter: Ivan Mushketyk


Should be "field" instead of "positionToMaxBy" in some methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5349) Fix code sample for Twitter connector

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5349:
-

 Summary: Fix code sample for Twitter connector
 Key: FLINK-5349
 URL: https://issues.apache.org/jira/browse/FLINK-5349
 Project: Flink
  Issue Type: Bug
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk


There is a typo in code sample for Twitter connector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5245) Add support for BipartiteGraph mutations

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5245:
-

 Summary: Add support for BipartiteGraph mutations
 Key: FLINK-5245
 URL: https://issues.apache.org/jira/browse/FLINK-5245
 Project: Flink
  Issue Type: Improvement
  Components: Gelly
Reporter: Ivan Mushketyk


Implement methods for adding and removing vertices and edges similarly to Graph 
class.

Depends on https://issues.apache.org/jira/browse/FLINK-2254



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5244) Implement methods for BipartiteGraph transformations

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5244:
-

 Summary: Implement methods for BipartiteGraph transformations
 Key: FLINK-5244
 URL: https://issues.apache.org/jira/browse/FLINK-5244
 Project: Flink
  Issue Type: Improvement
  Components: Gelly
Reporter: Ivan Mushketyk


BipartiteGraph should implement methods for transforming graph, like map, 
filter, join, union, difference, etc. similarly to Graph class.

Depends on: https://issues.apache.org/jira/browse/FLINK-2254



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5243) Implement an example for BipartiteGraph

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5243:
-

 Summary: Implement an example for BipartiteGraph
 Key: FLINK-5243
 URL: https://issues.apache.org/jira/browse/FLINK-5243
 Project: Flink
  Issue Type: New Feature
  Components: Gelly
Reporter: Ivan Mushketyk


Should implement example for BipartiteGraph in gelly-examples project similarly 
to examples for Graph class.

Depends on this: https://issues.apache.org/jira/browse/FLINK-2254



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5242) Implement Scala API for BipartiteGraph

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5242:
-

 Summary: Implement Scala API for BipartiteGraph
 Key: FLINK-5242
 URL: https://issues.apache.org/jira/browse/FLINK-5242
 Project: Flink
  Issue Type: New Feature
  Components: Gelly
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk


Should implement BipartiteGraph in flink-gelly-scala project similarly to Graph 
class.

Depends on this: https://issues.apache.org/jira/browse/FLINK-2254



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5104) Implement BipartiteGraph validator

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-5104:
-

 Summary: Implement BipartiteGraph validator
 Key: FLINK-5104
 URL: https://issues.apache.org/jira/browse/FLINK-5104
 Project: Flink
  Issue Type: Sub-task
  Components: Gelly
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk


BipartiteGraph should have a validator similar to GraphValidator for Graph 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4649) Implement bipartite graph metrics

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4649:
-

 Summary: Implement bipartite graph metrics
 Key: FLINK-4649
 URL: https://issues.apache.org/jira/browse/FLINK-4649
 Project: Flink
  Issue Type: Sub-task
  Components: Gelly
Reporter: Ivan Mushketyk


Implement metrics calculation for a bipartite graph. Should be similar to 
EdgeMetrics and VertexMetrics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4648) Implement bipartite graph generators

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4648:
-

 Summary: Implement bipartite graph generators
 Key: FLINK-4648
 URL: https://issues.apache.org/jira/browse/FLINK-4648
 Project: Flink
  Issue Type: Sub-task
  Components: Gelly
Reporter: Ivan Mushketyk


Implement generators for bipartite graphs.
Should implement at least:
* *BipartiteGraphGenerator* (maybe requires a better name) that will generate a 
bipartite graph where every vertex of one set is connected only to some 
vertices  from another set
* *CompleteBipartiteGraphGenerator* that will generate a graph where every 
vertex of one set is conneted to every vertex of another set




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4647) Implement BipartiteGraph reader

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4647:
-

 Summary: Implement BipartiteGraph reader
 Key: FLINK-4647
 URL: https://issues.apache.org/jira/browse/FLINK-4647
 Project: Flink
  Issue Type: Sub-task
  Components: Gelly
Reporter: Ivan Mushketyk


Implement reading bipartite graph from a CSV. Should be similar to how regular 
graph is read from a file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4646) Add BipartiteGraph class

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4646:
-

 Summary: Add BipartiteGraph class
 Key: FLINK-4646
 URL: https://issues.apache.org/jira/browse/FLINK-4646
 Project: Flink
  Issue Type: Sub-task
Reporter: Ivan Mushketyk






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: build error

[Ivan Mushketyk]

Hi Ameet,

The new issue that you are facing is caused by a build-time validator that
checks if every file in Flink sources has an Apache license text in it.
It suggests that there are two files without a proper license:
* build_log_maven_flink
* log.oracle.jdk

If you remove them or move from the Flink source folder this error will go
away.

Best regards,
Ivan.

On Thu, 15 Sep 2016 at 15:57 Ameet BD <ameetbigd...@gmail.com> wrote:

> Hi Ivan,
>
> Thanks for reply. I tried installing Oracle JDK. The build error log is
> attached for details.
> Error reported is "Too many files with unapproved license".
> The file rat.txt is attached.
>
> Please tell me what could be going wrong. I am using Ubuntu 14.04, Maven
> 3.0.5 and Oracle JDK 8.
> Regards,
> Ameet
>
> On Thu, Sep 15, 2016 at 5:45 PM, Ivan Mushketyk <ivan.mushke...@gmail.com>
> wrote:
>
>> Hi Ameet,
>>
>> This seems like a possible JVM bug.
>> Could you try to build Flink using OracleJDK instead?
>>
>> Best regards,
>> Ivan.
>>
>> On Thu, 15 Sep 2016 at 12:30 Ameet BD <ameetbigd...@gmail.com> wrote:
>>
>> > Hi All,
>> >
>> > I am trying to build latest Flink code from github.com by running
>> command
>> >
>> > $ git clone https://github.com/apache/flink
>> > After that I am running command
>> > $ mvn clean install -DskipTests
>> >
>> > The build fails I have attached the build log.
>> >
>> > Kindly let me know why it may be failing.
>> >
>> > Regards.
>> > Ameet
>> >
>>
>
>

Re: build error

[Ivan Mushketyk]

Hi Ameet,

This seems like a possible JVM bug.
Could you try to build Flink using OracleJDK instead?

Best regards,
Ivan.

On Thu, 15 Sep 2016 at 12:30 Ameet BD  wrote:

> Hi All,
>
> I am trying to build latest Flink code from github.com by running command
>
> $ git clone https://github.com/apache/flink
> After that I am running command
> $ mvn clean install -DskipTests
>
> The build fails I have attached the build log.
>
> Kindly let me know why it may be failing.
>
> Regards.
> Ameet
>

[DISCUSS] Automated code quality check in Flink

[Ivan Mushketyk]

Hi!

Flink uses Travis and Jenkis to check if new PR passes unit tests, but
there are other tools that can automatically check code quality like:
https://www.codacy.com/
https://codeclimate.com/

These tools promise to track test coverage, check code style, detect code
duplication and so on.
Since they are free for Open Source projects, it seems like it could be
quite beneficial to enable them.
Have you ever tried any of them? What do you think about this?

Best regards,
Ivan.

[jira] [Created] (FLINK-4521) Fix "Submit new Job" panel in development mode

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4521:
-

 Summary: Fix "Submit new Job" panel in development mode
 Key: FLINK-4521
 URL: https://issues.apache.org/jira/browse/FLINK-4521
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk


If web frontend is started in the development mode, "Submit new Job" panel is 
empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS][FLINK-3414] Scala API for CEP's pattern definition

[Ivan Mushketyk]

Hi,

Currently CEP supports API for pattern definition, but it is very similar
to the Java one. Using Scala we could achieve much more concise API.

In JIRA comment for this task I've described a proposal for how this API
could look like: [1]

Could you please review it and write your comments?

Best regards,
Ivan.



[1] -
https://issues.apache.org/jira/browse/FLINK-3414?focusedCommentId=15404852=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15404852

[DISCUSS] [FLINK-3950] Meter interface

[Ivan Mushketyk]

Hi,

I was working on the creating new Meter metric type, but I need some input
to define a proper interface for it.
Initial idea developed in this pull request [1] was to follow Dropwizard's
meter interface, but zentol suggested that we may need a different
interface for Flink use-cases. The issue with Dropwizard Meter interface is
that it only supports 1, 5, and 15 minutes rates, while we may need
different rates (e.g. sub-minute rate) or may not need one of these rates.

I think that a way to implement a flexible Meter type would be by adding
two methods:

List getRates()  // returns a list of rates a Meter supports,
e.g. {1, 5, 15} would mean that a particular Meter supports 1, 5, and 15
minutes rates
double getRate(double rate)   // get moving average for provided rate

What do you think about it?

Best regards,
Ivan.

[1] - https://github.com/apache/flink/pull/2374

[jira] [Created] (FLINK-4440) Make API for edge/vertex creation less verbose

[Ivan Mushketyk (JIRA)]

Ivan Mushketyk created FLINK-4440:
-

 Summary: Make API for edge/vertex creation less verbose
 Key: FLINK-4440
 URL: https://issues.apache.org/jira/browse/FLINK-4440
 Project: Flink
  Issue Type: Improvement
  Components: Gelly
Reporter: Ivan Mushketyk
Assignee: Ivan Mushketyk
Priority: Trivial


It would be better if one could create vertex/edges like this:

{code:java}
Vertex<Integer, NullValue> v = Vertex.create(42);
Edge<Integer, Integer, NullValue> e = Edge.create(5, 6);
{code}

Instead of this:
{code:java}
Vertex<Integer, NullValue> v = new Vertex<Integer, NullValue>(42, 
NullValue.getInstance());
Edge<Integer, Integer, NullValue> e = new Edge<Integer, Integer, NullValue>(5, 
6, NullValue.getInstance());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Python API for Fllink libraries

[Ivan Mushketyk]

Hi Chesnay,

Thank you for you repply.
Out of curiosity, do you know why Python API reception was  *tumbleweed*?

Regarding the Python API, do you know what specifically should be done
there? I have some Python background so I was considering to contribute,
but I didn't find much tasks in the "Python" component:
https://issues.apache.org/jira/browse/FLINK-1926?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

Best regards,
Ivan.


On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler <ches...@apache.org> wrote:

> Hello,
>
> I would say no, as the general reception of the Python API was
> *tumbleweed* so far.
>
> In my opinion this would just lead to a massive increase in code to
> maintain; we would need at least 2-3 active long-term python contributors.
> Especially so since ML, CEP and Table are afaik still in heavy development.
>
> If anything, before thinking about porting the libraries to python it
> would make more sense to implement a python streaming API.
> Or maybe /finish/ porting the DataSet API...
>
> Regards,
> Chesnay
>
> On 19.08.2016 22:07, Ivan Mushketyk wrote:
> > Hi Flink developers,
> >
> > It seems to me that Flink has two important "selling points":
> >
> > 1. It has Java, Scala and Python APIs
> > 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
> >
> > But as far as I understand, currently users cannot use any of these
> > libraries using a Python API. It seems to be a gap worth filling.
> >
> > What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
> > Python APIs?
> >
> > Best regards,
> > Ivan.
> >
>
>

[DISCUSS] Python API for Fllink libraries

[Ivan Mushketyk]

Hi Flink developers,

It seems to me that Flink has two important "selling points":

1. It has Java, Scala and Python APIs
2. I has a number of useful libraries (ML, Gelly, CEP, and Table)

But as far as I understand, currently users cannot use any of these
libraries using a Python API. It seems to be a gap worth filling.

What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
Python APIs?

Best regards,
Ivan.

Re: Jenkins and Travis CI failing

[Ivan Mushketyk]

Hi,

Some Flink tests are flaky and fail on some platforms, so failures may have
nothing to do with your changes.
Since your build passes on most platforms, I would assume that the code is
fine, but it is up to someone more experienced in Flink to decide.
Flink community is working towards improving this, but right now we have to
deal with it.

Best regards,
Ivan.

On Mon, 15 Aug 2016 at 17:41 Sunny T  wrote:

> Hi,
>
> I created a pull request for FLINK-4309 last night. Here is the pull
> request: https://github.com/apache/flink/pull/2371.
>
> The jenkins and Travis CI are failing. But the Travis CI passed on my
> forked repository.
>
> Can anyone help me understand what is wrong here?
>
> Thanks,
> Sunny
>

Re: Assigning an issue to self on JIRA

[Ivan Mushketyk]

Hi Sunny,

I don't think I did anything except creating a JIRA account.
I wanted to set you as the assignee for this task, but your name does not
come up in the auto-complete form. I wonder if somehow you didn't finish
the account creation process.
If the issues persists, maybe someone from the core team can help you with
this issue?

Best regards,
Ivan.

On Sun, 14 Aug 2016 at 22:59 Sunny T <tsunny@gmail.com> wrote:

> Hi Ivan,
>
> I am not able to assign the Assignee Field, it's not clickable for me. I
> created my JIRA account few hours back. Do I need to be added to some group
> for me to be able to edit the JIRA item?
>
> Thanks,
> Sunny
>
> On Sun, Aug 14, 2016 at 2:37 PM, Ivan Mushketyk <ivan.mushke...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am new to Flink myself, but I don't think there is any issues with
> > assigning issues to yourself.
> >
> > Best regards,
> > Ivan.
> >
> > On Sun, 14 Aug 2016 at 21:48 Sunny T <tsunny@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would like to contribute to Apache Flink. And I have chosen this
> minor
> > > bug to work on: FLINK-4309
> > > <https://issues.apache.org/jira/browse/FLINK-4309>. However, I am not
> > able
> > > to assign the issue to myself. How can I assign the issue to myself?
> > >
> > > Thanks,
> > > Sunny
> > >
> >
>

Re: Assigning an issue to self on JIRA

[Ivan Mushketyk]

Hi,

I am new to Flink myself, but I don't think there is any issues with
assigning issues to yourself.

Best regards,
Ivan.

On Sun, 14 Aug 2016 at 21:48 Sunny T  wrote:

> Hi,
>
> I would like to contribute to Apache Flink. And I have chosen this minor
> bug to work on: FLINK-4309
> . However, I am not able
> to assign the issue to myself. How can I assign the issue to myself?
>
> Thanks,
> Sunny
>

Implementing multi-field "sum" function [FLINK-4029]

[Ivan Mushketyk]

Hi,

I would like to implement a multi-field "sum" function[1], but I am not
sure what would be the best way to approach it.
Should I add a new method to the DataStream class or would it be better to
add it to a specialized stream?
Should it be based on existing operators/streams, or should I implement a
new operator/stream?
What do you think about this?

[1] - https://issues.apache.org/jira/browse/FLINK-4029

Best regards,
Ivan.