[jira] [Created] (FLINK-4516) ResourceManager leadership election

[zhangjing (JIRA)]

zhangjing created FLINK-4516:


 Summary: ResourceManager leadership election
 Key: FLINK-4516
 URL: https://issues.apache.org/jira/browse/FLINK-4516
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Reporter: zhangjing


1. When a resourceManager is started, it starts the leadership election service 
first and take part in contending for leadership
2. Every resourceManager contains a ResourceManagerLeaderContender, when it is 
granted leadership, it will start SlotManager and other main components. when 
it is revoked leadership, it will stop all its components and clear everything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1985: [FLINK-1979] Add logistic loss, hinge loss and regulariza...

[skavulya]

Github user skavulya commented on the issue:

https://github.com/apache/flink/pull/1985
  
hi @chiwanpark sorry, I hadn't checked this PR for a while. I merged the 
latest master. Do you have any preference on an alternative name for 
RegularizationPenalty? Would Regularizer work better?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1979) Implement Loss Functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440354#comment-15440354
 ] 

ASF GitHub Bot commented on FLINK-1979:
---

Github user skavulya commented on the issue:

https://github.com/apache/flink/pull/1985
  
hi @chiwanpark sorry, I hadn't checked this PR for a while. I merged the 
latest master. Do you have any preference on an alternative name for 
RegularizationPenalty? Would Regularizer work better?


> Implement Loss Functions
> 
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
>  Issue Type: Improvement
>  Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
>  Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a 
> pluggable implementation of a loss function and its first derivative.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4516) ResourceManager leadership election

[zhangjing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangjing reassigned FLINK-4516:


Assignee: zhangjing

> ResourceManager leadership election
> ---
>
> Key: FLINK-4516
> URL: https://issues.apache.org/jira/browse/FLINK-4516
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: zhangjing
>Assignee: zhangjing
>
> 1. When a resourceManager is started, it starts the leadership election 
> service first and take part in contending for leadership
> 2. Every resourceManager contains a ResourceManagerLeaderContender, when it 
> is granted leadership, it will start SlotManager and other main components. 
> when it is revoked leadership, it will stop all its components and clear 
> everything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2388: [FLINK-4347][cluster management] Implement SlotManager co...

[KurtYoung]

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/2388
  
I think i shall do the resolving thing.
The commits are squashed and conflicts been resolved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4347) Implement SlotManager core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440532#comment-15440532
 ] 

ASF GitHub Bot commented on FLINK-4347:
---

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/2388
  
I think i shall do the resolving thing.
The commits are squashed and conflicts been resolved.


> Implement SlotManager core
> --
>
> Key: FLINK-4347
> URL: https://issues.apache.org/jira/browse/FLINK-4347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> The slot manager is responsible to maintain the list of slot requests and 
> slot allocations. It allows to request slots from the registered 
> TaskExecutors and issues container allocation requests in case that there are 
> not enough available resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2109: [FLINK-3677] FileInputFormat: Allow to specify include/ex...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2109
  
@aljoscha As already pointed out in the JIRA issue, the method is not new. 
A now obsolete method has been deprecated (the diff is hard to read).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438938#comment-15438938
 ] 

ASF GitHub Bot commented on FLINK-3677:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2109
  
@aljoscha As already pointed out in the JIRA issue, the method is not new. 
A now obsolete method has been deprecated (the diff is hard to read).


> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438936#comment-15438936
 ] 

Maximilian Michels commented on FLINK-3677:
---

No worries. Your changes are not wrong in any way. 

A lot of people simply do not like the Guava library because it usually just 
adds an additional dependency without adding much simplification to the code. I 
think it is fine to have it in testing scope.

Further, I think it is ok to first deprecate an old API method before removing 
it. Even if it is PublicEvolving.

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3950) Add Meter Metric Type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438875#comment-15438875
 ] 

ASF GitHub Bot commented on FLINK-3950:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76410471
  
--- Diff: 
flink-metrics/flink-metrics-dropwizard/src/test/java/org/apache/flink/dropwizard/metrics/DropwizardMeterWrapperTest.java
 ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.dropwizard.metrics;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class DropwizardMeterWrapperTest {
+   private static final double DELTA = 0.0001;
+
+   @Test
+   public void testWrapper() {
+   com.codahale.metrics.Meter dropwizardMeter = 
mock(com.codahale.metrics.Meter.class);
+   when(dropwizardMeter.getOneMinuteRate()).thenReturn(1.0);
+   when(dropwizardMeter.getCount()).thenReturn(100L);
+
+   DropwizardMeterWrapper wrapper = new 
DropwizardMeterWrapper(dropwizardMeter);
+
+   assertEquals(1.0, wrapper.getRate(), DELTA);
--- End diff --

this is minor, but i don't think we need a constant if it is only used once.


> Add Meter Metric Type
> -
>
> Key: FLINK-3950
> URL: https://issues.apache.org/jira/browse/FLINK-3950
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Stephan Ewen
>Assignee: Ivan Mushketyk
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4487) Need tools for managing the yarn-session better

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439112#comment-15439112
 ] 

Maximilian Michels commented on FLINK-4487:
---

I totally forgot that there is also a way to shutdown the detached yarn session 
through {{ yarn-session.sh -id }} which resumes the yarn 
session. After resuming you can "stop" the Yarn session cli to shutdown the 
yarn session and application.

We could add an additional parameter to just shutdown instead of resuming it 
first.

> Need tools for managing the yarn-session better
> ---
>
> Key: FLINK-4487
> URL: https://issues.apache.org/jira/browse/FLINK-4487
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> There is a script yarn-session.sh which starts an empty JobManager on a Yarn 
> cluster. 
> Desired improvements:
> # If there is already a yarn-session running then yarn-session does not start 
> a new one (or it kills the old one?). Note that the file with ip/port may 
> exist yet the corresponding JobManager may have been killed in an other way.
> # A script that effectively lets me stop a yarn session and cleanup the file 
> that contains the ip/port of this yarn session and the .flink directory on 
> HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2391: [FLINK-3580] [table] Implement FLOOR/CEIL for time...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2391


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3580) Reintroduce Date/Time and implement scalar functions for it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439125#comment-15439125
 ] 

ASF GitHub Bot commented on FLINK-3580:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2391


> Reintroduce Date/Time and implement scalar functions for it
> ---
>
> Key: FLINK-3580
> URL: https://issues.apache.org/jira/browse/FLINK-3580
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> This task includes:
> {code}
> DATETIME_PLUS
> EXTRACT_DATE
> FLOOR
> CEIL
> CURRENT_TIME
> CURRENT_TIMESTAMP
> LOCALTIME
> LOCALTIMESTAMP
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4487) Need tools for managing the yarn-session better

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438662#comment-15438662
 ] 

Aljoscha Krettek commented on FLINK-4487:
-

This could also already be addressed by the work on FLIP-6. (See my answer on 
FLINK-4486).

Could [~StephanEwen] or [~trohrm...@apache.org] chime in on this?

> Need tools for managing the yarn-session better
> ---
>
> Key: FLINK-4487
> URL: https://issues.apache.org/jira/browse/FLINK-4487
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> There is a script yarn-session.sh which starts an empty JobManager on a Yarn 
> cluster. 
> Desired improvements:
> # If there is already a yarn-session running then yarn-session does not start 
> a new one (or it kills the old one?). Note that the file with ip/port may 
> exist yet the corresponding JobManager may have been killed in an other way.
> # A script that effectively lets me stop a yarn session and cleanup the file 
> that contains the ip/port of this yarn session and the .flink directory on 
> HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2422: FLINK-4499: Introduce findbugs maven plugin

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2422
  
Thanks for adding this.

I think that pure warning by the plugin will not be read and respected 
really. So it makes only sense when a "bug" detected by the plugin fails the 
build.

How about starting with a low set of settings and configuring it to fail 
when detecting errors? We can then gradually activate more and more visitors.

Also, do you have any experience in how finbugs usually affects the build 
times?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[Ivan Mushketyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438716#comment-15438716
 ] 

Ivan Mushketyk commented on FLINK-3677:
---

Stephan, Aljoscha

Thank you for pointing this out.
I'll remove the new method and Guava dependency today and submit a new PR.
Sorry for the inconvenience.

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4502) Cassandra connector documentation has misleading consistency guarantees

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438754#comment-15438754
 ] 

Chesnay Schepler commented on FLINK-4502:
-

Alright, so let's break this down point by point:

You're description of how the WAL works is correct. However, it only attempts 
to prevent *a different version* of a checkpoint from being committed. 

You are correct that the sink should stop sending data when sendValues returns 
false which can lead to some data for that checkpoint from never being written. 
However, you are incorrect in regards to this potentially overwriting newer 
data. We manually set the time stamp of the queries based on the checkpoint id; 
newer checkpoint => newer timestamp, which means that even if the query is 
submitted to cassandra it will not overwrite anything.

The documentation states that we provide exactly-once semantics for idempotent 
updates, as such by definition the writes to cassandra at any point cannot be 
non-idempotent.

The sink does in fact not guarantee exactly-once *delivery*, but it doesn't 
claim that it does. It fulfills exactly-once *semantics* in so far that, which 
pretty much means that if you look at any point in time at the data in 
cassandra it will show a state that would be reached if the messages would be 
delivered exactly-once.

> Cassandra connector documentation has misleading consistency guarantees
> ---
>
> Key: FLINK-4502
> URL: https://issues.apache.org/jira/browse/FLINK-4502
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.1.0
>Reporter: Elias Levy
>
> The Cassandra connector documentation states that  "enableWriteAheadLog() is 
> an optional method, that allows exactly-once processing for non-deterministic 
> algorithms."  This claim appears to be false.
> From what I gather, the write ahead log feature of the connector works as 
> follows:
> - The sink is replaced with a stateful operator that writes incoming messages 
> to the state backend based on checkpoint they belong in.
> - When the operator is notified that a Flink checkpoint has been completed 
> it, for each set of checkpoints older than and including the committed one:
>   * reads its messages from the state backend
>   * writes them to Cassandra
>   * records that it has committed them to Cassandra for the specific 
> checkpoint and operator instance
>* and erases them from the state backend.
> This process attempts to avoid resubmitting queries to Cassandra that would 
> otherwise occur when recovering a job from a checkpoint and having messages 
> replayed.
> Alas, this does not guarantee exactly once semantics at the sink.  The writes 
> to Cassandra that occur when the operator is notified that checkpoint is 
> completed are not atomic and they are potentially non-idempotent.  If the job 
> dies while writing to Cassandra or before committing the checkpoint via 
> committer, queries will be replayed when the job recovers.  Thus the 
> documentation appear to be incorrect in stating this provides exactly-once 
> semantics.
> There also seems to be an issue in GenericWriteAheadSink's 
> notifyOfCompletedCheckpoint which may result in incorrect output.  If 
> sendValues returns false because a write failed, instead of bailing, it 
> simply moves on to the next checkpoint to commit if there is one, keeping the 
> previous one around to try again later.  But that can result in newer data 
> being overwritten with older data when the previous checkpoint is retried.  
> Although given that CassandraCommitter implements isCheckpointCommitted as 
> checkpointID <= this.lastCommittedCheckpointID, it actually means that when 
> it goes back to try the uncommitted older checkpoint it will consider it 
> committed, even though some of its data may not have been written out, and 
> the data will be discarded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4190) Generalise RollingSink to work with arbitrary buckets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438770#comment-15438770
 ] 

ASF GitHub Bot commented on FLINK-4190:
---

Github user joshfg commented on the issue:

https://github.com/apache/flink/pull/2269
  
Hi Aljoscha, just wanted to remind you about this - any idea when the 
changes will be merged in? Thanks!


> Generalise RollingSink to work with arbitrary buckets
> -
>
> Key: FLINK-4190
> URL: https://issues.apache.org/jira/browse/FLINK-4190
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector, Streaming Connectors
>Reporter: Josh Forman-Gornall
>Assignee: Josh Forman-Gornall
>Priority: Minor
>
> The current RollingSink implementation appears to be intended for writing to 
> directories that are bucketed by system time (e.g. minutely) and to only be 
> writing to one file within one bucket at any point in time. When the system 
> time determines that the current bucket should be changed, the current bucket 
> and file are closed and a new bucket and file are created. The sink cannot be 
> used for the more general problem of writing to arbitrary buckets, perhaps 
> determined by an attribute on the element/tuple being processed.
> There are three limitations which prevent the existing sink from being used 
> for more general problems:
> - Only bucketing by the current system time is supported, and not by e.g. an 
> attribute of the element being processed by the sink.
> - Whenever the sink sees a change in the bucket being written to, it flushes 
> the file and moves on to the new bucket. Therefore the sink cannot have more 
> than one bucket/file open at a time. Additionally the checkpointing mechanics 
> only support saving the state of one active bucket and file.
> - The sink determines that it should 'close' an active bucket and file when 
> the bucket path changes. We need another way to determine when a bucket has 
> become inactive and needs to be closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3950) Add Meter Metric Type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438884#comment-15438884
 ] 

ASF GitHub Bot commented on FLINK-3950:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76411144
  
--- Diff: 
flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/Meter.java
 ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+/**
+ * Metric for measuring average throughput.
--- End diff --

can we cut `average`? Maybe it's just me, but i think we should provide raw 
data and let the metric backends handle averaging/weighing and all that fancy 
stuff.


> Add Meter Metric Type
> -
>
> Key: FLINK-3950
> URL: https://issues.apache.org/jira/browse/FLINK-3950
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Stephan Ewen
>Assignee: Ivan Mushketyk
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4487) Need tools for managing the yarn-session better

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438883#comment-15438883
 ] 

Maximilian Michels commented on FLINK-4487:
---

{quote}
1. If there is already a yarn-session running then yarn-session does not start 
a new one (or it kills the old one?). Note that the file with ip/port may exist 
yet the corresponding JobManager may have been killed in an other way.
{quote}

The yarn properties file does not contain the ip/port anymore. Instead, it uses 
the Yarn application id. A new Yarn session always overrides the previous 
session information.

{quote}
2. A script that effectively lets me stop a yarn session and cleanup the file 
that contains the ip/port of this yarn session and the .flink directory on HDFS.
{quote}

If you use the non-detached Yarn session, cleanup is automatically done. If you 
use the detached session, you will have to stop the yarn application (using 
yarn commands) and delete the yarn properties file in the temp directory. Good 
idea to let a script do that! We can now do that reliably because we have the 
yarn application id in the properties file.

> Need tools for managing the yarn-session better
> ---
>
> Key: FLINK-4487
> URL: https://issues.apache.org/jira/browse/FLINK-4487
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> There is a script yarn-session.sh which starts an empty JobManager on a Yarn 
> cluster. 
> Desired improvements:
> # If there is already a yarn-session running then yarn-session does not start 
> a new one (or it kills the old one?). Note that the file with ip/port may 
> exist yet the corresponding JobManager may have been killed in an other way.
> # A script that effectively lets me stop a yarn session and cleanup the file 
> that contains the ip/port of this yarn session and the .flink directory on 
> HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2374: [FLINK-3950] Add Meter Metric Type

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76411144
  
--- Diff: 
flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/Meter.java
 ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+/**
+ * Metric for measuring average throughput.
--- End diff --

can we cut `average`? Maybe it's just me, but i think we should provide raw 
data and let the metric backends handle averaging/weighing and all that fancy 
stuff.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438935#comment-15438935
 ] 

Maximilian Michels commented on FLINK-3677:
---

I reviewed the pull requests and I was aware of these change.

Guava is only a test dependency which shouldn't affect users and download size. 
I think it is fine to have it in testing scope.

The diff is hard to read, but he deprecated the old read method and introduced 
a new one. I explicitly told him to not simply change the existing method. We 
can safely remove the deprecated method after the next release. 

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4311) TableInputFormat fails when reused on next split

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439102#comment-15439102
 ] 

ASF GitHub Bot commented on FLINK-4311:
---

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2330
  
I managed to resolve the problems with running these unit tests. 
These problems were caused by version conflicts in guava.
Now we have a HBaseMiniCluster that is started, a table with multiple 
regions is created. And the TableInputFormat is used to extract the rows again. 
By setting the paralellism to 1 the same TableInputFormat instance is used for 
multiple regions and succeeds (the problem this all started with).

Please review.


> TableInputFormat fails when reused on next split
> 
>
> Key: FLINK-4311
> URL: https://issues.apache.org/jira/browse/FLINK-4311
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.0.3
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Critical
>
> We have written a batch job that uses data from HBase by means of using the 
> TableInputFormat.
> We have found that this class sometimes fails with this exception:
> {quote}
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: 
> Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
>  rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
>   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
>   at 
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:152)
>   at 
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:47)
>   at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:147)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.RejectedExecutionException: Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
>  rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:142)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:269)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:165)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
>   ... 10 more
> {quote}
> As you can see the ThreadPoolExecutor was terminated at this point.
> We tracked it down to the fact that 
> # the configure method opens the table
> # the open method obtains the result scanner
> # the closes method closes the table.
> If a second split arrives on the same instance then the open method will fail 
> because the table has already been closed.
> We also found that this error varies with the versions of HBase that are 
> used. I have also seen this exception:
> {quote}
> Caused by: java.io.IOException: hconnection-0x19d37183 closed
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
>   ... 37 more
> {quote}
> I found that in the [documentation of the InputFormat 
> interface|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/io/InputFormat.html]
>  is clearly states
> 

[GitHub] flink pull request #2374: [FLINK-3950] Add Meter Metric Type

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76410471
  
--- Diff: 
flink-metrics/flink-metrics-dropwizard/src/test/java/org/apache/flink/dropwizard/metrics/DropwizardMeterWrapperTest.java
 ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.dropwizard.metrics;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class DropwizardMeterWrapperTest {
+   private static final double DELTA = 0.0001;
+
+   @Test
+   public void testWrapper() {
+   com.codahale.metrics.Meter dropwizardMeter = 
mock(com.codahale.metrics.Meter.class);
+   when(dropwizardMeter.getOneMinuteRate()).thenReturn(1.0);
+   when(dropwizardMeter.getCount()).thenReturn(100L);
+
+   DropwizardMeterWrapper wrapper = new 
DropwizardMeterWrapper(dropwizardMeter);
+
+   assertEquals(1.0, wrapper.getRate(), DELTA);
--- End diff --

this is minor, but i don't think we need a constant if it is only used once.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4508) Consolidate DummyEnvironment and MockEnvironment for Tests

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-4508:
-

 Summary: Consolidate DummyEnvironment and MockEnvironment for Tests
 Key: FLINK-4508
 URL: https://issues.apache.org/jira/browse/FLINK-4508
 Project: Flink
  Issue Type: Test
Reporter: Stefan Richter
Priority: Minor


Currently we {{DummyEnvironment}} and {{MockEnvironment}} as implementations of 
Environments for our test. Both serve a similar purpose, but offer slightly 
different features. We should consolidate this by merging them into one class 
that offers the best of both previous implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat

[nielsbasjes]

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2330
  
I managed to resolve the problems with running these unit tests. 
These problems were caused by version conflicts in guava.
Now we have a HBaseMiniCluster that is started, a table with multiple 
regions is created. And the TableInputFormat is used to extract the rows again. 
By setting the paralellism to 1 the same TableInputFormat instance is used for 
multiple regions and succeeds (the problem this all started with).

Please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2391: [FLINK-3580] [table] Implement FLOOR/CEIL for time points

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2391
  
Thanks @fhueske. I will address your comments and merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3580) Reintroduce Date/Time and implement scalar functions for it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439108#comment-15439108
 ] 

ASF GitHub Bot commented on FLINK-3580:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2391
  
Thanks @fhueske. I will address your comments and merge this.


> Reintroduce Date/Time and implement scalar functions for it
> ---
>
> Key: FLINK-3580
> URL: https://issues.apache.org/jira/browse/FLINK-3580
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> This task includes:
> {code}
> DATETIME_PLUS
> EXTRACT_DATE
> FLOOR
> CEIL
> CURRENT_TIME
> CURRENT_TIMESTAMP
> LOCALTIME
> LOCALTIMESTAMP
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2374: [FLINK-3950] Add Meter Metric Type

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76410073
  
--- Diff: docs/monitoring/metrics.md ---
@@ -155,6 +155,55 @@ public class MyMapper extends RichMapFunction {
 }
 {% endhighlight %}
 
+ Meter
+
+A `Meter` measures an average throughput. An occurrence of an event can be 
registered with the `markEvent()` method. Occurrence of multiple events at the 
same time can be registered with `markEvent(long n)` method.
+You can register a meter by calling `meter(String name, Meter histogram)` 
on a `MetricGroup`.
+
+{% highlight java %}
+public class MyMapper extends RichMapFunction {
+  private Meter meter;
+
+  @Override
+  public void open(Configuration config) {
+this.meter = getRuntimeContext()
+  .getMetricGroup()
+  .meter("myMeter", new MyMeter());
+  }
+
+  @public Integer map(Long value) throws Exception {
+this.meter.markEvent();
+  }
+}
+{% endhighlight %}
+
+Flink offers a {% gh_link 
flink-metrics/flink-metrics-dropwizard/src/main/java/org/apache/flink/dropwizard/metrics/DropwizardMeterWrapper.java
 "Wrapper" %} that allows usage of Codahale/DropWizard meters.
+To use this wrapper add the following dependency in your `pom.xml`:
+{% highlight xml %}
+
+  org.apache.flink
+  flink-metrics-dropwizard
+  {{site.version}}
+
+{% endhighlight %}
+
+You can then register a Codahale/DropWizard meter like this:
+
+{% highlight java %}
+public class MyMapper extends RichMapFunction {
+  private Meter meter;
+
+  @Override
+  public void open(Configuration config) {
+com.codahale.metrics.Meter meter = new com.codahale.metrics.Meter();
+
+this.meter = getRuntimeContext()
+  .getMetricGroup()
+  .histogram("myMeter", new DropWizardMeterWrapper(meter));
--- End diff --

`.histogram` -> `.meter`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3950) Add Meter Metric Type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438868#comment-15438868
 ] 

ASF GitHub Bot commented on FLINK-3950:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2374#discussion_r76410073
  
--- Diff: docs/monitoring/metrics.md ---
@@ -155,6 +155,55 @@ public class MyMapper extends RichMapFunction {
 }
 {% endhighlight %}
 
+ Meter
+
+A `Meter` measures an average throughput. An occurrence of an event can be 
registered with the `markEvent()` method. Occurrence of multiple events at the 
same time can be registered with `markEvent(long n)` method.
+You can register a meter by calling `meter(String name, Meter histogram)` 
on a `MetricGroup`.
+
+{% highlight java %}
+public class MyMapper extends RichMapFunction {
+  private Meter meter;
+
+  @Override
+  public void open(Configuration config) {
+this.meter = getRuntimeContext()
+  .getMetricGroup()
+  .meter("myMeter", new MyMeter());
+  }
+
+  @public Integer map(Long value) throws Exception {
+this.meter.markEvent();
+  }
+}
+{% endhighlight %}
+
+Flink offers a {% gh_link 
flink-metrics/flink-metrics-dropwizard/src/main/java/org/apache/flink/dropwizard/metrics/DropwizardMeterWrapper.java
 "Wrapper" %} that allows usage of Codahale/DropWizard meters.
+To use this wrapper add the following dependency in your `pom.xml`:
+{% highlight xml %}
+
+  org.apache.flink
+  flink-metrics-dropwizard
+  {{site.version}}
+
+{% endhighlight %}
+
+You can then register a Codahale/DropWizard meter like this:
+
+{% highlight java %}
+public class MyMapper extends RichMapFunction {
+  private Meter meter;
+
+  @Override
+  public void open(Configuration config) {
+com.codahale.metrics.Meter meter = new com.codahale.metrics.Meter();
+
+this.meter = getRuntimeContext()
+  .getMetricGroup()
+  .histogram("myMeter", new DropWizardMeterWrapper(meter));
--- End diff --

`.histogram` -> `.meter`


> Add Meter Metric Type
> -
>
> Key: FLINK-3950
> URL: https://issues.apache.org/jira/browse/FLINK-3950
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Stephan Ewen
>Assignee: Ivan Mushketyk
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438685#comment-15438685
 ] 

Maximilian Michels commented on FLINK-4486:
---

The {{YarnClusterClient}} delays checking for the startup of the cluster until 
actual client-side operations are made (e.g job submission). The yarn-session 
checks for cluster startup when you execute it without the detached {{-d}} flag 
but not in detached mode because there is no cluster interaction afterwards. We 
should fix that.

> JobManager not fully running when yarn-session.sh finishes
> --
>
> Key: FLINK-4486
> URL: https://issues.apache.org/jira/browse/FLINK-4486
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> I start a detached yarn-session.sh.
> If the Yarn cluster is very busy then the yarn-session.sh script completes 
> BEFORE all the task slots have been allocated. As a consequence I sometimes 
> have a jobmanager without any task slots. Over time these task slots are 
> assigned by the Yarn cluster but these are not available for the first job 
> that is submitted.
> As a consequence I have found that the first few tasks in my job fail with 
> this error "Not enough free slots available to run the job.".
> I think the desirable behavior is that yarn-session waits until the 
> jobmanager is fully functional and capable of actually running the jobs.
> {code}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
> Not enough free slots available to run the job. You can decrease the operator 
> parallelism or increase the number of slots per TaskManager in the 
> configuration. Task to schedule: < Attempt #0 (CHAIN DataSource (Read prefix 
> '4') -> Map (Map prefix '4') (8/10)) @ (unassigned) - [SCHEDULED] > with 
> groupID < cd6c37df290564e603da908a8783a9bf > in sharing group < 
> SlotSharingGroup [c0b6eff6ce93967182cdb6dfeae9359b, 
> 8b2c3b39f3a55adf9f123243ab03c9c1, 55fb94dd8a3e5f59a10dbbf5c4925db4, 
> 433b2e4a05a5e685b48c517249755a89, 8c74690c35454064e4815ac3756cdca2, 
> 4b4fbd24f3483030fd852b38ff2249c1, 5e36a56ea4dece18fe5ba04352d90dc8, 
> cd6c37df290564e603da908a8783a9bf, 64eafa845087bee70735f7250df9994f, 
> 706a5d6fe48ae57724a00a9fce5dae8a, 7bee4297e0e839e53a153dfcbcca8624, 
> 21b58f7d408d237540ae7b4734f81a1d, b429b1ff338d9d73677f42717cfc0dbc, 
> cc7491db641f557c6aa8c749ebc2de62, f61cbf0ae00331f67aaf60ace78b05aa, 
> 606f02ea9e0f4ad57f0cc0232dd70842] >. Resources available to scheduler: Number 
> of instances=1, total number of slots=7, available slots=0
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:256)
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:306)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:454)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:326)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:734)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1332)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2109: [FLINK-3677] FileInputFormat: Allow to specify include/ex...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2109
  
Hi,
I just saw that this was merged but I still have some comments. What is the 
reason for the added Guava (test)-dependency? (Probably a leftover but should 
nevertheless not be in this) And what's the reason for the new `readFile` 
method on `StreamExecutionEnvironment` that is marked as both `@Deprecated` and 
`@PublicEvolving` and provides no functionality except setting the input path 
filter on the provided file input format? I think this should also not be in 
there. It's maybe also a leftover?

Sorry for not noticing this earlier, I'm just trying to keep the API 
somewhat clean/consistent. 😅 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438684#comment-15438684
 ] 

ASF GitHub Bot commented on FLINK-3677:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2109
  
Hi,
I just saw that this was merged but I still have some comments. What is the 
reason for the added Guava (test)-dependency? (Probably a leftover but should 
nevertheless not be in this) And what's the reason for the new `readFile` 
method on `StreamExecutionEnvironment` that is marked as both `@Deprecated` and 
`@PublicEvolving` and provides no functionality except setting the input path 
filter on the provided file input format? I think this should also not be in 
there. It's maybe also a leftover?

Sorry for not noticing this earlier, I'm just trying to keep the API 
somewhat clean/consistent.  


> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4499) Introduce findbugs maven plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438683#comment-15438683
 ] 

ASF GitHub Bot commented on FLINK-4499:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2422
  
Thanks for adding this.

I think that pure warning by the plugin will not be read and respected 
really. So it makes only sense when a "bug" detected by the plugin fails the 
build.

How about starting with a low set of settings and configuring it to fail 
when detecting errors? We can then gradually activate more and more visitors.

Also, do you have any experience in how finbugs usually affects the build 
times?


> Introduce findbugs maven plugin
> ---
>
> Key: FLINK-4499
> URL: https://issues.apache.org/jira/browse/FLINK-4499
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> As suggested by Stephan in FLINK-4482, this issue is to add 
> findbugs-maven-plugin into the build process so that we can detect lack of 
> proper locking and other defects automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438697#comment-15438697
 ] 

Stephan Ewen commented on FLINK-3677:
-

I agree with Aljoscha.

I am against adding that new method the {{StreamExecutionEnvironment}}. Next to 
{{DataStream}} that is the most sensitive part in terms of API maintenance and 
very susceptible to API clutter. New methods should have reasonable orthogonal 
functionality on top existing methods.

We also spent a lot of time getting rid of Guava wherever possible, so I am 
quite against adding this again. Adding a 5MB library simply to write 
{{Lists.newArrayList(a, b)}} instead of {{Arrays.asList(a, b)}} seems like not 
a good idea.

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438733#comment-15438733
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @StephanEwen , comments addressed.  

I'm not sure whether the comments are correct. Let me know if I'm wrong.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438634#comment-15438634
 ] 

ASF GitHub Bot commented on FLINK-3874:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2244


> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4486:
--
Fix Version/s: 1.1.2
   1.2.0

> JobManager not fully running when yarn-session.sh finishes
> --
>
> Key: FLINK-4486
> URL: https://issues.apache.org/jira/browse/FLINK-4486
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
> Fix For: 1.2.0, 1.1.2
>
>
> I start a detached yarn-session.sh.
> If the Yarn cluster is very busy then the yarn-session.sh script completes 
> BEFORE all the task slots have been allocated. As a consequence I sometimes 
> have a jobmanager without any task slots. Over time these task slots are 
> assigned by the Yarn cluster but these are not available for the first job 
> that is submitted.
> As a consequence I have found that the first few tasks in my job fail with 
> this error "Not enough free slots available to run the job.".
> I think the desirable behavior is that yarn-session waits until the 
> jobmanager is fully functional and capable of actually running the jobs.
> {code}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
> Not enough free slots available to run the job. You can decrease the operator 
> parallelism or increase the number of slots per TaskManager in the 
> configuration. Task to schedule: < Attempt #0 (CHAIN DataSource (Read prefix 
> '4') -> Map (Map prefix '4') (8/10)) @ (unassigned) - [SCHEDULED] > with 
> groupID < cd6c37df290564e603da908a8783a9bf > in sharing group < 
> SlotSharingGroup [c0b6eff6ce93967182cdb6dfeae9359b, 
> 8b2c3b39f3a55adf9f123243ab03c9c1, 55fb94dd8a3e5f59a10dbbf5c4925db4, 
> 433b2e4a05a5e685b48c517249755a89, 8c74690c35454064e4815ac3756cdca2, 
> 4b4fbd24f3483030fd852b38ff2249c1, 5e36a56ea4dece18fe5ba04352d90dc8, 
> cd6c37df290564e603da908a8783a9bf, 64eafa845087bee70735f7250df9994f, 
> 706a5d6fe48ae57724a00a9fce5dae8a, 7bee4297e0e839e53a153dfcbcca8624, 
> 21b58f7d408d237540ae7b4734f81a1d, b429b1ff338d9d73677f42717cfc0dbc, 
> cc7491db641f557c6aa8c749ebc2de62, f61cbf0ae00331f67aaf60ace78b05aa, 
> 606f02ea9e0f4ad57f0cc0232dd70842] >. Resources available to scheduler: Number 
> of instances=1, total number of slots=7, available slots=0
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:256)
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:306)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:454)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:326)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:734)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1332)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #:

[mushketyk]

Github user mushketyk commented on the pull request:


https://github.com/apache/flink/commit/abb4496781883937a935113c1e33ae1174aafa73#commitcomment-18789946
  
Hi Stephan.

I think you are right.
I'll update the testing as soon as possible and create a different PR to 
address your comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned FLINK-4486:
-

Assignee: Maximilian Michels

> JobManager not fully running when yarn-session.sh finishes
> --
>
> Key: FLINK-4486
> URL: https://issues.apache.org/jira/browse/FLINK-4486
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>
> I start a detached yarn-session.sh.
> If the Yarn cluster is very busy then the yarn-session.sh script completes 
> BEFORE all the task slots have been allocated. As a consequence I sometimes 
> have a jobmanager without any task slots. Over time these task slots are 
> assigned by the Yarn cluster but these are not available for the first job 
> that is submitted.
> As a consequence I have found that the first few tasks in my job fail with 
> this error "Not enough free slots available to run the job.".
> I think the desirable behavior is that yarn-session waits until the 
> jobmanager is fully functional and capable of actually running the jobs.
> {code}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
> Not enough free slots available to run the job. You can decrease the operator 
> parallelism or increase the number of slots per TaskManager in the 
> configuration. Task to schedule: < Attempt #0 (CHAIN DataSource (Read prefix 
> '4') -> Map (Map prefix '4') (8/10)) @ (unassigned) - [SCHEDULED] > with 
> groupID < cd6c37df290564e603da908a8783a9bf > in sharing group < 
> SlotSharingGroup [c0b6eff6ce93967182cdb6dfeae9359b, 
> 8b2c3b39f3a55adf9f123243ab03c9c1, 55fb94dd8a3e5f59a10dbbf5c4925db4, 
> 433b2e4a05a5e685b48c517249755a89, 8c74690c35454064e4815ac3756cdca2, 
> 4b4fbd24f3483030fd852b38ff2249c1, 5e36a56ea4dece18fe5ba04352d90dc8, 
> cd6c37df290564e603da908a8783a9bf, 64eafa845087bee70735f7250df9994f, 
> 706a5d6fe48ae57724a00a9fce5dae8a, 7bee4297e0e839e53a153dfcbcca8624, 
> 21b58f7d408d237540ae7b4734f81a1d, b429b1ff338d9d73677f42717cfc0dbc, 
> cc7491db641f557c6aa8c749ebc2de62, f61cbf0ae00331f67aaf60ace78b05aa, 
> 606f02ea9e0f4ad57f0cc0232dd70842] >. Resources available to scheduler: Number 
> of instances=1, total number of slots=7, available slots=0
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:256)
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:306)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:454)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:326)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:734)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1332)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4494) Expose the TimeServiceProvider from the Task to each Operator.

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438695#comment-15438695
 ] 

Aljoscha Krettek commented on FLINK-4494:
-

+1 that would be a very good change

> Expose the TimeServiceProvider from the Task to each Operator.
> --
>
> Key: FLINK-4494
> URL: https://issues.apache.org/jira/browse/FLINK-4494
> Project: Flink
>  Issue Type: Bug
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>
> This change aims at simplifying the {{StreamTask}} class by exposing directly 
> the {{TimeServiceProvider}} to the operators being executed. 
> This implies removing the {{registerTimer()}} and 
> {{getCurrentProcessingTime()}} methods from the {{StreamTask}}. Now, to 
> register a timer and query the time, each operator will be able to get the 
> {{TimeServiceProvider}} and call the corresponding methods directly on it.
> In addition, this will simplify many of the tests which now implement their 
> own time providers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2269: [FLINK-4190] Generalise RollingSink to work with arbitrar...

[joshfg]

Github user joshfg commented on the issue:

https://github.com/apache/flink/pull/2269
  
Hi Aljoscha, just wanted to remind you about this - any idea when the 
changes will be merged in? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438661#comment-15438661
 ] 

Aljoscha Krettek commented on FLINK-4486:
-

Or that a {{JobManager}} takes the job and blocks and waits until it gets the 
required resources from Yarn. I think this is already being addressed by the 
work on FLIP-6, if I'm not mistaken.

Could [~StephanEwen] or [~trohrm...@apache.org] chime in on this?

> JobManager not fully running when yarn-session.sh finishes
> --
>
> Key: FLINK-4486
> URL: https://issues.apache.org/jira/browse/FLINK-4486
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> I start a detached yarn-session.sh.
> If the Yarn cluster is very busy then the yarn-session.sh script completes 
> BEFORE all the task slots have been allocated. As a consequence I sometimes 
> have a jobmanager without any task slots. Over time these task slots are 
> assigned by the Yarn cluster but these are not available for the first job 
> that is submitted.
> As a consequence I have found that the first few tasks in my job fail with 
> this error "Not enough free slots available to run the job.".
> I think the desirable behavior is that yarn-session waits until the 
> jobmanager is fully functional and capable of actually running the jobs.
> {code}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
> Not enough free slots available to run the job. You can decrease the operator 
> parallelism or increase the number of slots per TaskManager in the 
> configuration. Task to schedule: < Attempt #0 (CHAIN DataSource (Read prefix 
> '4') -> Map (Map prefix '4') (8/10)) @ (unassigned) - [SCHEDULED] > with 
> groupID < cd6c37df290564e603da908a8783a9bf > in sharing group < 
> SlotSharingGroup [c0b6eff6ce93967182cdb6dfeae9359b, 
> 8b2c3b39f3a55adf9f123243ab03c9c1, 55fb94dd8a3e5f59a10dbbf5c4925db4, 
> 433b2e4a05a5e685b48c517249755a89, 8c74690c35454064e4815ac3756cdca2, 
> 4b4fbd24f3483030fd852b38ff2249c1, 5e36a56ea4dece18fe5ba04352d90dc8, 
> cd6c37df290564e603da908a8783a9bf, 64eafa845087bee70735f7250df9994f, 
> 706a5d6fe48ae57724a00a9fce5dae8a, 7bee4297e0e839e53a153dfcbcca8624, 
> 21b58f7d408d237540ae7b4734f81a1d, b429b1ff338d9d73677f42717cfc0dbc, 
> cc7491db641f557c6aa8c749ebc2de62, f61cbf0ae00331f67aaf60ace78b05aa, 
> 606f02ea9e0f4ad57f0cc0232dd70842] >. Resources available to scheduler: Number 
> of instances=1, total number of slots=7, available slots=0
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:256)
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:306)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:454)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:326)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:734)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1332)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438680#comment-15438680
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
The methods look good. They need a `@PublicEvolving` annotation and I think 
we should mark them with `@Deprecated` and comment that they are a temporary 
workaround while the `apply()` method has the wrong return type.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2305: [FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and ...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
The methods look good. They need a `@PublicEvolving` annotation and I think 
we should mark them with `@Deprecated` and comment that they are a temporary 
workaround while the `apply()` method has the wrong return type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4363) Implement TaskManager basic startup of all components in java

[Zhijiang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438745#comment-15438745
 ] 

Zhijiang Wang commented on FLINK-4363:
--

Hi [~mxm], I have created the FLINK-4505 for TaskExecutorFactory issue. As you 
mentioned, it should be an abstract class and provide an abstract method maybe 
called 'createAndStartTaskExecutor()'. There may be at least three different 
specific factories(testing, yarn, standalone) extend TaskExecutorFactory to 
implement the method 'createAndStartTaskExecutor'. The parameters in 
constructor of specific factories are different based on different modes. For 
example: for StandaloneTaskExecutorFactory, the constructor parameter should be 
(Configuration configuration,ResourceID resourceID,
RpcService rpcService,String taskManagerHostname,HighAvailabilityServices 
haServices,boolean localTaskManagerCommunication), and in the 
'createAndStartTaskExecutor()' method it can invoke 
‘startTaskManagerComponentsAndActor' method in {{TaskExecutor}} to bring up 
{{TaskExecutor}}. Do you have any other advices, then I can start this subtask 
later.
https://issues.apache.org/jira/browse/FLINK-4505

> Implement TaskManager basic startup of all components in java
> -
>
> Key: FLINK-4363
> URL: https://issues.apache.org/jira/browse/FLINK-4363
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> Similar with current {{TaskManager}},but implement initialization and startup 
> all components in java instead of scala.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4502) Cassandra connector documentation has misleading consistency guarantees

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-4502:

Component/s: Cassandra Connector

> Cassandra connector documentation has misleading consistency guarantees
> ---
>
> Key: FLINK-4502
> URL: https://issues.apache.org/jira/browse/FLINK-4502
> Project: Flink
>  Issue Type: Bug
>  Components: Cassandra Connector, Documentation
>Affects Versions: 1.1.0
>Reporter: Elias Levy
>
> The Cassandra connector documentation states that  "enableWriteAheadLog() is 
> an optional method, that allows exactly-once processing for non-deterministic 
> algorithms."  This claim appears to be false.
> From what I gather, the write ahead log feature of the connector works as 
> follows:
> - The sink is replaced with a stateful operator that writes incoming messages 
> to the state backend based on checkpoint they belong in.
> - When the operator is notified that a Flink checkpoint has been completed 
> it, for each set of checkpoints older than and including the committed one:
>   * reads its messages from the state backend
>   * writes them to Cassandra
>   * records that it has committed them to Cassandra for the specific 
> checkpoint and operator instance
>* and erases them from the state backend.
> This process attempts to avoid resubmitting queries to Cassandra that would 
> otherwise occur when recovering a job from a checkpoint and having messages 
> replayed.
> Alas, this does not guarantee exactly once semantics at the sink.  The writes 
> to Cassandra that occur when the operator is notified that checkpoint is 
> completed are not atomic and they are potentially non-idempotent.  If the job 
> dies while writing to Cassandra or before committing the checkpoint via 
> committer, queries will be replayed when the job recovers.  Thus the 
> documentation appear to be incorrect in stating this provides exactly-once 
> semantics.
> There also seems to be an issue in GenericWriteAheadSink's 
> notifyOfCompletedCheckpoint which may result in incorrect output.  If 
> sendValues returns false because a write failed, instead of bailing, it 
> simply moves on to the next checkpoint to commit if there is one, keeping the 
> previous one around to try again later.  But that can result in newer data 
> being overwritten with older data when the previous checkpoint is retried.  
> Although given that CassandraCommitter implements isCheckpointCommitted as 
> checkpointID <= this.lastCommittedCheckpointID, it actually means that when 
> it goes back to try the uncommitted older checkpoint it will consider it 
> committed, even though some of its data may not have been written out, and 
> the data will be discarded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4498) Better Cassandra sink documentation

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-4498:

Component/s: Cassandra Connector

> Better Cassandra sink documentation
> ---
>
> Key: FLINK-4498
> URL: https://issues.apache.org/jira/browse/FLINK-4498
> Project: Flink
>  Issue Type: Improvement
>  Components: Cassandra Connector, Documentation
>Affects Versions: 1.1.0
>Reporter: Elias Levy
>
> The Cassandra sink documentation is somewhat muddled and could be improved.  
> For instance, the fact that is only supports tuples and POJO's that use 
> DataStax Mapper annotations is only mentioned in passing, and it is not clear 
> that the reference to tuples only applies to Flink Java tuples and not Scala 
> tuples.  
> The documentation also does not mention that setQuery() is only necessary for 
> tuple streams. 
> The explanation of the write ahead log could use some cleaning up to clarify 
> when it is appropriate to use, ideally with an example.  Maybe this would be 
> best as a blog post to expand on the type of non-deterministic streams this 
> applies to.
> It would also be useful to mention that tuple elements will be mapped to 
> Cassandra columns using the Datastax Java driver's default encoders, which 
> are somewhat limited (e.g. to write to a blob column the type in the tuple 
> must be a java.nio.ByteBuffer and not just a byte[]).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3318) Add support for quantifiers to CEP's pattern API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438769#comment-15438769
 ] 

ASF GitHub Bot commented on FLINK-3318:
---

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2361
  
Could someone please take a look at this PR? It has been here without a 
review for more than 2 weeks.


> Add support for quantifiers to CEP's pattern API
> 
>
> Key: FLINK-3318
> URL: https://issues.apache.org/jira/browse/FLINK-3318
> Project: Flink
>  Issue Type: Improvement
>  Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> It would be a good addition to extend the pattern API to support quantifiers 
> known from regular expressions (e.g. Kleene star, ?, +, or count bounds). 
> This would considerably enrich the set of supported patterns.
> Implementing the count bounds could be done by unrolling the pattern state. 
> In order to support the Kleene star operator, the {{NFACompiler}} has to be 
> extended to insert epsilon-transition between a Kleene start state and the 
> succeeding pattern state. In order to support {{?}}, one could insert two 
> paths from the preceding state, one which accepts the event and another which 
> directly goes into the next pattern state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4507) Deprecate savepoint backend config

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-4507:
--

 Summary: Deprecate savepoint backend config
 Key: FLINK-4507
 URL: https://issues.apache.org/jira/browse/FLINK-4507
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi


The savepoint backend configuration allows both {{jobmanager}} and 
{{filesystem}} as values. The {{jobmanager}} variant is used as default if 
nothing is configured.

As part of FLIP-10, we want to get rid of this distinction and make all 
savepoints go to a file. Savepoints backed by JobManagers are only relevant for 
testing. Users could only recover from them if they did not shut down the 
current cluster.

Deprecate the {{savepoints.state.backend.fs.dir}} and add 
{{state.savepoints.dir}} as new config key. This is used as the default 
savepoint directory. Users can overwrite this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2244: [FLINK-3874] Add a Kafka TableSink with JSON seria...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2244


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #:

[StephanEwen]

Github user StephanEwen commented on the pull request:


https://github.com/apache/flink/commit/4810910431e01bf143ae77a6e93a86f2fafbccd0#commitcomment-18789532
  
There is some followup discussion on this in the JIRA issue: 
https://issues.apache.org/jira/browse/FLINK-3677


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438710#comment-15438710
 ] 

Stephan Ewen commented on FLINK-3874:
-

I think this needs some improvement. I also commented on the issue:


Do we need tests here that fire up Kafka clusters? Those are very time 
intensive, they are tough to harden so that the build is reliable, and what do 
they really add here?
Assuming the the Kafka connector works, this should test the "add on" by the 
TableSource. That can be done with a mock FlinkKafkaConsumer.

I think it would be very good to update this.

> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3874) Add a Kafka TableSink with JSON serialization

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438710#comment-15438710
 ] 

Stephan Ewen edited comment on FLINK-3874 at 8/26/16 9:25 AM:
--

I think this needs some improvement. I also commented on the commit:


Do we need tests here that fire up Kafka clusters? Those are very time 
intensive, they are tough to harden so that the build is reliable, and what do 
they really add here?
Assuming the the Kafka connector works, this should test the "add on" by the 
TableSource. That can be done with a mock FlinkKafkaConsumer.

I think it would be very good to update this.


was (Author: stephanewen):
I think this needs some improvement. I also commented on the issue:


Do we need tests here that fire up Kafka clusters? Those are very time 
intensive, they are tough to harden so that the build is reliable, and what do 
they really add here?
Assuming the the Kafka connector works, this should test the "add on" by the 
TableSource. That can be done with a mock FlinkKafkaConsumer.

I think it would be very good to update this.

> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #:

[StephanEwen]

Github user StephanEwen commented on the pull request:


https://github.com/apache/flink/commit/abb4496781883937a935113c1e33ae1174aafa73#commitcomment-18789807
  
Can I re-iterate on this issue?

Do we need tests here that fire up Kafka clusters? Those are very time 
intensive, they are tough to harden so that the build is reliable, and what do 
they really add here?
Assuming the the Kafka connector works, this should test the "add on" by 
the TableSource. That can be done with a mock FlinkKafkaConsumer.

I think it would be very good to update this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2305: [FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and ...

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @StephanEwen , comments addressed.  

I'm not sure whether the comments are correct. Let me know if I'm wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2423: [FLINK-4486] detached YarnSession: wait until clus...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2423

[FLINK-4486] detached YarnSession: wait until cluster startup is complete



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-4486

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2423


commit 2ec30c2c25204ff04270db9d072085f85909c8be
Author: Maximilian Michels 
Date:   2016-08-26T10:06:36Z

[FLINK-4486] detached YarnSession: wait until cluster startup is complete




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438759#comment-15438759
 ] 

ASF GitHub Bot commented on FLINK-4486:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2423

[FLINK-4486] detached YarnSession: wait until cluster startup is complete



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-4486

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2423


commit 2ec30c2c25204ff04270db9d072085f85909c8be
Author: Maximilian Michels 
Date:   2016-08-26T10:06:36Z

[FLINK-4486] detached YarnSession: wait until cluster startup is complete




> JobManager not fully running when yarn-session.sh finishes
> --
>
> Key: FLINK-4486
> URL: https://issues.apache.org/jira/browse/FLINK-4486
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
> Fix For: 1.2.0, 1.1.2
>
>
> I start a detached yarn-session.sh.
> If the Yarn cluster is very busy then the yarn-session.sh script completes 
> BEFORE all the task slots have been allocated. As a consequence I sometimes 
> have a jobmanager without any task slots. Over time these task slots are 
> assigned by the Yarn cluster but these are not available for the first job 
> that is submitted.
> As a consequence I have found that the first few tasks in my job fail with 
> this error "Not enough free slots available to run the job.".
> I think the desirable behavior is that yarn-session waits until the 
> jobmanager is fully functional and capable of actually running the jobs.
> {code}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
> Not enough free slots available to run the job. You can decrease the operator 
> parallelism or increase the number of slots per TaskManager in the 
> configuration. Task to schedule: < Attempt #0 (CHAIN DataSource (Read prefix 
> '4') -> Map (Map prefix '4') (8/10)) @ (unassigned) - [SCHEDULED] > with 
> groupID < cd6c37df290564e603da908a8783a9bf > in sharing group < 
> SlotSharingGroup [c0b6eff6ce93967182cdb6dfeae9359b, 
> 8b2c3b39f3a55adf9f123243ab03c9c1, 55fb94dd8a3e5f59a10dbbf5c4925db4, 
> 433b2e4a05a5e685b48c517249755a89, 8c74690c35454064e4815ac3756cdca2, 
> 4b4fbd24f3483030fd852b38ff2249c1, 5e36a56ea4dece18fe5ba04352d90dc8, 
> cd6c37df290564e603da908a8783a9bf, 64eafa845087bee70735f7250df9994f, 
> 706a5d6fe48ae57724a00a9fce5dae8a, 7bee4297e0e839e53a153dfcbcca8624, 
> 21b58f7d408d237540ae7b4734f81a1d, b429b1ff338d9d73677f42717cfc0dbc, 
> cc7491db641f557c6aa8c749ebc2de62, f61cbf0ae00331f67aaf60ace78b05aa, 
> 606f02ea9e0f4ad57f0cc0232dd70842] >. Resources available to scheduler: Number 
> of instances=1, total number of slots=7, available slots=0
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:256)
>   at 
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:306)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:454)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:326)
>   at 
> org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:734)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1332)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> 

[GitHub] flink issue #2361: [FLINK-3318][cep] Add support for quantifiers to CEP's pa...

[mushketyk]

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2361
  
Could someone please take a look at this PR? It has been here without a 
review for more than 2 weeks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4487) Need tools for managing the yarn-session better

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438690#comment-15438690
 ] 

Stephan Ewen commented on FLINK-4487:
-

I think that the last changes for cluster management by [~mxm] will probably 
address most of that.

> Need tools for managing the yarn-session better
> ---
>
> Key: FLINK-4487
> URL: https://issues.apache.org/jira/browse/FLINK-4487
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>
> There is a script yarn-session.sh which starts an empty JobManager on a Yarn 
> cluster. 
> Desired improvements:
> # If there is already a yarn-session running then yarn-session does not start 
> a new one (or it kills the old one?). Note that the file with ip/port may 
> exist yet the corresponding JobManager may have been killed in an other way.
> # A script that effectively lets me stop a yarn session and cleanup the file 
> that contains the ip/port of this yarn session and the .flink directory on 
> HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4505) Implement abstract TaskManagerFactory to bring up TaskManager for different modes

[Zhijiang Wang (JIRA)]

Zhijiang Wang created FLINK-4505:


 Summary: Implement abstract TaskManagerFactory to bring up 
TaskManager for different modes
 Key: FLINK-4505
 URL: https://issues.apache.org/jira/browse/FLINK-4505
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Reporter: Zhijiang Wang
Priority: Minor


Implement {{TaskExecutorFactory}} that should be an abstract class with the 
helper methods to bring up the {{TaskManager}}. The factory can be implemented 
by some classes to start a {{TaskManager}} in different modes (testing, 
standalone, yarn).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4505) Implement TaskManagerFactory to bring up TaskManager for different modes

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4505:
-
Summary: Implement TaskManagerFactory to bring up TaskManager for different 
modes  (was: Implement abstract TaskManagerFactory to bring up TaskManager for 
different modes)

> Implement TaskManagerFactory to bring up TaskManager for different modes
> 
>
> Key: FLINK-4505
> URL: https://issues.apache.org/jira/browse/FLINK-4505
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Priority: Minor
>
> Implement {{TaskExecutorFactory}} that should be an abstract class with the 
> helper methods to bring up the {{TaskManager}}. The factory can be 
> implemented by some classes to start a {{TaskManager}} in different modes 
> (testing, standalone, yarn).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4505) Implement TaskManagerFactory to bring up TaskManager for different modes

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang reassigned FLINK-4505:


Assignee: Zhijiang Wang

> Implement TaskManagerFactory to bring up TaskManager for different modes
> 
>
> Key: FLINK-4505
> URL: https://issues.apache.org/jira/browse/FLINK-4505
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>Priority: Minor
>
> Implement {{TaskExecutorFactory}} that should be an abstract class with the 
> helper methods to bring up the {{TaskManager}}. The factory can be 
> implemented by some classes to start a {{TaskManager}} in different modes 
> (testing, standalone, yarn).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4506) CsvOutputFormat defaults allowNullValues to false, even though doc and declaration says true

[Michael Wong (JIRA)]

Michael Wong created FLINK-4506:
---

 Summary: CsvOutputFormat defaults allowNullValues to false, even 
though doc and declaration says true
 Key: FLINK-4506
 URL: https://issues.apache.org/jira/browse/FLINK-4506
 Project: Flink
  Issue Type: Bug
  Components: Batch Connectors and Input/Output Formats, Documentation
Reporter: Michael Wong
Priority: Minor


In the constructor, it has this

{code}
this.allowNullValues = false;
{code}

But in the setAllowNullValues() method, the doc says the allowNullValues is 
true by default. Also, in the declaration of allowNullValues, the value is set 
to true. It probably makes the most sense to change the constructor.

{code}
/**
 * Configures the format to either allow null values (writing an empty 
field),
 * or to throw an exception when encountering a null field.
 * 
 * by default, null values are allowed.
 *
 * @param allowNulls Flag to indicate whether the output format should 
accept null values.
 */
public void setAllowNullValues(boolean allowNulls) {
this.allowNullValues = allowNulls;
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4511) Schedule periodic savepoints

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-4511:
--

 Summary: Schedule periodic savepoints
 Key: FLINK-4511
 URL: https://issues.apache.org/jira/browse/FLINK-4511
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Ufuk Celebi


Allow triggering of periodic savepoints, which are kept in a bounded queue 
(like completed checkpoints currently, but separate).

If there is no periodic checkpointing enabled, only periodic savepoints should 
be schedulded.

If periodic checkpointing is enabled, the periodic savepoints should not be 
scheduled independently, but instead the checkpoint scheduler should trigger a 
savepoint instead. This will ensure that no unexpected interference between 
checkpoints and savepoints happens. For this, I would restrict the savepoint 
interval to be a multiple of the checkpointing interval (if enabled).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4512) Add option for persistent checkpoints

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-4512:
--

 Summary: Add option for persistent checkpoints
 Key: FLINK-4512
 URL: https://issues.apache.org/jira/browse/FLINK-4512
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Ufuk Celebi


Allow periodic checkpoints to be persisted by writing out their meta data. This 
is what we currently do for savepoints, but in the future checkpoints and 
savepoints are likely to diverge with respect to guarantees they give for 
updatability, etc.

This means that the difference between persistent checkpoints and savepoints in 
the long term will be that persistent checkpoints can only be restored with the 
same job settings (like parallelism, etc.)

Regular and persisted checkpoints should behave differently with respect to 
disposal in *globally* terminal job states (FINISHED, CANCELLED, FAILED): 
regular checkpoints are cleaned up in all of these cases whereas persistent 
checkpoints only on FINISHED. Maybe with the option to customize behaviour on 
CANCELLED or FAILED.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4510) Always create CheckpointCoordinator

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen updated FLINK-4510:

Assignee: Jark Wu

> Always create CheckpointCoordinator
> ---
>
> Key: FLINK-4510
> URL: https://issues.apache.org/jira/browse/FLINK-4510
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>Assignee: Jark Wu
>
> The checkpoint coordinator is only created if a checkpointing interval is 
> configured. This means that no savepoints can be triggered if there is no 
> checkpointing interval specified.
> Instead we should always create it and allow an interval of 0 for disabled 
> periodic checkpoints. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439611#comment-15439611
 ] 

ASF GitHub Bot commented on FLINK-3930:
---

GitHub user vijikarthi opened a pull request:

https://github.com/apache/flink/pull/2425

FLINK-3930 Added shared secret based authorization for Flink service …

This PR addresses FLINK-3930 requirements. It enables shared secret based 
secure cookie authorization for the following components

- Akka layer
- Blob Service
- Web UI

Secure cookie authentication can be enabled by providing below 
configurations to Flink configuration file.

- `security.enabled`: A boolean value (true|false) indicating security is 
enabled or not.
- `security.cookie` : Secure cookie value to be used for authentication. 
For standalone deployment mode, the secure cookie value is mandatory when 
security is enabled but for the Yarn mode it is optional (auto-generated if not 
provided).

Alternatively, secure cookie value can be provided through Flink/Yarn CLI 
using "-k" or "--cookie" parameter option.

The web runtime module prompts for secure cookie using standard basic HTTP 
authentication mechanism, where the user id field is a noop and the password 
field will be used to capture the secure cookie. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vijikarthi/flink FLINK-3930

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2425


commit 33d391cb17e68dd203328a91fa6b63218884b49d
Author: Vijay Srinivasaraghavan 
Date:   2016-08-26T19:02:20Z

FLINK-3930 Added shared secret based authorization for Flink service 
components




> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Service-level authorization is the initial authorization mechanism to ensure 
> clients (or servers) connecting to the Flink cluster are authorized to do so. 
>   The purpose is to prevent a cluster from being used by an unauthorized 
> user, whether to execute jobs, disrupt cluster functionality, or gain access 
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4514) ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-4514:
---
Affects Version/s: 1.1.1
   1.1.0

> ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of 
> stream
> 
>
> Key: FLINK-4514
> URL: https://issues.apache.org/jira/browse/FLINK-4514
> Project: Flink
>  Issue Type: Bug
>  Components: Kinesis Connector
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.2
>
>
> Original mailing thread for the reported issue:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html
> Normally, the exception is thrown when the consumer uses the same shard 
> iterator after 5 minutes since it was retrieved. I've still yet to clarify & 
> reproduce the root cause of the {{ExpiredIteratorException}}, because from 
> the code this seems to be impossible. I'm leaning towards suspecting this is 
> a Kinesis-side issue (from the description in the ML, the behaviour also 
> seems indeterminate).
> Either way, the exception can be fairly easily handled so that the consumer 
> doesn't just fail. When caught, we request a new shard iterator from Kinesis 
> with the last sequence number.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2418: [FLINK-4245] JMXReporter exposes all defined variables

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2418
  
The funky thing is that internally the ObjectName reconverts the Hashtable 
to a Map...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2425: FLINK-3930 Added shared secret based authorization...

[vijikarthi]

GitHub user vijikarthi opened a pull request:

https://github.com/apache/flink/pull/2425

FLINK-3930 Added shared secret based authorization for Flink service …

This PR addresses FLINK-3930 requirements. It enables shared secret based 
secure cookie authorization for the following components

- Akka layer
- Blob Service
- Web UI

Secure cookie authentication can be enabled by providing below 
configurations to Flink configuration file.

- `security.enabled`: A boolean value (true|false) indicating security is 
enabled or not.
- `security.cookie` : Secure cookie value to be used for authentication. 
For standalone deployment mode, the secure cookie value is mandatory when 
security is enabled but for the Yarn mode it is optional (auto-generated if not 
provided).

Alternatively, secure cookie value can be provided through Flink/Yarn CLI 
using "-k" or "--cookie" parameter option.

The web runtime module prompts for secure cookie using standard basic HTTP 
authentication mechanism, where the user id field is a noop and the password 
field will be used to capture the secure cookie. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vijikarthi/flink FLINK-3930

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2425


commit 33d391cb17e68dd203328a91fa6b63218884b49d
Author: Vijay Srinivasaraghavan 
Date:   2016-08-26T19:02:20Z

FLINK-3930 Added shared secret based authorization for Flink service 
components




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4245) Metric naming improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439462#comment-15439462
 ] 

ASF GitHub Bot commented on FLINK-4245:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2418
  
The funky thing is that internally the ObjectName reconverts the Hashtable 
to a Map...


> Metric naming improvements
> --
>
> Key: FLINK-4245
> URL: https://issues.apache.org/jira/browse/FLINK-4245
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Reporter: Stephan Ewen
>
> A metric currently has two parts to it:
>   - The name of that particular metric
>   - The "scope" (or namespace), defined by the group that contains the metric.
> A metric group actually always implicitly has a map of naming "tags", like:
>   - taskmanager_host : 
>   - taskmanager_id : 
>   - task_name : "map() -> filter()"
> We derive the scope from that map, following the defined scope formats.
> For JMX (and some users that use JMX), it would be natural to expose that map 
> of tags. Some users reconstruct that map by parsing the metric scope. JMX, we 
> can expose a metric like:
>   - domain: "taskmanager.task.operator.io"
>   - name: "numRecordsIn"
>   - tags: { "hostname" -> "localhost", "operator_name" -> "map() at 
> X.java:123", ... }
> For many other reporters, the formatted scope makes a lot of sense, since 
> they think only in terms of (scope, metric-name).
> We may even have the formatted scope in JMX as well (in the domain), if we 
> want to go that route. 
> [~jgrier] and [~Zentol] - what do you think about that?
> [~mdaxini] Does that match your use of the metrics?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4508) Consolidate DummyEnvironment and MockEnvironment for Tests

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439165#comment-15439165
 ] 

Aljoscha Krettek commented on FLINK-4508:
-

Plus there's custom-mocked Environments in some tests.

And we should probably use the builder pattern for constructing the mock since 
there are so many parameters already.

> Consolidate DummyEnvironment and MockEnvironment for Tests
> --
>
> Key: FLINK-4508
> URL: https://issues.apache.org/jira/browse/FLINK-4508
> Project: Flink
>  Issue Type: Test
>Reporter: Stefan Richter
>Priority: Minor
>
> Currently we {{DummyEnvironment}} and {{MockEnvironment}} as implementations 
> of Environments for our test. Both serve a similar purpose, but offer 
> slightly different features. We should consolidate this by merging them into 
> one class that offers the best of both previous implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2424: [FLINK-4459][Scheduler] Introduce SlotProvider for...

[KurtYoung]

GitHub user KurtYoung opened a pull request:

https://github.com/apache/flink/pull/2424

[FLINK-4459][Scheduler] Introduce SlotProvider for Scheduler

Introduce SlotProvider, prepare for the further slot allocation refactoring

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KurtYoung/flink flink-4459

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2424


commit b88f024e6c6b134d00588af6aab8c03a189d2d3a
Author: Kurt Young 
Date:   2016-08-26T09:51:40Z

[FLINK-4459][Scheduler] Introduce SlotProvider for Scheduler




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4459) Introduce SlotProvider for Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439239#comment-15439239
 ] 

ASF GitHub Bot commented on FLINK-4459:
---

GitHub user KurtYoung opened a pull request:

https://github.com/apache/flink/pull/2424

[FLINK-4459][Scheduler] Introduce SlotProvider for Scheduler

Introduce SlotProvider, prepare for the further slot allocation refactoring

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KurtYoung/flink flink-4459

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2424


commit b88f024e6c6b134d00588af6aab8c03a189d2d3a
Author: Kurt Young 
Date:   2016-08-26T09:51:40Z

[FLINK-4459][Scheduler] Introduce SlotProvider for Scheduler




> Introduce SlotProvider for Scheduler
> 
>
> Key: FLINK-4459
> URL: https://issues.apache.org/jira/browse/FLINK-4459
> Project: Flink
>  Issue Type: Improvement
>  Components: Scheduler
>Reporter: Till Rohrmann
>Assignee: Kurt Young
>
> Currently the {{Scheduler}} maintains a queue of available instances which it 
> scans if it needs a new slot. If it finds a suitable instance (having free 
> slots available) it will allocate a slot from it. 
> This slot allocation logic can be factored out and be made available via a 
> {{SlotProvider}} interface. The {{SlotProvider}} has methods to allocate a 
> slot given a set of location preferences. Slots should be returned as 
> {{Futures}}, because in the future the slot allocation might happen 
> asynchronously (Flip-6). 
> In the first version, the {{SlotProvider}} implementation will simply 
> encapsulate the existing slot allocation logic extracted from the 
> {{Scheduler}}. When a slot is requested it will return a completed or failed 
> future since the allocation happens synchronously.
> The refactoring will have the advantage to simplify the {{Scheduler}} class 
> and to pave the way for upcoming refactorings (Flip-6).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2109: [FLINK-3677] FileInputFormat: Allow to specify include/ex...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2109
  
@mxm @mushketyk Technically this change does add a new method that is the 
now deprecated method minus the `FilePathFilter`. That's a good change, though, 
since it simplifies the API in the long run. So apologies for making a fuss 
about it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439277#comment-15439277
 ] 

ASF GitHub Bot commented on FLINK-3677:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2109
  
@mxm @mushketyk Technically this change does add a new method that is the 
now deprecated method minus the `FilePathFilter`. That's a good change, though, 
since it simplifies the API in the long run. So apologies for making a fuss 
about it. 


> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4510) Always create CheckpointCoordinator

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439292#comment-15439292
 ] 

Jark Wu commented on FLINK-4510:


Hi [~uce] , I would like to contribute this issue.

> Always create CheckpointCoordinator
> ---
>
> Key: FLINK-4510
> URL: https://issues.apache.org/jira/browse/FLINK-4510
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>
> The checkpoint coordinator is only created if a checkpointing interval is 
> configured. This means that no savepoints can be triggered if there is no 
> checkpointing interval specified.
> Instead we should always create it and allow an interval of 0 for disabled 
> periodic checkpoints. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2109: [FLINK-3677] FileInputFormat: Allow to specify include/ex...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2109
  
Ah, got it too now. The diff only makes it appear like the deprecated 
method is new. Then I take that back :-)
A followup on removing Guava would be nice, through...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2418: [FLINK-4245] JMXReporter exposes all defined variables

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2418
  
Looks good to me!

The only not-so-nice thing is that we have to convert the `Map` to a 
`HashTable` for every metric. Given how we tried to optimize the metric 
creation overhead, this it pretty tough.
Its seems very hard to "disguise" a map as a hashtable, though...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4245) Metric naming improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439348#comment-15439348
 ] 

ASF GitHub Bot commented on FLINK-4245:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2418
  
Looks good to me!

The only not-so-nice thing is that we have to convert the `Map` to a 
`HashTable` for every metric. Given how we tried to optimize the metric 
creation overhead, this it pretty tough.
Its seems very hard to "disguise" a map as a hashtable, though...


> Metric naming improvements
> --
>
> Key: FLINK-4245
> URL: https://issues.apache.org/jira/browse/FLINK-4245
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Reporter: Stephan Ewen
>
> A metric currently has two parts to it:
>   - The name of that particular metric
>   - The "scope" (or namespace), defined by the group that contains the metric.
> A metric group actually always implicitly has a map of naming "tags", like:
>   - taskmanager_host : 
>   - taskmanager_id : 
>   - task_name : "map() -> filter()"
> We derive the scope from that map, following the defined scope formats.
> For JMX (and some users that use JMX), it would be natural to expose that map 
> of tags. Some users reconstruct that map by parsing the metric scope. JMX, we 
> can expose a metric like:
>   - domain: "taskmanager.task.operator.io"
>   - name: "numRecordsIn"
>   - tags: { "hostname" -> "localhost", "operator_name" -> "map() at 
> X.java:123", ... }
> For many other reporters, the formatted scope makes a lot of sense, since 
> they think only in terms of (scope, metric-name).
> We may even have the formatted scope in JMX as well (in the domain), if we 
> want to go that route. 
> [~jgrier] and [~Zentol] - what do you think about that?
> [~mdaxini] Does that match your use of the metrics?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4347) Implement SlotManager core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439373#comment-15439373
 ] 

ASF GitHub Bot commented on FLINK-4347:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2388
  
Looks quite good all in all.

Be good to get this in, so that JobManager followup work can build on this.
I would take this, rebase it, and merge it. If I find some issues I would 
create followup issues.


> Implement SlotManager core
> --
>
> Key: FLINK-4347
> URL: https://issues.apache.org/jira/browse/FLINK-4347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> The slot manager is responsible to maintain the list of slot requests and 
> slot allocations. It allows to request slots from the registered 
> TaskExecutors and issues container allocation requests in case that there are 
> not enough available resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4502) Cassandra connector documentation has misleading consistency guarantees

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439445#comment-15439445
 ] 

Elias Levy commented on FLINK-4502:
---

Chesnay, thanks for correcting me about the overwriting issue.  I missed the 
implication of setting the timestamp on the queries.

I still think the documentation is not clear that idempotent queries are a 
prerequisite of exactly-once semantics when the WAL is enabled, as shown by the 
portion of the documentation I quoted.  The connector documentation page 
mentions idempotent once, in the paragraph previous to the one describing the 
WAL.

The WAL functionality description could also be clearer.  For instance, is says 
"The write-ahead log guarantees that the replayed checkpoint is identical to 
the first attempt."  But that does not seem accurate.  The WAL doesn't appear 
to guarantee that a replayed checkpoint is identical.  Rather, it guarantees 
that no output associated with a checkpoint is written to the sink until the 
checkpoint is complete, which avoids possibly incompatible duplicate outputs 
when data from non-completed checkpoints are replayed on recovery.


> Cassandra connector documentation has misleading consistency guarantees
> ---
>
> Key: FLINK-4502
> URL: https://issues.apache.org/jira/browse/FLINK-4502
> Project: Flink
>  Issue Type: Bug
>  Components: Cassandra Connector, Documentation
>Affects Versions: 1.1.0
>Reporter: Elias Levy
>
> The Cassandra connector documentation states that  "enableWriteAheadLog() is 
> an optional method, that allows exactly-once processing for non-deterministic 
> algorithms."  This claim appears to be false.
> From what I gather, the write ahead log feature of the connector works as 
> follows:
> - The sink is replaced with a stateful operator that writes incoming messages 
> to the state backend based on checkpoint they belong in.
> - When the operator is notified that a Flink checkpoint has been completed 
> it, for each set of checkpoints older than and including the committed one:
>   * reads its messages from the state backend
>   * writes them to Cassandra
>   * records that it has committed them to Cassandra for the specific 
> checkpoint and operator instance
>* and erases them from the state backend.
> This process attempts to avoid resubmitting queries to Cassandra that would 
> otherwise occur when recovering a job from a checkpoint and having messages 
> replayed.
> Alas, this does not guarantee exactly once semantics at the sink.  The writes 
> to Cassandra that occur when the operator is notified that checkpoint is 
> completed are not atomic and they are potentially non-idempotent.  If the job 
> dies while writing to Cassandra or before committing the checkpoint via 
> committer, queries will be replayed when the job recovers.  Thus the 
> documentation appear to be incorrect in stating this provides exactly-once 
> semantics.
> There also seems to be an issue in GenericWriteAheadSink's 
> notifyOfCompletedCheckpoint which may result in incorrect output.  If 
> sendValues returns false because a write failed, instead of bailing, it 
> simply moves on to the next checkpoint to commit if there is one, keeping the 
> previous one around to try again later.  But that can result in newer data 
> being overwritten with older data when the previous checkpoint is retried.  
> Although given that CassandraCommitter implements isCheckpointCommitted as 
> checkpointID <= this.lastCommittedCheckpointID, it actually means that when 
> it goes back to try the uncommitted older checkpoint it will consider it 
> committed, even though some of its data may not have been written out, and 
> the data will be discarded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3755) Introduce key groups for key-value state to support dynamic scaling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439372#comment-15439372
 ] 

ASF GitHub Bot commented on FLINK-3755:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2376
  
Very good work and very nice code!

Some comments after a joint review:

  - The most critical issue is that there should not be any blocking on 
async threads during task shutdown. This unnecessarily delays responses to 
canceling and redeployment.

  - At this point, the `KeyGroupAssigner` interface seems a bit useless, 
especially if it is not parametrized with variable key group mappings. For the 
sake of making this simpler and more efficient, one could just have a static 
method for that.

  - I would suggest to make the assumption that key groups are always used 
(they should be, even if their number is equal to the parallelism), and drop 
the checks for `numberOfKeyGroups > 0`, for example in the 
KeyGroupHashPartitioner.

  - A bit more difficult is what to assume as the default number of key 
groups. We thought about assuming a default of `128`. That has no overhead in 
state backends like RocksDB and also allows initial job deployments which did 
not think about properly configuring this to have some freedom to scale out. If 
the parallelism is >= 128, this should probably round to the next highest 
power-of-two.

  - There are some log statements which cause log flooding, like an INFO 
log statement for every checkpoint stream (factory) created.



> Introduce key groups for key-value state to support dynamic scaling
> ---
>
> Key: FLINK-3755
> URL: https://issues.apache.org/jira/browse/FLINK-3755
> Project: Flink
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> In order to support dynamic scaling, it is necessary to sub-partition the 
> key-value states of each operator. This sub-partitioning, which produces a 
> set of key groups, allows to easily scale in and out Flink jobs by simply 
> reassigning the different key groups to the new set of sub tasks. The idea of 
> key groups is described in this design document [1]. 
> [1] 
> https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2388: [FLINK-4347][cluster management] Implement SlotManager co...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2388
  
Looks quite good all in all.

Be good to get this in, so that JobManager followup work can build on this.
I would take this, rebase it, and merge it. If I find some issues I would 
create followup issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4488) Prevent cluster shutdown after job execution for non-detached jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439346#comment-15439346
 ] 

ASF GitHub Bot commented on FLINK-4488:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2419
  
Our currents tests make it difficult to test such behavior. Added a check 
to the `YarnTestBase`. Basically, I'm skipping the cluster shutdown to check if 
the JobManager is still alive and hasn't been shutdown through other means. 


> Prevent cluster shutdown after job execution for non-detached jobs
> --
>
> Key: FLINK-4488
> URL: https://issues.apache.org/jira/browse/FLINK-4488
> Project: Flink
>  Issue Type: Bug
>  Components: YARN Client
>Affects Versions: 1.2.0, 1.1.1
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.2.0, 1.1.2
>
>
> In per-job mode, the Yarn cluster currently shuts down after the first 
> interactively executed job. Users may want to execute multiple jobs in one 
> Jar. I would suggest to use this mechanism only for jobs which run detached. 
> For interactive jobs, shutdown of the cluster is additionally handled by the 
> CLI which should be sufficient to ensure cluster shutdown. Cluster shutdown 
> could only become a problem in case of a network partition to the cluster or 
> outage of the CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4245) Metric naming improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439359#comment-15439359
 ] 

ASF GitHub Bot commented on FLINK-4245:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2418
  
Actually, it may be possible, and not too bad if we assume the Hashtable is 
immutable. Something like this:

```java
public class HashtableWrapper extends Hashtable {

private final Map backingMap;
private final String name;

public HashtableWrapper(Map backingMap, String name) {
super(1);
this.backingMap = backingMap;
this.name = name;
}

@Override
public synchronized V get(Object key) {
if ("name".equals(key)) {
return name;
} else {
return backingMap.get(key);
}
}

@Override
public synchronized String put(String  key, String  value) {
throw new UnsupportedOperationException("immutable hashtable");
}

// wrappers for Iterator to Enumeration

@Override
public synchronized Enumeration keys() {
return new IteratorToEnumeration(backingMap.keySet());
}

@Override
public synchronized Enumeration elements() {
return new IteratorToEnumeration(backingMap.valueSet());
}

// and so on ...
}
```


> Metric naming improvements
> --
>
> Key: FLINK-4245
> URL: https://issues.apache.org/jira/browse/FLINK-4245
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Reporter: Stephan Ewen
>
> A metric currently has two parts to it:
>   - The name of that particular metric
>   - The "scope" (or namespace), defined by the group that contains the metric.
> A metric group actually always implicitly has a map of naming "tags", like:
>   - taskmanager_host : 
>   - taskmanager_id : 
>   - task_name : "map() -> filter()"
> We derive the scope from that map, following the defined scope formats.
> For JMX (and some users that use JMX), it would be natural to expose that map 
> of tags. Some users reconstruct that map by parsing the metric scope. JMX, we 
> can expose a metric like:
>   - domain: "taskmanager.task.operator.io"
>   - name: "numRecordsIn"
>   - tags: { "hostname" -> "localhost", "operator_name" -> "map() at 
> X.java:123", ... }
> For many other reporters, the formatted scope makes a lot of sense, since 
> they think only in terms of (scope, metric-name).
> We may even have the formatted scope in JMX as well (in the domain), if we 
> want to go that route. 
> [~jgrier] and [~Zentol] - what do you think about that?
> [~mdaxini] Does that match your use of the metrics?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2418: [FLINK-4245] JMXReporter exposes all defined variables

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2418
  
Actually, it may be possible, and not too bad if we assume the Hashtable is 
immutable. Something like this:

```java
public class HashtableWrapper extends Hashtable {

private final Map backingMap;
private final String name;

public HashtableWrapper(Map backingMap, String name) {
super(1);
this.backingMap = backingMap;
this.name = name;
}

@Override
public synchronized V get(Object key) {
if ("name".equals(key)) {
return name;
} else {
return backingMap.get(key);
}
}

@Override
public synchronized String put(String  key, String  value) {
throw new UnsupportedOperationException("immutable hashtable");
}

// wrappers for Iterator to Enumeration

@Override
public synchronized Enumeration keys() {
return new IteratorToEnumeration(backingMap.keySet());
}

@Override
public synchronized Enumeration elements() {
return new IteratorToEnumeration(backingMap.valueSet());
}

// and so on ...
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439364#comment-15439364
 ] 

ASF GitHub Bot commented on FLINK-3677:
---

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2109
  
@mxm @aljoscha @StephanEwen Thank you for your comments. Seems like this is 
sorted out :)
I'll remove Guava calls today at the evening.


> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Ivan Mushketyk
>Priority: Minor
>  Labels: starter
> Fix For: 1.2.0
>
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2109: [FLINK-3677] FileInputFormat: Allow to specify include/ex...

[mushketyk]

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2109
  
@mxm @aljoscha @StephanEwen Thank you for your comments. Seems like this is 
sorted out :)
I'll remove Guava calls today at the evening.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2376: [FLINK-3755] Introduce key groups for key-value state to ...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2376
  
Very good work and very nice code!

Some comments after a joint review:

  - The most critical issue is that there should not be any blocking on 
async threads during task shutdown. This unnecessarily delays responses to 
canceling and redeployment.

  - At this point, the `KeyGroupAssigner` interface seems a bit useless, 
especially if it is not parametrized with variable key group mappings. For the 
sake of making this simpler and more efficient, one could just have a static 
method for that.

  - I would suggest to make the assumption that key groups are always used 
(they should be, even if their number is equal to the parallelism), and drop 
the checks for `numberOfKeyGroups > 0`, for example in the 
KeyGroupHashPartitioner.

  - A bit more difficult is what to assume as the default number of key 
groups. We thought about assuming a default of `128`. That has no overhead in 
state backends like RocksDB and also allows initial job deployments which did 
not think about properly configuring this to have some freedom to scale out. If 
the parallelism is >= 128, this should probably round to the next highest 
power-of-two.

  - There are some log statements which cause log flooding, like an INFO 
log statement for every checkpoint stream (factory) created.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2383: [FLINK-4418] [client] Improve resilience when Inet...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2383


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4514) ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-4514:
--

 Summary: ExpiredIteratorException in Kinesis Consumer on long 
catch-ups to head of stream
 Key: FLINK-4514
 URL: https://issues.apache.org/jira/browse/FLINK-4514
 Project: Flink
  Issue Type: Bug
  Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
 Fix For: 1.2.0, 1.1.2


Original mailing thread for the reported issue:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html

Normally, the exception is thrown when the consumer uses the same shard 
iterator after 5 minutes since it was retrieved. I've still yet to clarify & 
reproduce the root cause of the {{ExpiredIteratorException}}, because from the 
code it seems to be impossible. I'm leaning towards suspecting this is a 
Kinesis-side issue (from the description in the ML, the behaviour also seems 
indeterminate).

Either way, the exception can be fairly easily handled so that the consumer 
doesn't just fail. When caught, we request a new shard iterator from Kinesis 
with the last sequence number.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4514) ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-4514:
---
Description: 
Original mailing thread for the reported issue:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html

Normally, the exception is thrown when the consumer uses the same shard 
iterator after 5 minutes since it was retrieved. I've still yet to clarify & 
reproduce the root cause of the {{ExpiredIteratorException}}, because from the 
code this seems to be impossible. I'm leaning towards suspecting this is a 
Kinesis-side issue (from the description in the ML, the behaviour also seems 
indeterminate).

Either way, the exception can be fairly easily handled so that the consumer 
doesn't just fail. When caught, we request a new shard iterator from Kinesis 
with the last sequence number.

  was:
Original mailing thread for the reported issue:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html

Normally, the exception is thrown when the consumer uses the same shard 
iterator after 5 minutes since it was retrieved. I've still yet to clarify & 
reproduce the root cause of the {{ExpiredIteratorException}}, because from the 
code it seems to be impossible. I'm leaning towards suspecting this is a 
Kinesis-side issue (from the description in the ML, the behaviour also seems 
indeterminate).

Either way, the exception can be fairly easily handled so that the consumer 
doesn't just fail. When caught, we request a new shard iterator from Kinesis 
with the last sequence number.


> ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of 
> stream
> 
>
> Key: FLINK-4514
> URL: https://issues.apache.org/jira/browse/FLINK-4514
> Project: Flink
>  Issue Type: Bug
>  Components: Kinesis Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.2
>
>
> Original mailing thread for the reported issue:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html
> Normally, the exception is thrown when the consumer uses the same shard 
> iterator after 5 minutes since it was retrieved. I've still yet to clarify & 
> reproduce the root cause of the {{ExpiredIteratorException}}, because from 
> the code this seems to be impossible. I'm leaning towards suspecting this is 
> a Kinesis-side issue (from the description in the ML, the behaviour also 
> seems indeterminate).
> Either way, the exception can be fairly easily handled so that the consumer 
> doesn't just fail. When caught, we request a new shard iterator from Kinesis 
> with the last sequence number.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2419: [FLINK-4488] only automatically shutdown clusters for det...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2419
  
Our currents tests make it difficult to test such behavior. Added a check 
to the `YarnTestBase`. Basically, I'm skipping the cluster shutdown to check if 
the JobManager is still alive and hasn't been shutdown through other means. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4418) ClusterClient/ConnectionUtils#findConnectingAddress fails immediately if InetAddress.getLocalHost throws exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439379#comment-15439379
 ] 

ASF GitHub Bot commented on FLINK-4418:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2383


> ClusterClient/ConnectionUtils#findConnectingAddress fails immediately if 
> InetAddress.getLocalHost throws exception
> --
>
> Key: FLINK-4418
> URL: https://issues.apache.org/jira/browse/FLINK-4418
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.1.0
>Reporter: Shannon Carey
>Assignee: Shannon Carey
>
> When attempting to connect to a cluster with a ClusterClient, if the 
> machine's hostname is not resolvable to an IP, an exception is thrown 
> preventing success.
> This is the case if, for example, the hostname is not present & mapped to a 
> local IP in /etc/hosts.
> The exception is below. I suggest that findAddressUsingStrategy() should 
> catch java.net.UnknownHostException thrown by InetAddress.getLocalHost() and 
> return null, allowing alternative strategies to be attempted by 
> findConnectingAddress(). I will open a PR to this effect. Ideally this could 
> be included in both 1.2 and 1.1.2.
> In the stack trace below, "ip-10-2-64-47" is the internal host name of an AWS 
> EC2 instance.
> {code}
> 21:11:35 org.apache.flink.client.program.ProgramInvocationException: Failed 
> to retrieve the JobManager gateway.
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient.runDetached(ClusterClient.java:430)
> 21:11:35  at 
> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:90)
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:389)
> 21:11:35  at 
> org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:75)
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:334)
> 21:11:35  at 
> com.expedia.www.flink.job.scheduler.FlinkJobSubmitter.get(FlinkJobSubmitter.java:81)
> 21:11:35  at 
> com.expedia.www.flink.job.scheduler.streaming.StreamingJobManager.run(StreamingJobManager.java:105)
> 21:11:35  at 
> com.expedia.www.flink.job.scheduler.JobScheduler.runStreamingApp(JobScheduler.java:69)
> 21:11:35  at 
> com.expedia.www.flink.job.scheduler.JobScheduler.main(JobScheduler.java:34)
> 21:11:35 Caused by: java.lang.RuntimeException: Failed to resolve JobManager 
> address at /10.2.89.80:43126
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient$LazyActorSystemLoader.get(ClusterClient.java:189)
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:649)
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient.runDetached(ClusterClient.java:428)
> 21:11:35  ... 8 more
> 21:11:35 Caused by: java.net.UnknownHostException: ip-10-2-64-47: 
> ip-10-2-64-47: unknown error
> 21:11:35  at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
> 21:11:35  at 
> org.apache.flink.runtime.net.ConnectionUtils.findAddressUsingStrategy(ConnectionUtils.java:232)
> 21:11:35  at 
> org.apache.flink.runtime.net.ConnectionUtils.findConnectingAddress(ConnectionUtils.java:123)
> 21:11:35  at 
> org.apache.flink.client.program.ClusterClient$LazyActorSystemLoader.get(ClusterClient.java:187)
> 21:11:35  ... 10 more
> 21:11:35 Caused by: java.net.UnknownHostException: ip-10-2-64-47: unknown 
> error
> 21:11:35  at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> 21:11:35  at 
> java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> 21:11:35  at 
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> 21:11:35  at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
> 21:11:35  ... 13 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2416: FLINK-4480: Incorrect link to elastic.co in docume...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2416


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4480) Incorrect link to elastic.co in documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439380#comment-15439380
 ] 

ASF GitHub Bot commented on FLINK-4480:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2416


> Incorrect link to elastic.co in documentation
> -
>
> Key: FLINK-4480
> URL: https://issues.apache.org/jira/browse/FLINK-4480
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.1.1
>Reporter: Fabian Hueske
>Assignee: Suneel Marthi
>Priority: Trivial
>
> The link URL of the entry "Elasticsearch 2x (sink)" on the connector's 
> documentation page 
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/index.html
>  is pointing to http://elastic.com but should point to http://elastic.co



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4440) Make API for edge/vertex creation less verbose

[Ivan Mushketyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Mushketyk closed FLINK-4440.
-
Resolution: Not A Problem

> Make API for edge/vertex creation less verbose
> --
>
> Key: FLINK-4440
> URL: https://issues.apache.org/jira/browse/FLINK-4440
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Reporter: Ivan Mushketyk
>Assignee: Ivan Mushketyk
>Priority: Trivial
>
> It would be better if one could create vertex/edges like this:
> {code:java}
> Vertex v = Vertex.create(42);
> Edge e = Edge.create(5, 6);
> {code}
> Instead of this:
> {code:java}
> Vertex v = new Vertex(42, 
> NullValue.getInstance());
> Edge e = new Edge NullValue>(5, 6, NullValue.getInstance());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2269: [FLINK-4190] Generalise RollingSink to work with arbitrar...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2269
  
Hi,
I'm very for the delays! I still have this sitting at the top of my list 
and I'm hoping to get this in by beginning of next week.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---