[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[Renkai Ge (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416602#comment-15416602
 ] 

Renkai Ge commented on FLINK-4282:
--

I have the same opinion with [~aljoscha],the offset property for TimeWindow is 
redundant,the offset value should just effect the point which to devide the 
stream into windows,not effect the window it self.If we want a window with 
offset,we only need to change lines like 

 ,and since SessionWindows decide where to devide stream into windows by the 
timestamp of first message comes,files like 
ProcessingTimeSessionWindows.java,SlidingProcessingTimeWindows.java
and TumblingProcessingTimeWindows.java do not need to be changed for this issue.

> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[Aditi Viswanathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416585#comment-15416585
 ] 

Aditi Viswanathan commented on FLINK-4282:
--

And if they want to use their own methods to do this, they can still do
that and convert to long before passing it to the {{WindowAssigner}}

Aditi Viswanathan | +91-9632130809
Data Engineer,
[24]7 Customer Ltd.

On Thu, Aug 11, 2016 at 11:16 AM, Aditi Viswanathan 



> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[Aditi Viswanathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416580#comment-15416580
 ] 

Aditi Viswanathan commented on FLINK-4282:
--

We could maybe also provide another overloaded method to allow for easy
conversion from time units (seconds, hours, days) to long.

Aditi Viswanathan | +91-9632130809
Data Engineer,
[24]7 Customer Ltd.

On Thu, Aug 11, 2016 at 11:11 AM, Aditi Viswanathan 



> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[Aditi Viswanathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416572#comment-15416572
 ] 

Aditi Viswanathan commented on FLINK-4282:
--

Hi Renkai,

If you look at the PR you'll see that I've provided 2 ways to get offsets -
either java.util.TimeZone or long:

public static ProcessingTimeSessionWindows withGap(Time size, TimeZone
timezone) {
   return new ProcessingTimeSessionWindows(size.toMilliseconds(),
timezone.getRawOffset());
}


public static ProcessingTimeSessionWindows withGap(Time size, long
offsetMillis) {
   return new ProcessingTimeSessionWindows(size.toMilliseconds(), offsetMillis);
}


Is this what you were thinking of?



Aditi Viswanathan | +91-9632130809
Data Engineer,
[24]7 Customer Ltd.




> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[Renkai Ge (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416556#comment-15416556
 ] 

Renkai Ge commented on FLINK-4282:
--

If there is no proper solution to this issue now,I want help solving it.Since 
relations between timezone and offsets is complex,there is a lot of ways to 
handle it(java.util.*,jodatime and java.time.* since Java 8),and people will 
have their own choice,so I think it should not be a part of Flink. We just give 
APIs to fill offsets in,and users chose their own way to get the offset.

> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4357) Implement TaskManager side of slot allocation from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang reassigned FLINK-4357:


Assignee: Zhijiang Wang

> Implement TaskManager side of slot allocation from ResourceManager
> --
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a 
> specific {{JobManager}}. 
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}}, 
> {{AllocationID}}, {{ResourceID}}, {{slotID}}).
> The TM will offer that slot to the JM. If not accepted, the TM notifies the 
> RM that the slot is in fact available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4360) Implement TaskManager registration slot at JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang reassigned FLINK-4360:


Assignee: Zhijiang Wang

> Implement TaskManager registration slot at JobManager
> -
>
> Key: FLINK-4360
> URL: https://issues.apache.org/jira/browse/FLINK-4360
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{TaskManager}} offers a slot to a {{JobManager}} at the 
> {{ResourceManager}}’s behest. That slot is then tied to that JM until the JM 
> releases the slot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4364) Implement TaskManager side of heartbeat from JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4364:
-
Summary: Implement TaskManager side of heartbeat from JobManager  (was: 
Implement TaskManager side of heartbeats from JobManager)

> Implement TaskManager side of heartbeat from JobManager
> ---
>
> Key: FLINK-4364
> URL: https://issues.apache.org/jira/browse/FLINK-4364
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> JM initiates heartbeat with info (JobID, JobManagerId), and TM responses to 
> JM with metrics info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4364) Implement TaskManager side of heartbeats from JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4364:
-
Summary: Implement TaskManager side of heartbeats from JobManager  (was: 
TaskManage responses heartbeats from JobManager)

> Implement TaskManager side of heartbeats from JobManager
> 
>
> Key: FLINK-4364
> URL: https://issues.apache.org/jira/browse/FLINK-4364
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> JM initiates heartbeat with info (JobID, JobManagerId), and TM responses to 
> JM with metrics info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4363) Implement TaskManager basic startup of all components in java

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4363:
-
Description: Similar with current {{TaskManager}},but implement 
initialization and startup all components in java instead of scala.  (was: 
similar with current TM, implement in java instead of scala.)

> Implement TaskManager basic startup of all components in java
> -
>
> Key: FLINK-4363
> URL: https://issues.apache.org/jira/browse/FLINK-4363
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> Similar with current {{TaskManager}},but implement initialization and startup 
> all components in java instead of scala.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4363) Implement TaskManager basic startup of all components in java

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4363:
-
Summary: Implement TaskManager basic startup of all components in java  
(was: TaskManager basic startup of all components)

> Implement TaskManager basic startup of all components in java
> -
>
> Key: FLINK-4363
> URL: https://issues.apache.org/jira/browse/FLINK-4363
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> similar with current TM, implement in java instead of scala.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4360) Implement TaskManager registration slot at JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4360:
-
Summary: Implement TaskManager registration slot at JobManager  (was: 
Implement TaskManager offer slot to JobManager)

> Implement TaskManager registration slot at JobManager
> -
>
> Key: FLINK-4360
> URL: https://issues.apache.org/jira/browse/FLINK-4360
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{TaskManager}} offers a slot to a {{JobManager}} at the 
> {{ResourceManager}}’s behest. That slot is then tied to that JM until the JM 
> releases the slot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4360) Implement TaskManager offer slot to JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4360:
-
Description: The {{TaskManager}} offers a slot to a {{JobManager}} at the 
{{ResourceManager}}’s behest. That slot is then tied to that JM until the JM 
releases the slot.  (was: TM requests slot registration to JM with info 
(JmLeaderID, JobID, ResourceID, AllocationID, profile)
)

> Implement TaskManager offer slot to JobManager
> --
>
> Key: FLINK-4360
> URL: https://issues.apache.org/jira/browse/FLINK-4360
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{TaskManager}} offers a slot to a {{JobManager}} at the 
> {{ResourceManager}}’s behest. That slot is then tied to that JM until the JM 
> releases the slot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4360) Implement TaskManager offer slot to JobManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4360:
-
Summary: Implement TaskManager offer slot to JobManager  (was: TaskManager 
slot registration at JobManager)

> Implement TaskManager offer slot to JobManager
> --
>
> Key: FLINK-4360
> URL: https://issues.apache.org/jira/browse/FLINK-4360
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> TM requests slot registration to JM with info (JmLeaderID, JobID, ResourceID, 
> AllocationID, profile)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4358) Implement TaskManager deployment task into slot

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4358:
-
Description: 
The {{TaskManager}} may receive task deployment request (bound to 
slot/allocationID) from {{JobManager}}.
The TM may decline request if slot is not allocated to that JM or slot does not 
exist.


  was:
TM receives task deployment request (bound to slot/allocationID) from JM.
TM may decline request if slot is not allocated to that JM or slot does not 
exist.



> Implement TaskManager deployment task into slot
> ---
>
> Key: FLINK-4358
> URL: https://issues.apache.org/jira/browse/FLINK-4358
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{TaskManager}} may receive task deployment request (bound to 
> slot/allocationID) from {{JobManager}}.
> The TM may decline request if slot is not allocated to that JM or slot does 
> not exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4358) Implement TaskManager deployment task into slot

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4358:
-
Summary: Implement TaskManager deployment task into slot  (was: TaskManager 
deploys task into slot)

> Implement TaskManager deployment task into slot
> ---
>
> Key: FLINK-4358
> URL: https://issues.apache.org/jira/browse/FLINK-4358
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> TM receives task deployment request (bound to slot/allocationID) from JM.
> TM may decline request if slot is not allocated to that JM or slot does not 
> exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4355) Implement TaskManager side of registration at ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4355:
-
Summary: Implement TaskManager side of registration at ResourceManager  
(was: TaskManager registration at ResourceManager)

> Implement TaskManager side of registration at ResourceManager
> -
>
> Key: FLINK-4355
> URL: https://issues.apache.org/jira/browse/FLINK-4355
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Stephan Ewen
>
> If the {{TaskManager}} is unregistered, it should try and register at the 
> {{ResourceManager}} leader. The registration messages are fenced via the 
> {{RmLeaderID}}.
> The ResourceManager may acknowledge the registration (or respond that the 
> TaskManager is AlreadyRegistered) or refuse the registration.
> Upon registration refusal, the TaskManager may have to kill itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4357) Implement TaskManager side of slot allocation from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4357:
-
Summary: Implement TaskManager side of slot allocation from ResourceManager 
 (was: Implement TaskManager side slot allocation from ResourceManager)

> Implement TaskManager side of slot allocation from ResourceManager
> --
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a 
> specific {{JobManager}}. 
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}}, 
> {{AllocationID}}, {{ResourceID}}, {{slotID}}).
> The TM will offer that slot to the JM. If not accepted, the TM notifies the 
> RM that the slot is in fact available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4357) Implement TaskManager side slot allocation from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4357:
-
Summary: Implement TaskManager side slot allocation from ResourceManager  
(was: TaskManager responses slot allocation from ResourceManager)

> Implement TaskManager side slot allocation from ResourceManager
> ---
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a 
> specific {{JobManager}}. 
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}}, 
> {{AllocationID}}, {{ResourceID}}, {{slotID}}).
> The TM will offer that slot to the JM. If not accepted, the TM notifies the 
> RM that the slot is in fact available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4357) TaskManager responses slot allocation from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4357:
-
Description: 
The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a 
specific {{JobManager}}. 
The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}}, 
{{AllocationID}}, {{ResourceID}}, {{slotID}}).
The TM will offer that slot to the JM. If not accepted, the TM notifies the RM 
that the slot is in fact available.

  was:
RM requests slot allocation to TM with info(RmLeaderID, JobID, AllocationID, 
ResourceID, slotID), and TM may response by confirm or reject allocation.



> TaskManager responses slot allocation from ResourceManager
> --
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a 
> specific {{JobManager}}. 
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}}, 
> {{AllocationID}}, {{ResourceID}}, {{slotID}}).
> The TM will offer that slot to the JM. If not accepted, the TM notifies the 
> RM that the slot is in fact available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4354) Implement TaskManager side of heartbeat from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4354:
-
Summary:  Implement TaskManager side of heartbeat from ResourceManager  
(was: TaskManager responses heartbeats from ResourceManager)

>  Implement TaskManager side of heartbeat from ResourceManager
> -
>
> Key: FLINK-4354
> URL: https://issues.apache.org/jira/browse/FLINK-4354
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{ResourceManager}} initiates heartbeat messages via the {{RmLeaderID}}. 
> The {{TaskManager}} transmits its slot availability with each heartbeat. That 
> way, the RM will always know about available slots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4354) TaskManager responses heartbeats from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang reassigned FLINK-4354:


Assignee: Zhijiang Wang

> TaskManager responses heartbeats from ResourceManager
> -
>
> Key: FLINK-4354
> URL: https://issues.apache.org/jira/browse/FLINK-4354
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{ResourceManager}} initiates heartbeat messages via the {{RmLeaderID}}. 
> The {{TaskManager}} transmits its slot availability with each heartbeat. That 
> way, the RM will always know about available slots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4354) TaskManager responses heartbeats from ResourceManager

[Zhijiang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijiang Wang updated FLINK-4354:
-
Description: 
The {{ResourceManager}} initiates heartbeat messages via the {{RmLeaderID}}. 
The {{TaskManager}} transmits its slot availability with each heartbeat. That 
way, the RM will always know about available slots.

  was:RM initiates Heartbeat with ResourceManagerLeaderID. TM responses to RM 
with slot allocation map, or marks RM as dead after a too long heartbeat pause.


> TaskManager responses heartbeats from ResourceManager
> -
>
> Key: FLINK-4354
> URL: https://issues.apache.org/jira/browse/FLINK-4354
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>
> The {{ResourceManager}} initiates heartbeat messages via the {{RmLeaderID}}. 
> The {{TaskManager}} transmits its slot availability with each heartbeat. That 
> way, the RM will always know about available slots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4366) Enforce parallelism=1 For AllWindowedStream

[Jark Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu reassigned FLINK-4366:
--

Assignee: Jark Wu

> Enforce parallelism=1 For AllWindowedStream
> ---
>
> Key: FLINK-4366
> URL: https://issues.apache.org/jira/browse/FLINK-4366
> Project: Flink
>  Issue Type: Improvement
>Reporter: Aljoscha Krettek
>Assignee: Jark Wu
>
> Right now, it is possible to use {{DataStream.windowAll/timeWindowAll}} and 
> then set a different parallelism afterwards. Flink will silently accept this 
> and spawn the number of parallel operators, only one instance of those will 
> do all the processing, though, since the elements are implicitly keyed by a 
> dummy key.
> We should throw an exception if users try to set a parallelism on an 
> all-windowed stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416179#comment-15416179
 ] 

ASF GitHub Bot commented on FLINK-4359:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2348
  
The first line is Scala Table API, second line is Java Table API and the 
string with the INTERVAL prefix is standard SQL. So we cannot change the 
standard ;)

Yes, I think that is the reason. Calcite does it like that and Oracle too.

https://docs.oracle.com/cd/B19306_01/server.102/b14200/sql_elements003.htm#i38598


> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2348: [FLINK-4359] [table] Add INTERVAL type

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2348
  
The first line is Scala Table API, second line is Java Table API and the 
string with the INTERVAL prefix is standard SQL. So we cannot change the 
standard ;)

Yes, I think that is the reason. Calcite does it like that and Oracle too.

https://docs.oracle.com/cd/B19306_01/server.102/b14200/sql_elements003.htm#i38598


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4355) TaskManager registration at ResourceManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416096#comment-15416096
 ] 

ASF GitHub Bot commented on FLINK-4355:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2353

[FLINK-4355] [cluster management] Implement TaskManager side of 
registration at ResourceManager.

This implements the first part of the `TaskManager` (here temporarily 
called `TaskExecutor` to avoid name clashes) as designed in FLIP-6.

Specifically, it accepts a leader retrieval service for the 
`ResourceManager` and triggers the registration at the ResourceManager.

Tests are pending, this pull request is intended for early review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
taskmanager_register

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2353


commit f1078b72f4b4b9d1549d55905f61fefd0663e9be
Author: Stephan Ewen 
Date:   2016-08-10T18:42:45Z

[FLINK-4355] [cluster management] Implement TaskManager side of 
registration at ResourceManager.




> TaskManager registration at ResourceManager
> ---
>
> Key: FLINK-4355
> URL: https://issues.apache.org/jira/browse/FLINK-4355
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Stephan Ewen
>
> If the {{TaskManager}} is unregistered, it should try and register at the 
> {{ResourceManager}} leader. The registration messages are fenced via the 
> {{RmLeaderID}}.
> The ResourceManager may acknowledge the registration (or respond that the 
> TaskManager is AlreadyRegistered) or refuse the registration.
> Upon registration refusal, the TaskManager may have to kill itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2353: [FLINK-4355] [cluster management] Implement TaskMa...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2353

[FLINK-4355] [cluster management] Implement TaskManager side of 
registration at ResourceManager.

This implements the first part of the `TaskManager` (here temporarily 
called `TaskExecutor` to avoid name clashes) as designed in FLIP-6.

Specifically, it accepts a leader retrieval service for the 
`ResourceManager` and triggers the registration at the ResourceManager.

Tests are pending, this pull request is intended for early review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
taskmanager_register

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2353


commit f1078b72f4b4b9d1549d55905f61fefd0663e9be
Author: Stephan Ewen 
Date:   2016-08-10T18:42:45Z

[FLINK-4355] [cluster management] Implement TaskManager side of 
registration at ResourceManager.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2055) Implement Streaming HBaseSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415984#comment-15415984
 ] 

ASF GitHub Bot commented on FLINK-2055:
---

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
BTW, the example runs well on my machine when I set hbase-client version to 
2.0.0-SNAPSHOT, but travis-ci build failed due to "Failed to collect 
dependencies at org.apache.habse:hbase-client:jar:2.0.0-SNAPSHOT" error. So I 
changed hbase dependency version back to 1.2.2 which unfortunately has a bug ( 
https://issues.apache.org/jira/browse/HBASE-14963 ) that makes the example fail 
to run. The bug is fixed for version 1.3.0+ and 2.0.0


> Implement Streaming HBaseSink
> -
>
> Key: FLINK-2055
> URL: https://issues.apache.org/jira/browse/FLINK-2055
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming, Streaming Connectors
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Hilmi Yildirim
>
> As per : 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Write-Stream-to-HBase-td1300.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2332: [FLINK-2055] Implement Streaming HBaseSink

[delding]

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
BTW, the example runs well on my machine when I set hbase-client version to 
2.0.0-SNAPSHOT, but travis-ci build failed due to "Failed to collect 
dependencies at org.apache.habse:hbase-client:jar:2.0.0-SNAPSHOT" error. So I 
changed hbase dependency version back to 1.2.2 which unfortunately has a bug ( 
https://issues.apache.org/jira/browse/HBASE-14963 ) that makes the example fail 
to run. The bug is fixed for version 1.3.0+ and 2.0.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4104) Restructure Gelly docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415897#comment-15415897
 ] 

ASF GitHub Bot commented on FLINK-4104:
---

Github user vasia commented on the issue:

https://github.com/apache/flink/pull/2258
  
Hi @greghogan, sorry I somehow missed this PR 
I really like the restructuring, thanks! I don't think it's a problem to 
rename to `gelly/index.html`. Just make sure to also update the link in 
`index.md`. Also, I think we shouldn't remove the `gelly.md` but add a 
re-direct instead.


> Restructure Gelly docs
> --
>
> Key: FLINK-4104
> URL: https://issues.apache.org/jira/browse/FLINK-4104
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> The Gelly documentation has grown sufficiently long to suggest dividing into 
> sub-pages. Leave "Using Gelly" on the main page and link to the following 
> topics as sub-pages:
> * Graph API
> * Iterative Graph Processing
> * Library Methods
> * Graph Algorithms
> * Graph Generators



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2258: [FLINK-4104] [docs] Restructure Gelly docs

[vasia]

Github user vasia commented on the issue:

https://github.com/apache/flink/pull/2258
  
Hi @greghogan, sorry I somehow missed this PR 😓
I really like the restructuring, thanks! I don't think it's a problem to 
rename to `gelly/index.html`. Just make sure to also update the link in 
`index.md`. Also, I think we shouldn't remove the `gelly.md` but add a 
re-direct instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4368) Eagerly initialize RrcProtocol members

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415735#comment-15415735
 ] 

ASF GitHub Bot commented on FLINK-4368:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2351
  
@tillrohrmann What do you think about this?

The downside that I can see is that the RPC endpoint can actually receive 
calls before the constructor is completed, basically as soon as the base class' 
constructor is complete. That may actually be an issue.


> Eagerly initialize RrcProtocol members
> --
>
> Key: FLINK-4368
> URL: https://issues.apache.org/jira/browse/FLINK-4368
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>
> The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
> {{start()}} call.
> I suggest to initialize them eagerly as they seem to be integral parts 
> without which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2351: [FLINK-4368] [distributed runtime] Eagerly initialize the...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2351
  
@tillrohrmann What do you think about this?

The downside that I can see is that the RPC endpoint can actually receive 
calls before the constructor is completed, basically as soon as the base class' 
constructor is complete. That may actually be an issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4372) Add ability to take savepoints from job manager web UI

[Zhenzhong Xu (JIRA)]

Zhenzhong Xu created FLINK-4372:
---

 Summary: Add ability to take savepoints from job manager web UI
 Key: FLINK-4372
 URL: https://issues.apache.org/jira/browse/FLINK-4372
 Project: Flink
  Issue Type: Sub-task
Reporter: Zhenzhong Xu


subtask of FLINK-4336



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4371) Add ability to take savepoints from job manager RESTful API

[Zhenzhong Xu (JIRA)]

Zhenzhong Xu created FLINK-4371:
---

 Summary: Add ability to take savepoints from job manager RESTful 
API
 Key: FLINK-4371
 URL: https://issues.apache.org/jira/browse/FLINK-4371
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing, Webfrontend
Reporter: Zhenzhong Xu


subtask of FLINK-4336



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4336) Expose ability to take a savepoint from job manager

[Zhenzhong Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenzhong Xu updated FLINK-4336:

Summary: Expose ability to take a savepoint from job manager  (was: Expose 
ability to take a savepoint from job manager rest api)

> Expose ability to take a savepoint from job manager
> ---
>
> Key: FLINK-4336
> URL: https://issues.apache.org/jira/browse/FLINK-4336
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> There is a need to interact with job manager rest api to manage savepoint 
> snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4336) Expose ability to take a savepoint from job manager rest api

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415687#comment-15415687
 ] 

Robert Metzger commented on FLINK-4336:
---

Yes. Its a good idea to split the backend and frontend changes. Can you create 
two sub-tasks for this JIRA?

> Expose ability to take a savepoint from job manager rest api
> 
>
> Key: FLINK-4336
> URL: https://issues.apache.org/jira/browse/FLINK-4336
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> There is a need to interact with job manager rest api to manage savepoint 
> snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4336) Expose ability to take a savepoint from job manager rest api

[Zhenzhong Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415639#comment-15415639
 ] 

Zhenzhong Xu commented on FLINK-4336:
-

[~rmetzger] Sure, let me look into the implementation and send a proposal soon. 
Regarding this feature, I am thinking maybe we should split it into two pieces, 
step 1 is to make taking savepoint functionality available on the RESTful 
interface, step 2 is to make it available for the UI. Should we split them into 
two different JIRAs?

> Expose ability to take a savepoint from job manager rest api
> 
>
> Key: FLINK-4336
> URL: https://issues.apache.org/jira/browse/FLINK-4336
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> There is a need to interact with job manager rest api to manage savepoint 
> snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415624#comment-15415624
 ] 

ASF GitHub Bot commented on FLINK-4370:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2352

[FLINK-4370] Add an IntelliJ Inspections Profile

This adds an IntelliJ inspections profile for Flink in the `.idea` folder.
When the code is imported into IntelliJ, it should automatically pick up 
and use that inspection profile.

The profile adds some extra warnings and errors that I found useful in the 
past.
We should gradually activate more inspections (or deactivate some) as we 
feel it helps the development.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink inspections

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2352


commit e25e9ef6ac91dc5936d92212c74f96137646a0d1
Author: Stephan Ewen 
Date:   2016-08-10T17:21:28Z

[FLINK-4370] Add an IntelliJ Inspections Profile




> Offer a default IntelliJ inspection profile with Flink
> --
>
> Key: FLINK-4370
> URL: https://issues.apache.org/jira/browse/FLINK-4370
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
> should be automatically picked up when the code is checked out and imported 
> into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2352: [FLINK-4370] Add an IntelliJ Inspections Profile

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2352

[FLINK-4370] Add an IntelliJ Inspections Profile

This adds an IntelliJ inspections profile for Flink in the `.idea` folder.
When the code is imported into IntelliJ, it should automatically pick up 
and use that inspection profile.

The profile adds some extra warnings and errors that I found useful in the 
past.
We should gradually activate more inspections (or deactivate some) as we 
feel it helps the development.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink inspections

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2352


commit e25e9ef6ac91dc5936d92212c74f96137646a0d1
Author: Stephan Ewen 
Date:   2016-08-10T17:21:28Z

[FLINK-4370] Add an IntelliJ Inspections Profile




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4335) Add jar id, and job parameters information to job status rest call

[Zhenzhong Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415619#comment-15415619
 ] 

Zhenzhong Xu commented on FLINK-4335:
-

[~rmetzger] Thanks for the comment, I will implement the interface to register 
user defined parameters and entry class.

The jar ID is listed under http://localhost:8081/jars, it's the ID we need to 
trace back to which jar the job was executed from. We have one or more job Jars 
prepackaged in the jar upload directory in the docker image, we need to use the 
jar id associated with the running job to determine if we have the correct 
job(s) running in the cluster.

> Add jar id, and job parameters information to job status rest call
> --
>
> Key: FLINK-4335
> URL: https://issues.apache.org/jira/browse/FLINK-4335
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> From declarative, reconcilation based job management perspective, there is a 
> need to identify the job jar id, and all job parameters for a running job to 
> determine if the current job is up to date. 
> I think these information needs to be available through the job manager rest 
> call (/jobs/$id).
> * Jar ID
> * Job entry class
> * parallelism
> * all user defined parameters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4370:
---

 Summary: Offer a default IntelliJ inspection profile with Flink
 Key: FLINK-4370
 URL: https://issues.apache.org/jira/browse/FLINK-4370
 Project: Flink
  Issue Type: Improvement
Reporter: Stephan Ewen
Assignee: Stephan Ewen


We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
should be automatically picked up when the code is checked out and imported 
into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4369) Evictingwindowoperator Must Actually Evict Elements

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-4369:
---

 Summary: Evictingwindowoperator Must Actually Evict Elements
 Key: FLINK-4369
 URL: https://issues.apache.org/jira/browse/FLINK-4369
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 1.1.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


{{EvictingWindowOperator}} does not actually remove evicted elements from the 
state. They are only filtered from the Iterable that is given to the 
WindowFunction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4329) Fix Streaming File Source Timestamps/Watermarks Handling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415532#comment-15415532
 ] 

ASF GitHub Bot commented on FLINK-4329:
---

GitHub user kl0u opened a pull request:

https://github.com/apache/flink/pull/2350

[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kl0u/flink continuous_file_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2350


commit 6207a9f5da086d808331afe0e8caf0f03b3fabc5
Author: kl0u 
Date:   2016-08-09T12:11:45Z

[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling

Now the ContinuousFileReaderOperator ignores the watermarks sent by
the source function and emits its own watermarks in case we are
opearating on Ingestion time. In addition, and for Ingestion time
only, the reader also assigns the correct timestamps to the elements
that it reads.




> Fix Streaming File Source Timestamps/Watermarks Handling
> 
>
> Key: FLINK-4329
> URL: https://issues.apache.org/jira/browse/FLINK-4329
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Kostas Kloudas
> Fix For: 1.1.1
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks, 
> i.e. they are just passed through. This means that when the 
> {{ContinuousFileMonitoringFunction}} closes and emits a {{Long.MAX_VALUE}} 
> that watermark can "overtake" the records that are to be emitted in the 
> {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness" 
> setting in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion 
> timestamps since it is not technically a source but looks like one to the 
> user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4368) Eagerly initialize RrcProtocol members

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415533#comment-15415533
 ] 

ASF GitHub Bot commented on FLINK-4368:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2351

[FLINK-4368] [distributed runtime] Eagerly initialize the RPC endpoint 
members

This eagerly initialized the RPC members like
  - self gateway
  - self reference
  - main thread executor
and makes them immutable.

This also adds structural comments to the `RpcProtocol` class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink eager_initialize

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2351


commit 8fdb25c5bbb8d4133791442d37c57bf7f880a8b9
Author: Stephan Ewen 
Date:   2016-08-10T16:27:21Z

[FLINK-4368] [distributed runtime] Eagerly initialize the RPC endpoint 
members




> Eagerly initialize RrcProtocol members
> --
>
> Key: FLINK-4368
> URL: https://issues.apache.org/jira/browse/FLINK-4368
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>
> The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
> {{start()}} call.
> I suggest to initialize them eagerly as they seem to be integral parts 
> without which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2351: [FLINK-4368] [distributed runtime] Eagerly initial...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2351

[FLINK-4368] [distributed runtime] Eagerly initialize the RPC endpoint 
members

This eagerly initialized the RPC members like
  - self gateway
  - self reference
  - main thread executor
and makes them immutable.

This also adds structural comments to the `RpcProtocol` class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink eager_initialize

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2351


commit 8fdb25c5bbb8d4133791442d37c57bf7f880a8b9
Author: Stephan Ewen 
Date:   2016-08-10T16:27:21Z

[FLINK-4368] [distributed runtime] Eagerly initialize the RPC endpoint 
members




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2350: [FLINK-4329] Fix Streaming File Source Timestamps/...

[kl0u]

GitHub user kl0u opened a pull request:

https://github.com/apache/flink/pull/2350

[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kl0u/flink continuous_file_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2350


commit 6207a9f5da086d808331afe0e8caf0f03b3fabc5
Author: kl0u 
Date:   2016-08-09T12:11:45Z

[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling

Now the ContinuousFileReaderOperator ignores the watermarks sent by
the source function and emits its own watermarks in case we are
opearating on Ingestion time. In addition, and for Ingestion time
only, the reader also assigns the correct timestamps to the elements
that it reads.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4368) Eagerly initialize RrcProtocol members

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4368:
---

 Summary: Eagerly initialize RrcProtocol members
 Key: FLINK-4368
 URL: https://issues.apache.org/jira/browse/FLINK-4368
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
 Environment: FLIP-6 feature branch
Reporter: Stephan Ewen


The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
{{start()}} call.

I suggest to initialize them eagerly as they seem to be integral parts without 
which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415436#comment-15415436
 ] 

Pushpendra Jaiswal commented on FLINK-4331:
---

I also think this is the issue with closer. 
I will add this in the best practices section if anyone else faces this issue.
Marking it as resolved and closed.
Thanks a lot !!!


> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>Priority: Minor
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushpendra Jaiswal resolved FLINK-4331.
---
Resolution: Fixed

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>Priority: Minor
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushpendra Jaiswal closed FLINK-4331.
-

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>Priority: Minor
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415410#comment-15415410
 ] 

Stephan Ewen commented on FLINK-4331:
-

Can you share some more of the code? Without knowing what "func" is, what the 
data types of any of the other objects are, it is hard to figure something out.

In the end, this is a Scala Closure serialization issue. Users always have to 
bear the closures they form in mind (that they are serializable)

Flink can only try to help via the CosureCleaner. It could be a case where the 
Flink Closure Cleaner does not clean as much as it could, which should be 
checked.

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>Priority: Minor
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushpendra Jaiswal updated FLINK-4331:
--
Priority: Minor  (was: Major)

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>Priority: Minor
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415368#comment-15415368
 ] 

ASF GitHub Bot commented on FLINK-4035:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2231
  
Okay, cool. Thank you. I'll probably open a pull request with your and my 
changes. I'll let you know so that you can help reviewing it.



> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415397#comment-15415397
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74261518
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

Notice that I have another argument against the mirrors and it is the 
security issue, imagine that a malign mirror puts a wrong version of flink, I 
wonder what is the recomended approach to validate this case (of course notice 
that I asume that hacking apache would be harder).


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[iemejia]

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74261518
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

Notice that I have another argument against the mirrors and it is the 
security issue, imagine that a malign mirror puts a wrong version of flink, I 
wonder what is the recomended approach to validate this case (of course notice 
that I asume that hacking apache would be harder).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415394#comment-15415394
 ] 

ASF GitHub Bot commented on FLINK-4340:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2345
  
We are very much trying to avoid different db instances per operator. Each 
db instance will add quite a big new memory footprint. For higher 
max-parallelism, this will be a problem.

The current philosophy in Flink was to avoid such situations, where a user 
has to very carefully choose a parameter (like max parallelism) and where a 
wrong setting for the parameter would lead to an unstable system.


> Remove RocksDB Semi-Async Checkpoint Mode
> -
>
> Key: FLINK-4340
> URL: https://issues.apache.org/jira/browse/FLINK-4340
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> This seems to be causing to many problems and is also incompatible with the 
> upcoming key-group/sharding changes that will allow rescaling of keyed state.
> Once this is done we can also close FLINK-4228.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2345
  
We are very much trying to avoid different db instances per operator. Each 
db instance will add quite a big new memory footprint. For higher 
max-parallelism, this will be a problem.

The current philosophy in Flink was to avoid such situations, where a user 
has to very carefully choose a parameter (like max parallelism) and where a 
wrong setting for the parameter would lead to an unstable system.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415382#comment-15415382
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
We should maybe wait for 1.1.1 to be up before doing this merge, I will add 
the commit for this and probably rebase everything once it is ready. Is it ok ?


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415381#comment-15415381
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74259291
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
+  tar xvz -C /opt/ && \
--- End diff --

Ok.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
We should maybe wait for 1.1.1 to be up before doing this merge, I will add 
the commit for this and probably rebase everything once it is ready. Is it ok ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[iemejia]

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74259291
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
+  tar xvz -C /opt/ && \
--- End diff --

Ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4367) Offer separate API for watermark generation and timestamp extraction

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-4367:
-

 Summary: Offer separate API for watermark generation and timestamp 
extraction
 Key: FLINK-4367
 URL: https://issues.apache.org/jira/browse/FLINK-4367
 Project: Flink
  Issue Type: Sub-task
  Components: DataStream API
Affects Versions: 1.1.0
Reporter: Robert Metzger
 Fix For: 2.0.0


Right now, the {{AssignerWithPunctuatedWatermarks}} and 
{{AssignerWithPeriodicWatermarks}} interfaces also require implementing a 
{{TimestampAssigner}}.

For cases where the source emits records with timestamps, its not necessary to 
extract timestamps again from the records, we just want to generate watermarks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415360#comment-15415360
 ] 

Pushpendra Jaiswal edited comment on FLINK-4331 at 8/10/16 2:45 PM:


Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Because I don't understand how that changes serialization.
Sure you can close the ticket .
 
  val mapStream = stream
  .map(impression => { 
val values = func(impression.map ) //  replace it by 
impression.getMap()   
new Record(impression, values)  
 })

~Pushpendra Jaiswal


was (Author: pushpendra.jaiswa...@gmail.com):
Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Because I don't understand how that changes serialization.
Sure you can close the ticket .
 
  val mapStream = stream
  .map(impression => { 
val values = func(impression.map ) //  replace it by 
impression.getMap() 
new Record(impression, values)  
 })

~Pushpendra Jaiswal

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415360#comment-15415360
 ] 

Pushpendra Jaiswal edited comment on FLINK-4331 at 8/10/16 2:43 PM:


Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Because I don't understand how that changes serialization.
Sure you can close the ticket .
 
  val mapStream = stream
  .map(impression => { 
val values = func(impression.map ) //  replace it by 
impression.getMap() 
new Record(impression, values)  
 })

~Pushpendra Jaiswal


was (Author: pushpendra.jaiswa...@gmail.com):
Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal








> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2231
  
Okay, cool. Thank you. I'll probably open a pull request with your and my 
changes. I'll let you know so that you can help reviewing it.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415365#comment-15415365
 ] 

ASF GitHub Bot commented on FLINK-4035:
---

Github user radekg commented on the issue:

https://github.com/apache/flink/pull/2231
  
@rmetzger it's absolutely fine to reuse the code. If I can help in any way, 
please let me know.


> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...

[radekg]

Github user radekg commented on the issue:

https://github.com/apache/flink/pull/2231
  
@rmetzger it's absolutely fine to reuse the code. If I can help in any way, 
please let me know.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415363#comment-15415363
 ] 

ASF GitHub Bot commented on FLINK-4035:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2231
  
@radekg, are you okay with me using your pull request as a base for adding 
Kafka 0.10 to Flink?
I've started changing your code from the PR so that we don't need to copy 
so much code: https://github.com/rmetzger/flink/commits/kafka-0.10


> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2231
  
@radekg, are you okay with me using your pull request as a base for adding 
Kafka 0.10 to Flink?
I've started changing your code from the PR so that we don't need to copy 
so much code: https://github.com/rmetzger/flink/commits/kafka-0.10


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415360#comment-15415360
 ] 

Pushpendra Jaiswal commented on FLINK-4331:
---

Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's what it 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal








> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Pushpendra Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415360#comment-15415360
 ] 

Pushpendra Jaiswal edited comment on FLINK-4331 at 8/10/16 2:36 PM:


Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal









was (Author: pushpendra.jaiswa...@gmail.com):
Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's what it 
reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw 
an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal








> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415348#comment-15415348
 ] 

ASF GitHub Bot commented on FLINK-4340:
---

Github user wenlong88 commented on the issue:

https://github.com/apache/flink/pull/2345
  
3q, @aljoscha 
Maybe we can use different db instance for different key group. this 
approach can only work well when key group number is limited to a certain 
number which can not be too large. 
Since the max number of key groups is limited by the max parallelism 
config, I think different db instances can work.




> Remove RocksDB Semi-Async Checkpoint Mode
> -
>
> Key: FLINK-4340
> URL: https://issues.apache.org/jira/browse/FLINK-4340
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> This seems to be causing to many problems and is also incompatible with the 
> upcoming key-group/sharding changes that will allow rescaling of keyed state.
> Once this is done we can also close FLINK-4228.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

[wenlong88]

Github user wenlong88 commented on the issue:

https://github.com/apache/flink/pull/2345
  
3q, @aljoscha 
Maybe we can use different db instance for different key group. this 
approach can only work well when key group number is limited to a certain 
number which can not be too large. 
Since the max number of key groups is limited by the max parallelism 
config, I think different db instances can work.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4365) [metrics] MetricConfig has no documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415343#comment-15415343
 ] 

ASF GitHub Bot commented on FLINK-4365:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2349

[FLINK-4365] [metrics] Add documentation to MetricConfig

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 4265_metricconfig_docu

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2349


commit c1bb371a47ccca83bc81f1473109c46556dbd1da
Author: zentol 
Date:   2016-08-10T14:23:54Z

[FLINK-4365] [metrics] Add documentation to MetricConfig




> [metrics] MetricConfig has no documentation
> ---
>
> Key: FLINK-4365
> URL: https://issues.apache.org/jira/browse/FLINK-4365
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2349: [FLINK-4365] [metrics] Add documentation to Metric...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2349

[FLINK-4365] [metrics] Add documentation to MetricConfig

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 4265_metricconfig_docu

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2349


commit c1bb371a47ccca83bc81f1473109c46556dbd1da
Author: zentol 
Date:   2016-08-10T14:23:54Z

[FLINK-4365] [metrics] Add documentation to MetricConfig




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2348: [FLINK-4359] [table] Add INTERVAL type

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2348
  
Is the `INTERVAL` prefix needed to make it work with the parser? Otherwise 
it would be nice to just write `2 DAYS`, or something like that.

By the way, the reason for the two different interval types is that the 
millis type does not have enough range to represent years?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415303#comment-15415303
 ] 

ASF GitHub Bot commented on FLINK-4359:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2348
  
Is the `INTERVAL` prefix needed to make it work with the parser? Otherwise 
it would be nice to just write `2 DAYS`, or something like that.

By the way, the reason for the two different interval types is that the 
millis type does not have enough range to represent years?




> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat

[nielsbasjes]

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2330
  
I will add a unit test for this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4311) TableInputFormat fails when reused on next split

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415291#comment-15415291
 ] 

ASF GitHub Bot commented on FLINK-4311:
---

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2330
  
I will add a unit test for this.


> TableInputFormat fails when reused on next split
> 
>
> Key: FLINK-4311
> URL: https://issues.apache.org/jira/browse/FLINK-4311
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.0.3
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Critical
>
> We have written a batch job that uses data from HBase by means of using the 
> TableInputFormat.
> We have found that this class sometimes fails with this exception:
> {quote}
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: 
> Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
>  rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
>   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
>   at 
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:152)
>   at 
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:47)
>   at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:147)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.RejectedExecutionException: Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
>  rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:142)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:269)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:165)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
>   ... 10 more
> {quote}
> As you can see the ThreadPoolExecutor was terminated at this point.
> We tracked it down to the fact that 
> # the configure method opens the table
> # the open method obtains the result scanner
> # the closes method closes the table.
> If a second split arrives on the same instance then the open method will fail 
> because the table has already been closed.
> We also found that this error varies with the versions of HBase that are 
> used. I have also seen this exception:
> {quote}
> Caused by: java.io.IOException: hconnection-0x19d37183 closed
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
>   ... 37 more
> {quote}
> I found that in the [documentation of the InputFormat 
> interface|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/io/InputFormat.html]
>  is clearly states
> {quote}IMPORTANT NOTE: Input formats must be written such that an instance 
> can be opened again after it was closed. That is due to the fact that the 
> input format is used for potentially multiple splits. After a split is done, 
> the format's close function is invoked and, if another split is available, 
> the open function is invoked afterwards for the next split.{quote}
> It appears that this specific InputFormat has not 

[jira] [Commented] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415274#comment-15415274
 ] 

Stephan Ewen commented on FLINK-4331:
-

[~pushpendra.jaiswa...@gmail.com] Any update on this issue?
Otherwise I would like to close it as a mixup.

> Flink is not able to serialize scala classes / Task Not Serializable
> 
>
> Key: FLINK-4331
> URL: https://issues.apache.org/jira/browse/FLINK-4331
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.0
>Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it 
> doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public 
> then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==
>   val preAggregate = stream
> .filter(impression => {
> true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] 
> INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid 
> POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - 
> accessedFields: Map(class -> Set()) Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: Task not serializable at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
>  at ) Caused by: java.io.NotSerializableException: 
> org.apache.flink.streaming.api.scala.DataStream at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
> at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
>  at 
> org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
>  ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415264#comment-15415264
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74243197
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

I agree, I just changed it because I find the older code so ugly (and 
fragile) for what it does, also my other argument to do this was the fact that 
people will probably won't build the image as much as they will pull it from 
the official hub.docker image (once we publish this image as the official one). 
I can let it like it was if this is better for you.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[iemejia]

Github user iemejia commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74243197
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

I agree, I just changed it because I find the older code so ugly (and 
fragile) for what it does, also my other argument to do this was the fact that 
people will probably won't build the image as much as they will pull it from 
the official hub.docker image (once we publish this image as the official one). 
I can let it like it was if this is better for you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4366) Enforce parallelism=1 For AllWindowedStream

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-4366:
---

 Summary: Enforce parallelism=1 For AllWindowedStream
 Key: FLINK-4366
 URL: https://issues.apache.org/jira/browse/FLINK-4366
 Project: Flink
  Issue Type: Improvement
Reporter: Aljoscha Krettek


Right now, it is possible to use {{DataStream.windowAll/timeWindowAll}} and 
then set a different parallelism afterwards. Flink will silently accept this 
and spawn the number of parallel operators, only one instance of those will do 
all the processing, though, since the elements are implicitly keyed by a dummy 
key.

We should throw an exception if users try to set a parallelism on an 
all-windowed stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4336) Expose ability to take a savepoint from job manager rest api

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415259#comment-15415259
 ] 

Robert Metzger commented on FLINK-4336:
---

I think this is a useful feature for the web interface.
Maybe it would make sense to put it into the "Checkpoints" tab of a job.

Would you like to contribute this feature to Flink? If so, we can help you with 
the implementation.

> Expose ability to take a savepoint from job manager rest api
> 
>
> Key: FLINK-4336
> URL: https://issues.apache.org/jira/browse/FLINK-4336
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> There is a need to interact with job manager rest api to manage savepoint 
> snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4336) Expose ability to take a savepoint from job manager rest api

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-4336:
--
Component/s: State Backends, Checkpointing

> Expose ability to take a savepoint from job manager rest api
> 
>
> Key: FLINK-4336
> URL: https://issues.apache.org/jira/browse/FLINK-4336
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> There is a need to interact with job manager rest api to manage savepoint 
> snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4335) Add jar id, and job parameters information to job status rest call

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415251#comment-15415251
 ] 

Robert Metzger commented on FLINK-4335:
---

Hi Zhenzhong,
the parallelism and user defined parameters are already available in the web 
interface.
The URL http://localhost:8081/jobs//config returns the parallelism and 
all user defined parameters. For registering the user defined parameters, you 
have to pass them to the execution config's global job parameters. Its a very 
simple interface you need to implement to expose the key value pairs.
Flink's {{ParameterTool}} utility also implements that interface:
{code}
final ParameterTool pt = ParameterTool.fromPropertiesFile(args[0]);
see.getConfig().setGlobalJobParameters(pt);
{code}
I think its easy to expose the job entry class using that mechanism as well.
What do you mean by the "jar ID" ?



> Add jar id, and job parameters information to job status rest call
> --
>
> Key: FLINK-4335
> URL: https://issues.apache.org/jira/browse/FLINK-4335
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> From declarative, reconcilation based job management perspective, there is a 
> need to identify the job jar id, and all job parameters for a running job to 
> determine if the current job is up to date. 
> I think these information needs to be available through the job manager rest 
> call (/jobs/$id).
> * Jar ID
> * Job entry class
> * parallelism
> * all user defined parameters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2348: [FLINK-4359] [table] Add INTERVAL type

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2348
  
@aljoscha would be great if you could have a quick look on it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415205#comment-15415205
 ] 

ASF GitHub Bot commented on FLINK-4359:
---

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2348

[FLINK-4359] [table] Add INTERVAL type

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR introduces INTERVAL types for Table API/SQL. Here are some examples 
how intervals can be used in the APIs. More examples can be found in 
`TemporalTypesTest`.

```
 testAllApis(
1.year,
"1.year",
"INTERVAL '1' YEAR",
"+1-00")

testAllApis(
'f0 + 2.day,
"f0 + 2.day",
"f0 + INTERVAL '2' DAY",
"1990-10-16")
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4359

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2348


commit 67540ed4f6da59c0853e1c56134166173329c1d9
Author: twalthr 
Date:   2016-08-08T08:15:49Z

[FLINK-4359] [table] Add INTERVAL type




> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415204#comment-15415204
 ] 

Jark Wu commented on FLINK-4359:


Do we have any ideas how to implement StreamSQL window? Because currently 
calcite doesn't support window in StreamSQL.

> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2348: [FLINK-4359] [table] Add INTERVAL type

[twalthr]

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2348

[FLINK-4359] [table] Add INTERVAL type

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR introduces INTERVAL types for Table API/SQL. Here are some examples 
how intervals can be used in the APIs. More examples can be found in 
`TemporalTypesTest`.

```
 testAllApis(
1.year,
"1.year",
"INTERVAL '1' YEAR",
"+1-00")

testAllApis(
'f0 + 2.day,
"f0 + 2.day",
"f0 + INTERVAL '2' DAY",
"1990-10-16")
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4359

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2348


commit 67540ed4f6da59c0853e1c56134166173329c1d9
Author: twalthr 
Date:   2016-08-08T08:15:49Z

[FLINK-4359] [table] Add INTERVAL type




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415206#comment-15415206
 ] 

ASF GitHub Bot commented on FLINK-4359:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2348
  
@aljoscha would be great if you could have a quick look on it.


> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4329) Fix Streaming File Source Timestamps/Watermarks Handling

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas updated FLINK-4329:
--
Summary: Fix Streaming File Source Timestamps/Watermarks Handling  (was: 
Fixes Streaming File Source Timestamps/Watermarks Handling)

> Fix Streaming File Source Timestamps/Watermarks Handling
> 
>
> Key: FLINK-4329
> URL: https://issues.apache.org/jira/browse/FLINK-4329
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Kostas Kloudas
> Fix For: 1.1.1
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks, 
> i.e. they are just passed through. This means that when the 
> {{ContinuousFileMonitoringFunction}} closes and emits a {{Long.MAX_VALUE}} 
> that watermark can "overtake" the records that are to be emitted in the 
> {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness" 
> setting in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion 
> timestamps since it is not technically a source but looks like one to the 
> user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2340
  
Thanks for the PR! It would be nice to have the install path configurable. 
Also, we should use ASF's load balancing script for downloading the binaries.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415160#comment-15415160
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2340
  
Thanks for the PR! It would be nice to have the install path configurable. 
Also, we should use ASF's load balancing script for downloading the binaries.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415159#comment-15415159
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74229989
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
+  tar xvz -C /opt/ && \
--- End diff --

Could we make the install path configurable?


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74229989
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
+  tar xvz -C /opt/ && \
--- End diff --

Could we make the install path configurable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415158#comment-15415158
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74229920
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

The use of the archive mirror is discouraged. It's better to use the load 
balancing script which you removed above.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74229920
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

The use of the archive mirror is discouraged. It's better to use the load 
balancing script which you removed above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---