[jira] [Created] (FLINK-6078) ZooKeeper based high availability services should not close the underlying CuratorFramework

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6078:


 Summary: ZooKeeper based high availability services should not 
close the underlying CuratorFramework
 Key: FLINK-6078
 URL: https://issues.apache.org/jira/browse/FLINK-6078
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.3.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.3.0


ZooKeeper based high availability tools like 
{{ZooKeeperLeaderRetrievalService}} and {{ZooKeeperLeaderElectionService}} 
expect that every instance of the services have a dedicated 
{{CuratorFramework}} instance assigned. Thus, they also close this 
{{CuratorFramework}} when the service is closed. This does not play well along 
with the newly introduced {{HighAvailabilityServices}} which caches a single 
{{CuratorFramework}} and shares it among all created services. In order to make 
it work properly together I propose to change the behaviour such that we no 
longer close the {{CuratorFramework}} clients in the ZooKeeper based services.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6079) Ineffective null check in FlinkKafkaConsumerBase#open()

[Ted Yu (JIRA)]

Ted Yu created FLINK-6079:
-

 Summary: Ineffective null check in FlinkKafkaConsumerBase#open()
 Key: FLINK-6079
 URL: https://issues.apache.org/jira/browse/FLINK-6079
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Here is related code:
{code}
List kafkaTopicPartitions = getKafkaPartitions(topics);

subscribedPartitionsToStartOffsets = new 
HashMap<>(kafkaTopicPartitions.size());

if (kafkaTopicPartitions != null) {
{code}
Prior to the null check, kafkaTopicPartitions is already dereferenced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Future of Queryable State Feature

[Joe Olson]

I have a question regarding the future direction of the queryable state feature.


We are current using this feature in production implemented in a RichFlatMap. 
It is doing what we need it to do at the scale we need it done in, with the 
resources we have assigned to it. Win.


However, we'd also like to use this feature in conjunction with Flink's 
windowing. The "Rich" execution environment is not exposed in any of the 
windows / triggers / apply hierarchy, so we cannot expose any of the state 
managed within the windows outside of Flink. Many of our use cases require us 
to have access to values as they are being accumulated, as well as the 
aggregated result.


We can get by with the RichFlatMap for now. I'd like some clarification as to 
whether or not the queryable state feature is going to be extended to the 
windowing components for the next milestone release. This will determine our 
Flink development milestones for the next few months. From consulting the open 
items in JIRA, it does not look like it is on the docket.


I'd be more than willing to help out implementing this feature, but I don't 
think I have the experience to submit this change on my own.

[jira] [Created] (FLINK-6080) Unclosed ObjectOutputStream in NFA#serialize()

[Ted Yu (JIRA)]

Ted Yu created FLINK-6080:
-

 Summary: Unclosed ObjectOutputStream in NFA#serialize()
 Key: FLINK-6080
 URL: https://issues.apache.org/jira/browse/FLINK-6080
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
public void serialize(NFA record, DataOutputView target) throws 
IOException {
  ObjectOutputStream oos = new ObjectOutputStream(new 
DataOutputViewStream(target));
  oos.writeObject(record);
  oos.flush();
}
{code}
ObjectOutputStream should be closed before returning from method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6074) Fix

[sunjincheng (JIRA)]

sunjincheng created FLINK-6074:
--

 Summary: Fix 
 Key: FLINK-6074
 URL: https://issues.apache.org/jira/browse/FLINK-6074
 Project: Flink
  Issue Type: Bug
Reporter: sunjincheng






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6077) Support In/Exists/Except/Any /Some/All for Stream SQL

[radu (JIRA)]

radu created FLINK-6077:
---

 Summary: Support In/Exists/Except/Any /Some/All for Stream SQL
 Key: FLINK-6077
 URL: https://issues.apache.org/jira/browse/FLINK-6077
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: radu


Time target: Proc Time

SQL targeted query examples:


With inner query

Q1. ```SELECT client FROM stream1 WHERE id 
IN 
((SELECT id FROM stream2 GROUP BY FLOOR(proctime TO HOUR), WHERE salary> 4500 
))```

Comment: A concrete example for this query can be to consider selecting the 
customers where their country is the list of countries of suppliers (`Select 
customer FROM customers WHERE Country IN (Select Country FROM suppliers)` )

Comment: This implementation depends on the implementation of the inner
query. The structure can be the same as for inner query support with the
difference that the LogicalJoin between main query and inner query is
conditional.

Comment: The inner query needs a bound as otherwise it cannot be decided
when to trigger.

Comment: If the value is not triggered by the grouping expression then
the inner query must based on when that expression changes value.

Comments: boundaries should be supported over all options: group by
clauses; windows or time expressions (\[floor/ceil\](rowtime/proctime to
hour),)

With collection

Q2. ```SELECT * FROM stream1 WHERE b 
IN 
(5000, 7000, 8000, 9000)```

Comment: This can be checked if it is supported by the DataStreamCalc
implementation. If not it can be transformed as a sub-JIRA task to
extend the DataStreamCalc functionality to implement this conditional
behavior.

Comment: A similar functionality can be provided if the collection is a
table rather than a set of values.

With table

```SELECT client FROM stream1 WHERE id 
IN 
((SELECT id FROM table1 where stream1.id = table1.id))```

Comment: This can be a sub-JIRA issue, perhaps  within the context of dynamic 
tables, to support the join with tables and filtering operations based on 
contents from an external table


General comments: **Except** is similar in behavior with IN or EXISTS as
it filters out outputs of the main stream based on data from a secondary
stream. The implementation will follow exactly the same logic as for
IN/Exists by filtering the outputs in the join function between the main
stream and the secondary stream. Additionally, we apply the same
restrictions for the secondary/inner queries.

```SELECT ID, NAME FROM CUSTOMERS LEFT JOIN ORDERS ON CUSTOMERS.ID = 
ORDERS.CUSTOMER\_ID 
EXCEPT
SELECT ID, NAME FROM CUSTOMERS RIGHT JOIN ORDERS ON CUSTOMERS.ID = 
ORDERS.CUSTOMER\_ID GROUP BY FLOOR(procTime TO HOUR);```


Description:


The IN and EXISTS operators are conditional clauses in WHERE clause to
check for values in certain collections. The collections based on which
the restriction of the values is done can either be static (values,
tables, or parts of a stream). This JIRA issue is concerned with the
latter case of checking for values over a stream. In order for this
operation to work, the stream needs to be bounded such that the result
can trigger and the collection can be formed. This points out to using
some boundaries or groupings over the sub-query that forms the
collection over which IN is applied. This should be supported via 3
options as shown below. Each of these options can be a sub-JIRA issue.


1)  Group By clauses that are applied over some monotonic order of the
stream based on which ranges are defined.

`   [...] GROUP BY prodId`

3)  Window clauses to define rolling partitions of the data of the
stream, which evolve in time.

`[...] WINDOW HOUR AS (RANGE INTERVAL '10' MINUTE TO SECOND(3)   
PRECEDING);`

Functionality example
-

We exemplify below the functionality of the IN/Exists when working with
streams.

```SELECT * FROM stream1 WHERE id IN ((SELECT id2  FROM stream2 GROUP BY 
FLOOR(PROCTIME TO HOUR) WHERE b>10   ))```


Note: The inner query triggers only once an hour. For the next hour the result 
of the previous hour from the inner query will be the one used to filter the 
results from the main query as they come. This is consistent also with how the 
inner queries are translated (see inner queries)


||IngestionTime(Event)||Stream1||Stream 2||Output||
|10:00:01|  Id1,10| |nil|
|10:02:00|  |Id2,2| |   
|11:25:00|  |Id3,15| |  
|12:3:00|   Id2,15| |nil|
|12:05:00|  Id3,11| |Id3,11|
|12:06:00|  |Id2,30| |  
|12:07:00|  |Id3,2|  |  
|12:09:00|  Id2.17| |nil|
|12:10:00|  Id3,20| |Id3,20|
|...|



Implementation option
-

Considering that the query only makes sense in the context of 1) window
boundaries and 2) over sub-queries that extract collections of data, the
main design of this is 

[jira] [Created] (FLINK-6075) Support Limit/Top(Sort) for Stream SQL

[radu (JIRA)]

radu created FLINK-6075:
---

 Summary: Support Limit/Top(Sort) for Stream SQL
 Key: FLINK-6075
 URL: https://issues.apache.org/jira/browse/FLINK-6075
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: radu


These will be split in 3 separated JIRA issues. However, the design is the same 
only the processing function differs in terms of the output. Hence, the design 
is the same for all of them.

Time target: Proc Time


**SQL targeted query examples:**

*Sort example*

Q1)` SELECT a FROM stream1 GROUP BY HOP(proctime, INTERVAL '1' HOUR, INTERVAL 
'3' HOUR) ORDER BY b` 

Comment: window is defined using GROUP BY

Comment: ASC or DESC keywords can be placed to mark the ordering type



*Limit example*

Q2) `SELECT a FROM stream1 WHERE rowtime BETWEEN current_timestamp - INTERVAL 
'1' HOUR AND current_timestamp ORDER BY b LIMIT 10`

Comment: window is defined using time ranges in the WHERE clause

Comment: window is row triggered


*Top example*

Q3) `SELECT sum(a) OVER (ORDER BY proctime RANGE INTERVAL '1' HOUR PRECEDING 
LIMIT 10) FROM stream1`  

Comment: limit over the contents of the sliding window


General Comments:
-All these SQL clauses are supported only over windows (bounded collections of 
data). 
-Each of the 3 operators will be supported with each of the types of expressing 
the windows. 



**Description**

The 3 operations (limit, top and sort) are similar in behavior as they all 
require a sorted collection of the data on which the logic will be applied 
(i.e., select a subset of the items or the entire sorted set). These functions 
would make sense in the streaming context only in the context of a window. 
Without defining a window the functions could never emit as the sort operation 
would never trigger. If an SQL query will be provided without limits an error 
will be thrown (`SELECT a FROM stream1 TOP 10` -> ERROR). Although not targeted 
by this JIRA, in the case of working based on event time order, the retraction 
mechanisms of windows and the lateness mechanisms can be used to deal with out 
of order events and retraction/updates of results.


**Functionality example**

We exemplify with the query below for all the 3 types of operators (sorting, 
limit and top). Rowtime indicates when the HOP window will trigger – which can 
be observed in the fact that outputs are generated only at those moments. The 
HOP windows will trigger at every hour (fixed hour) and each event will 
contribute/ be duplicated for 2 consecutive hour intervals. Proctime indicates 
the processing time when a new event arrives in the system. Events are of the 
type (a,b) with the ordering being applied on the b field.


`SELECT a FROM stream1 HOP(proctime, INTERVAL '1' HOUR, INTERVAL '2' HOUR) 
ORDER BY b (LIMIT 2/ TOP 2 / [ASC/DESC] `)

||Rowtime|| Proctime||  Stream1||   Limit 2||   Top 2|| Sort 
[ASC]||
| |10:00:00  |(aaa, 11) |   | ||
| |10:05:00  |(aab, 7)  |   | ||
|10-11|11:00:00  |  |   aab,aaa |aab,aaa  | aab,aaa|
| |11:03:00  |(aac,21)  |   | ||

|11-12|12:00:00  |  |   aab,aaa |aab,aaa  | aab,aaa,aac|
| |12:10:00  |(abb,12)  |   | ||

| |12:15:00  |(abb,12)  |   | ||

|12-13|13:00:00  |  |   abb,abb | abb,abb | abb,abb,aac|
|...|


**Implementation option**

Considering that the SQL operators will be associated with window boundaries, 
the functionality will be implemented within the logic of the window as follows.

* Window assigner – selected based on the type of window used in SQL (TUMBLING, 
SLIDING…)
* Evictor/ Trigger – time or count evictor based on the definition of the 
window boundaries
* Apply – window function that sorts data and selects the output to trigger 
(based on LIMIT/TOP parameters). All data will be sorted at once and result 
outputted when the window is triggered

An alternative implementation can be to use a fold window function to sort the 
elements as they arrive, one at a time followed by a flatMap to filter the 
number of outputs. 


> see picture in the attached document  



**General logic of Join**

```
inputDataStream.window(new [Slide/Tumble][Time/Count]Window())
//.trigger(new [Time/Count]Trigger()) – use default
//.evictor(new [Time/Count]Evictor()) – use default
.apply(SortAndFilter());
```




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6084) Cassandra connector does not declare all dependencies

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-6084:
-

 Summary: Cassandra connector does not declare all dependencies
 Key: FLINK-6084
 URL: https://issues.apache.org/jira/browse/FLINK-6084
 Project: Flink
  Issue Type: Bug
  Components: Cassandra Connector
Affects Versions: 1.2.0
Reporter: Robert Metzger
Assignee: Robert Metzger
Priority: Critical


This has been reported by a user: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-2-and-Cassandra-Connector-td11546.html

The cassandra client defines metrics-core as a dependency, but the shading is 
dropping the dependency when building the dependency reduced pom.
To resolve the issue, we need to add the following line into the shading config 
of the cassandra module:

true

This makes the metrics dependency appear again in the dep red pom.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6082) Support window definition for SQL Queries based on WHERE clause with time condition

[radu (JIRA)]

radu created FLINK-6082:
---

 Summary: Support window definition for SQL Queries based on WHERE 
clause with time condition
 Key: FLINK-6082
 URL: https://issues.apache.org/jira/browse/FLINK-6082
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: radu


Time target: Proc Time

Calcite documentation refers to query examples where the (time)
boundaries are defined as condition within the WHERE clause. As Flink
community targets compatibility with Calcite, it makes sense to support
the definition of windows via this method as well as corresponding
aggregation on top of them.

SQL targeted query examples:


```SELECT productId, count(\*) FROM stream1 WHERE proctime BETWEEN current\_ 
timestamp - INTERVAL '1' HOUR AND current\_timestamp```

General comment:

1)  window boundaries are defined as conditions in WHERE clause.

2)  For indicating the usage of different stream times, rowtime and
proctime can be used

3)  The boundaries are defined based on special construct provided by
calcite: current\_timestamp and time operations

Description:


The logic of this operator is strictly related to supporting aggregates
over sliding windows defined with OVER
([FLINK-5653](https://issues.apache.org/jira/browse/FLINK-5653),
[FLINK-5654](https://issues.apache.org/jira/browse/FLINK-5654),
[FLINK-5655](https://issues.apache.org/jira/browse/FLINK-5655),
[FLINK-5658](https://issues.apache.org/jira/browse/FLINK-5658),
[FLINK-5656](https://issues.apache.org/jira/browse/FLINK-5656)). In this
issue the design considered queries where the window is defined with the
syntax of OVER clause and aggregates are applied over this period. This
is similar in behavior with the only exception that the window
boundaries are defined with respect to the WHERE conditions. Besides
this the logic and the types of aggregates to be supported should be the
same (sum, count, avg, min, max). Supporting these types of query is
related to the pie chart problem tackled by calcite.

Similar as for the OVER windows, the construct should build rolling
windows (i.e., windows that are triggered and move with every incoming
event).

Functionality example
-

We exemplify below the functionality of the IN/Exists when working with
streams.

`SELECT a, count(*) FROM stream1 WHERE proctime BETWEEN current_ timestamp - 
INTERVAL '1' HOUR AND current_timestamp;`

||IngestionTime(Event)||Stream1||   Output||
|10:00:01   |Id1,10 |Id1,1|
|10:02:00   |Id2,2  |Id2,2|
|11:25:00   |Id3,2  |Id3,1|
|12:03:00   |Id4,15 |Id4,2|
|12:05:00   |Id5,11 |Id5,3|
|12:56:00   |Id6,20 |Id6,3|
|...|

Implementation option
-

Considering that the query follows the same functionality as for the
aggregates over window, the implementation should follow the same
implementation as for the OVER clause. Considering that the WHERE
condition are typically related to timing, this means that in case of
one unbound boundary the
[FLINK-5658](https://issues.apache.org/jira/browse/FLINK-5658) should be
used, while for bounded time windows the
[FLINK-5654](https://issues.apache.org/jira/browse/FLINK-5654) design
should be used.

The window boundaries will be extracted from the WHERE condition.

The rule will not be mapped anymore to a LogicalWindow, which means that
the conversion to this would need to happen from the current
DataStreamCalc rule. In this sense, a dedicated condition will be added
such that in case the WHERE clause has time conditions, the operator
implementation of the Over clause (used in the previous issues) should
be used.

```
class DataStreamCalcRule

  
---
  {
  --- 
---
  

  def convert(rel: RelNode): RelNode = {

  val calc: LogicalCalc = rel.asInstanceOf\[LogicalCalc\]

  val traitSet: RelTraitSet = 
rel.getTraitSet.replace(DataStreamConvention.INSTANCE)

  val convInput: RelNode = RelOptRule.convert(calc.getInput, 
DataStreamConvention.INSTANCE)
  
  IF(WHERE contains TIME limits)
  
  {
  
 >   IF(bounded)
>   
>   new DataStreamProcTimeTimeAggregate
>   
>   ELSE
>   
>   new DataStreamSlideEventTimeRowAgg
>   
>   }
>  
  
  Else
  
  **{**

  

  new DataStreamCalc(

  rel.getCluster,

  traitSet,

  convInput,

  rel.getRowType,

  calc.getProgram,

  description)

  }
  
  }

  }
  
---
```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6085) flink as micro service

[Chen Qin (JIRA)]

Chen Qin created FLINK-6085:
---

 Summary: flink as micro service
 Key: FLINK-6085
 URL: https://issues.apache.org/jira/browse/FLINK-6085
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API, JobManager
Reporter: Chen Qin
Priority: Minor


Track discussion around run flink as a micro service, includes but not limited 
to 
- RPC (web service endpoint) source
  as web service endpoint accept RPC call, ingest to the streaming job(only one)
- callback mechanism 
- task assignment should honor deployment group (web tier hosts should be 
isolated from rest of task assignment)




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

[Robert Metzger]

The cassandra connector is probably not usable in Flink 1.2.0. I would like
to include a fix in 1.2.1:
https://issues.apache.org/jira/browse/FLINK-6084

Please let me know if this fix becomes a blocker for the 1.2.1 release. If
so, I can validate the fix myself to speed up things.

On Thu, Mar 16, 2017 at 9:41 AM, Jinkui Shi  wrote:

> @Tzu-li(Fordon)Tai
>
> FLINK-5650 is fix by [1]. Chesnay Scheduler push a PR please.
>
> [1] https://github.com/zentol/flink/tree/5650_python_test_debug <
> https://github.com/zentol/flink/tree/5650_python_test_debug>
>
>
> > 在 2017年3月16日，上午3:37，Stephan Ewen  写道：
> >
> > Thanks for the update!
> >
> > Just merged to 1.2.1 also: [FLINK-5962] [checkpoints] Remove scheduled
> > cancel-task from timer queue to prevent memory leaks
> >
> > The remaining issue list looks good, but I would say that (5) is
> optional.
> > It is not a critical production bug.
> >
> >
> >
> > On Wed, Mar 15, 2017 at 5:38 PM, Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>
> > wrote:
> >
> >> Thanks a lot for the updates so far everyone!
> >>
> >> From the discussion so far, the below is the still unfixed pending
> issues
> >> for 1.1.5 / 1.2.1 release.
> >>
> >> Since there’s only one backport for 1.1.5 left, I think having an RC for
> >> 1.1.5 near the end of this week / early next week is very promising, as
> >> basically everything is already in.
> >> I’d be happy to volunteer to help manage the release for 1.1.5, and
> >> prepare the RC when it’s ready :)
> >>
> >> For 1.2.1, we can leave the pending list here for tracking, and come
> back
> >> to update it in the near future.
> >>
> >> If there’s anything I missed, please let me know!
> >>
> >>
> >> === Still pending for Flink 1.1.5 ===
> >>
> >> (1) https://issues.apache.org/jira/browse/FLINK-5701
> >> Broken at-least-once Kafka producer.
> >> Status: backport PR pending - https://github.com/apache/flink/pull/3549
> .
> >> Since it is a relatively self-contained change, I expect this to be a
> fast
> >> fix.
> >>
> >>
> >>
> >> === Still pending for Flink 1.2.1 ===
> >>
> >> (1) https://issues.apache.org/jira/browse/FLINK-5808
> >> Fix Missing verification for setParallelism and setMaxParallelism
> >> Status: PR - https://github.com/apache/flink/pull/3509, review in
> progress
> >>
> >> (2) https://issues.apache.org/jira/browse/FLINK-5713
> >> Protect against NPE in WindowOperator window cleanup
> >> Status: PR - https://github.com/apache/flink/pull/3535, review pending
> >>
> >> (3) https://issues.apache.org/jira/browse/FLINK-6044
> >> TypeSerializerSerializationProxy.read() doesn't verify the read buffer
> >> length
> >> Status: Fixed for master, 1.2 backport pending
> >>
> >> (4) https://issues.apache.org/jira/browse/FLINK-5985
> >> Flink treats every task as stateful (making topology changes impossible)
> >> Status: PR - https://github.com/apache/flink/pull/3543, review in
> progress
> >>
> >> (5) https://issues.apache.org/jira/browse/FLINK-5650
> >> Flink-python tests taking up too much time
> >> Status: I think Chesnay currently has some progress with this one, we
> can
> >> see if we want to make this a blocker
> >>
> >>
> >> Cheers,
> >> Gordon
> >>
> >> On March 15, 2017 at 7:16:53 PM, Jinkui Shi (shijinkui...@163.com)
> wrote:
> >>
> >> Can we fix this issue in the 1.2.1:
> >>
> >> Flink-python tests cost too long time
> >> https://issues.apache.org/jira/browse/FLINK-5650 <
> >> https://issues.apache.org/jira/browse/FLINK-5650>
> >>
> >>> 在 2017年3月15日，下午6:29，Vladislav Pernin  写道：
> >>>
> >>> I just tested in in my reproducer. It works.
> >>>
> >>> 2017-03-15 11:22 GMT+01:00 Aljoscha Krettek :
> >>>
>  I did in fact just open a PR for
> > https://issues.apache.org/jira/browse/FLINK-6001
> > NPE on TumblingEventTimeWindows with ContinuousEventTimeTrigger and
> > allowedLateness
> 
> 
>  On Tue, Mar 14, 2017, at 18:20, Vladislav Pernin wrote:
> > Hi,
> >
> > I would also include the following (not yet resolved) issue in the
> >> 1.2.1
> > scope :
> >
> > https://issues.apache.org/jira/browse/FLINK-6001
> > NPE on TumblingEventTimeWindows with ContinuousEventTimeTrigger and
> > allowedLateness
> >
> > 2017-03-14 17:34 GMT+01:00 Ufuk Celebi :
> >
> >> Big +1 Gordon!
> >>
> >> I think (10) is very critical to have in 1.2.1.
> >>
> >> – Ufuk
> >>
> >>
> >> On Tue, Mar 14, 2017 at 3:37 PM, Stefan Richter
> >>  wrote:
> >>> Hi,
> >>>
> >>> I would suggest to also include in 1.2.1:
> >>>
> >>> (9) https://issues.apache.org/jira/browse/FLINK-6044 <
> >> https://issues.apache.org/jira/browse/FLINK-6044>
> >>> Replaces unintentional calls to InputStream#read(…) with the
> intended
> >>> and correct InputStream#readFully(…)
> >>> Status: PR
> 

Bumping API stability check version

[Greg Hogan]

Hi,

I see in the parent pom.xml that 1.3-SNAPSHOT is checking for API stability 
against 1.1.4. Also, that this version was only bumped with FLINK-5617 late in 
the 1.2 development cycle.

Should we bump this version as part of the release process, i.e. on the 1.2.0 
release updating 1.3-SNAPSHOT to verify against 1.2.0? And should we be setting 
‘breakBuildOnModifications’ [0] to true for patch releases (overridden when 
necessary)?

[0] https://siom79.github.io/japicmp/MavenPlugin.html

Greg

Re: Bumping API stability check version

[Robert Metzger]

Hi Greg,

I was not able to update the version to 1.2.0 when updating to
1.3-SNAPSHOT, because 1.2.0 was not released at that point in time :)
But I agree that we should update that version once the release is out. I
forgot to do it.

I'm not sure if I completely understand "breakBuildOnModifications". Does
it fail the build even if somebody did a change that is non API breaking on
a @Public class?

On Thu, Mar 16, 2017 at 3:37 PM, Greg Hogan  wrote:

> Hi,
>
> I see in the parent pom.xml that 1.3-SNAPSHOT is checking for API
> stability against 1.1.4. Also, that this version was only bumped with
> FLINK-5617 late in the 1.2 development cycle.
>
> Should we bump this version as part of the release process, i.e. on the
> 1.2.0 release updating 1.3-SNAPSHOT to verify against 1.2.0? And should we
> be setting ‘breakBuildOnModifications’ [0] to true for patch releases
> (overridden when necessary)?
>
> [0] https://siom79.github.io/japicmp/MavenPlugin.html
>
> Greg

Extended SQL support for Streaming

[Radu Tudoran]

Hi,

I have created several JIRA issues with proposal implementation to support more 
SQL functions for streaming. As there is an ongoing discussion about supporting 
retraction for the Stream SQL, these features focus on the processing time. 
Extending them for event time would be done (in most of the cases similar) 
after the new retraction mechanism are merged. The JIRA issues are:
1.
2. FLINK-6082 - Support 
window definition for SQL Queries based on WHERE clause with time condition
3. FLINK-6081 - 
Offset/Fetch support for SQL Streaming
4. FLINK-6077 - Support 
In/Exists/Except/Any /Some/All for Stream SQL
5. FLINK-6075 - Support 
Limit/Top(Sort) for Stream SQL
6. FLINK-6073 - Support 
for SQL inner queries for proctime
7.
Please let me know what you think and of course any feedback about the 
design/priority for the roadmap/implementation is more than welcomed.

Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

[DISCUSS] Could we Improve tests time and stability?

[Dmytro Shkvyra]

Hi everyone,
May be we should remove -XX:-UseGCOverheadLimit option from 
maven-surefire-plugin args and increase -Xmx to 1536m for forks?
We have about 4 GB RAM and 2 cores at test VMs. I think we can make test faster 
than now. When I tried testing flink-runtime some tests work too slow due to GC 
overhead.
May be you also faced to problem when Travis build was fallen by timeout?
Also we can use GC algorithms explicitly for forks execution.
BTW, we run tests with java 7 and 8 and these versions use by default different 
GC algorithms (GC1 for 8 and Parallel GC for 7). IMHO when we have strict 
limitations of RAM and time of build we should avoid any ambiguity.
In case when some tests can generate very big datasets very fast, paralel GC 
can do not have time to clean up. I do not know how G1 work in this case 
exactly, but may be would better use old good -XX:+UseSerialGC. We have only 1 
core per fork so we anyway cant use all advantages of G1 and ParralelGC. If we 
use SerialGC (use stop the world) we can be sure that GC collect almost all 
garbage before test continue.
What do you think about my idea?
May be someone has another ideas how to improve tests time and stability?


Dmytro Shkvyra
Senior Software Engineer

Office: +380 44 390 5457 x 65346   Cell: 
+380 50 357 6828   Email: 
dmytro_shkv...@epam.com
Kyiv, Ukraine (GMT+3)   epam.com

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) 
to which it is addressed and contains information that is legally privileged 
and confidential. If you are not the intended recipient, or the person 
responsible for delivering the message to the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited. All unintended recipients are obliged to 
delete this message and destroy any printed copies.

[jira] [Created] (FLINK-6083) [TaskManager] Support readiness/liveness probes

[Andrey (JIRA)]

Andrey created FLINK-6083:
-

 Summary: [TaskManager] Support readiness/liveness probes
 Key: FLINK-6083
 URL: https://issues.apache.org/jira/browse/FLINK-6083
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.2.0, 1.3.0
Reporter: Andrey


Currently there is no way to tell if TaskManager is operating or not. Operating 
means "its connected to JobManager and ready to process requests". 

TaskManager should provide metric "jobManagerConnection":
* 0 - disconnected
* 1 - connected

Or any other API which could expose connection status



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Could we Improve tests time and stability?

[Robert Metzger]

Hi Dmytro,

I'm happy to hear that you are trying to help us improving the test time
situation :)
We have another discussion here on the dev@ list to split the project into
two git repositories to resolve the problem.

I agree that your proposed changes could improve the build times, but I'm
not sure if they are enough to resolve them forever. Some tests just waste
time by waiting on stuff to happen :)
If you want, you can enable travis for your own Flink fork on GitHub, add
your proposed changes to the travis / maven files and see how much they
improve the build time.


On Thu, Mar 16, 2017 at 5:06 PM, Dmytro Shkvyra 
wrote:

> Hi everyone,
> May be we should remove -XX:-UseGCOverheadLimit option from
> maven-surefire-plugin args and increase -Xmx to 1536m for forks?
> We have about 4 GB RAM and 2 cores at test VMs. I think we can make test
> faster than now. When I tried testing flink-runtime some tests work too
> slow due to GC overhead.
> May be you also faced to problem when Travis build was fallen by timeout?
> Also we can use GC algorithms explicitly for forks execution.
> BTW, we run tests with java 7 and 8 and these versions use by default
> different GC algorithms (GC1 for 8 and Parallel GC for 7). IMHO when we
> have strict limitations of RAM and time of build we should avoid any
> ambiguity.
> In case when some tests can generate very big datasets very fast, paralel
> GC can do not have time to clean up. I do not know how G1 work in this case
> exactly, but may be would better use old good -XX:+UseSerialGC. We have
> only 1 core per fork so we anyway cant use all advantages of G1 and
> ParralelGC. If we use SerialGC (use stop the world) we can be sure that GC
> collect almost all garbage before test continue.
> What do you think about my idea?
> May be someone has another ideas how to improve tests time and stability?
>
>
> Dmytro Shkvyra
> Senior Software Engineer
>
> Office: +380 44 390 5457 x 65346
>  Cell: +380 50 357 6828   Email:
> dmytro_shkv...@epam.com
> Kyiv, Ukraine (GMT+3)   epam.com
>
> CONFIDENTIALITY CAUTION AND DISCLAIMER
> This message is intended only for the use of the individual(s) or
> entity(ies) to which it is addressed and contains information that is
> legally privileged and confidential. If you are not the intended recipient,
> or the person responsible for delivering the message to the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. All unintended
> recipients are obliged to delete this message and destroy any printed
> copies.
>
>

[jira] [Created] (FLINK-6081) Offset/Fetch support for SQL Streaming

[radu (JIRA)]

radu created FLINK-6081:
---

 Summary: Offset/Fetch support for SQL Streaming
 Key: FLINK-6081
 URL: https://issues.apache.org/jira/browse/FLINK-6081
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: radu


Time target: Proc Time

The main scope of Offset/Fetch is for pagination support. In the context
of streaming Offset and Fetch would make sense within the scope of
certain window constructs as they refer to buffered data from the stream
(with a main usage to restrict the output that is shown at a certain
moment). Therefore they should be applied to the output of the types of
windows supported by the ORDER BY clauses. Moreover, in accordance to
the SQL best practices, they can only be used with an ORDER BY clause.

SQL targeted query examples:


Window defined based on group by clause

```Q1: SELECT a ORDER BY b OFFSET n ROWS FROM stream1 GROUP BY HOP(proctime, 
INTERVAL '1' HOUR, INTERVAL '3' HOUR) ```

Window defined based on where clause time boundaries

```Q2: SELECT a ORDER BY b OFFSET n WHERE procTime() BETWEEN current\_timestamp 
- INTERVAL '1' HOUR AND current\_timestamp FROM stream1 ```


~~Window defined as sliding windows (aggregates) ~~

``` Q3: ~~SELECT SUM(a) OVER (ORDER BY proctime RANGE INTERVAL '1' HOUR 
PRECEDING b OFFSET n ROWS) FROM stream1~~ ```

Comment: Supporting offset over sliding windows (within the window) does
not make sense because the main scope of OFFSET/FETCH is for pagination
support. Therefore this functionality example should only be supported in 
relation to the
output of a query. Hence, Q3 will not be supported

The general grammar (Calcite version) for OFFSET/FECTH with available
parameters is shown below:

```
Select […]

[ ORDER BY orderItem [, orderItem ]* ]

[ OFFSET start { ROW | ROWS } ]

[ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ]
```

Description
---

Offset and Fetch are primary used for pagination support (i.e., restrict
the output that is shown at some point). They were mainly designed to
support web page display of the contents. Building on this scenario we
can imagine a similar role for OFFSET and FETCH for streams that would
display contents via a web page. In such a scenario the number of
outputs to be displayed would be limited using such operators (probably
for pagination and aesthetic reasons). However, as for any stream
application there is a natural evolution in time, the operators output
should evolve with the update rate of the application. The fact that
there is an update rate and a collection of events related to a stream
points to window constructs. Therefore the OFFSET/FETCH functionality
would be related to the window mechanisms/boundaries defined by the
query. Hence when the window construct would be re-triggered the output
would be filtered again from the cardinality point of view based on the
logic of the OFFSET/FETCH.

Because of the primary reasons of supporting pagination (and controlling
the number of outputs) we limit the usage of OFFSET/Fetch for window
constructs that would be related to the output. Because of this
supporting those on sliding window with query aggregates (e.g., Q3 query
example) would not make sense. Additionally there is an implicit need
for some ordering clause due to the fact that OFFSET and FETCH point to
ordering positions. That is why these functions would be supported only
if an ORDER BY clause is present.

Functionality example
-

We exemplify the usage of OFFSET below using the following query. Event
schema is in the form (a,b).

``` SELECT a ORDER BY b OFFSET 2 ROWS FROM stream1 GROUP BY GROUP BY 
CEIL(proctime TO HOUR) ```


||Proctime||IngestionTime(Event)||  Stream1||   Output||
| |10:00:01|(a1, 7)| |  
| |10:05:00|(c1, 2)| |  
| |10:12:00|(b1,5)| |   
| |10:50:00|(d1,2)| |   
|10-11| |   |b1,a1|
| |11:03:00|(a2,10)|| |
|11-12| |   |nil|
|...|


Implementation option
-

There are 2 options to implement the logic of OFFSET/Fetch:

1)  Within the logic of the window (i.e. sorting window)

Similar as for sorting support (ORDER BY clause), considering that the
SQL operators will be associated with window boundaries, the
functionality will be implemented within the logic of the window as
follows. We extract the window boundaries and window type from the query
logic. These will be used to define the type of the window, triggering
policy. The logic of the query (i.e., the sorting of the events) will in
turn be implemented within the window function. In addition to this, the
logic of for filtering the output based on the cardinality logic of
OFFSET/FETCH will be added. With this implementation the logic of the
OFFSET and FETCH is combined with the one of ORDER BY clause. As ORDER
BY is always required, it does not 

[jira] [Created] (FLINK-6066) Separate logical and physical RelNode layer in Flink

[Kurt Young (JIRA)]

Kurt Young created FLINK-6066:
-

 Summary: Separate logical and physical RelNode layer in Flink
 Key: FLINK-6066
 URL: https://issues.apache.org/jira/browse/FLINK-6066
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: Kurt Young


Currently flink-table contains two layer of RelNodes to work with Calcite. One 
is actually from Calcite itself, such as TableScan, Project, Filter and so on. 
Then depends on what environment we are using, the RelNode translate to 
DataSetXXX or DataStreamXXX, like DataSetScan or DataStreamAggregate. All the 
optimization rules happened in the phase in a cost base manner. 

I suppose to further separate the second layer into two, one is more logical 
just like Calcite, and the other one is more physical. In the logical layer, we 
can do lots of optimization without real statistics involved, like partition 
pruning, projection pushdown. And we may even use rule-based optimization for 
logical optimize. In physical optimize phase, we then introduce some real 
statistics and to choose what ever physical strategy we want to use in a cost 
base manner, like join strategy selection or join reorder. 

Since the complexity for cost base optimization grows exponentially when the 
plan is complex. By separating the optimization can make it more efficient and 
easier to maintain.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6067) ProjectNode and FilterNode cannot merge in Batch TableAPI

[jingzhang (JIRA)]

jingzhang created FLINK-6067:


 Summary: ProjectNode and FilterNode cannot merge in Batch TableAPI
 Key: FLINK-6067
 URL: https://issues.apache.org/jira/browse/FLINK-6067
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: jingzhang
Assignee: jingzhang


{code}
val table1 = tEnv.scan( "tb1")
val table2 = tEnv.scan("tb2")
val result = table2
.where("d < 3")
.select('d *2, 'e, 'g.upperCase())
.unionAll(table1.select('a *2, 'b, 'c.upperCase()))
{code}
we run the above code in the BatchTableAPI, we would get the following 
optimizedPlan
{code}
DataSetUnion(union=[_c0, e, _c2])
DataSetCalc(select=[*(d, 2) AS _c0, e, UPPER(g) AS _c2])
DataSetCalc(select=[d, e, g], where=[<(d, 3)])
BatchTableSourceScan(table=[[tb2]], fields=[d, e, g])
DataSetCalc(select=[*(a, 2) AS _c0, b, UPPER(c) AS _c2])
BatchTableSourceScan(table=[[tb1]], fields=[a, b, c])
{code}
However, we run the above code in the Stream TableAPI, we would get the 
following optimizedPlan
{code}
DataStreamUnion(union=[_c0, e, _c2])
DataStreamCalc(select=[*(d, 2) AS _c0, e, UPPER(g) AS _c2], where=[<(d, 3)])
StreamTableSourceScan(table=[[test, db2, tb2]], fields=[d, e, g])
DataStreamCalc(select=[*(a, 2) AS _c0, b, UPPER(c) AS _c2])
StreamTableSourceScan(table=[[test, db1, tb1]], fields=[a, b, c])
{code}

we can find that in the batch tableAPI, the project and filterNode don't merge 
into a single node. However, in the Stream tableAPI, these two nodes could 
merge into one.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6068) Support If() in built in function of TableAPI

[Zhuoluo Yang (JIRA)]

Zhuoluo Yang created FLINK-6068:
---

 Summary: Support If() in built in function of TableAPI
 Key: FLINK-6068
 URL: https://issues.apache.org/jira/browse/FLINK-6068
 Project: Flink
  Issue Type: Improvement
Reporter: Zhuoluo Yang
Assignee: Zhuoluo Yang


Most sql system support if() as an built-in udf. However, we didn't register 
the if() in the function category. A great many of our users use syntax 'if(a, 
b, c)'. Also most sql systems support 'if(a, b, c)' syntax.

Mysql: 
https://dev.mysql.com/doc/refman/5.7/en/control-flow-functions.html#function_if

Hive: 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6069) Add documentation CEP - Watermarks

[Patrick Pircher (JIRA)]

Patrick Pircher created FLINK-6069:
--

 Summary: Add documentation CEP - Watermarks
 Key: FLINK-6069
 URL: https://issues.apache.org/jira/browse/FLINK-6069
 Project: Flink
  Issue Type: Improvement
  Components: CEP
Reporter: Patrick Pircher
Priority: Minor


I was trying to understand how CEP works with EventTime/Watermarks and 
specifically how it handles events that come out of order.
So the only thing I found was anemail archive and some pointers in the Source 
Code.

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/CEP-and-slightly-out-of-order-elements-td9439.html
https://github.com/apache/flink/blob/release-1.2.0/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/operator/AbstractCEPPatternOperator.java#L71
https://github.com/apache/flink/blob/release-1.2.0/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/operator/AbstractCEPPatternOperator.java#L96



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [POLL] Who still uses Java 7 with Flink ?

[Bowen Li]

There's always a tradeoff we need to make. I'm in favor of upgrading to
Java 8 to bring in all new Java features.

The common way I've seen (and I agree) other software upgrading major
things like this is 1) upgrade for next big release without backward
compatibility and notify everyone 2) maintain and patch current, old-tech
compatible version at a reasonably limited scope. Building backward
compatibility is too much for an open sourced project



On Wed, Mar 15, 2017 at 7:10 AM, Robert Metzger  wrote:

> I've put it also on our Twitter account:
> https://twitter.com/ApacheFlink/status/842015062667755521
>
> On Wed, Mar 15, 2017 at 2:19 PM, Martin Neumann 
> wrote:
>
> > I think this easier done in a straw poll than in an email conversation.
> > I created one at: http://www.strawpoll.me/12535073
> > (Note that you have multiple choices.)
> >
> >
> > Though I prefer Java 8 most of the time I have to work on Java 7. A lot
> of
> > the infrastructure I work on still runs Java 7, one of the companies I
> > build a prototype for a while back just updated to Java 7 2 years ago. I
> > doubt we can ditch Java 7 support any time soon if we want to make it
> easy
> > for companies to use Flink.
> >
> > cheers Martin
> >
> > //PS sorry if this gets sent twice, we just migrated to a new mail system
> > and a lot of things are broken
> >
> > 
> > From: Stephan Ewen 
> > Sent: Wednesday, March 15, 2017 12:30:24 PM
> > To: u...@flink.apache.org; dev@flink.apache.org
> > Subject: [POLL] Who still uses Java 7 with Flink ?
> >
> > Hi all!
> >
> > I would like to get a feeling how much Java 7 is still being used among
> > Flink users.
> >
> > At some point, it would be great to drop Java 7 support and make use of
> > Java 8's new features, but first we would need to get a feeling how much
> > Java 7 is still used.
> >
> > Would be happy if users on Java 7 respond here, or even users that have
> > some insights into how widespread they think Java 7 still is.
> >
> > Thanks,
> > Stephan
> >
> >
> >
> >
> >
>

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

[Jinkui Shi]

@Tzu-li(Fordon)Tai

FLINK-5650 is fix by [1]. Chesnay Scheduler push a PR please.

[1] https://github.com/zentol/flink/tree/5650_python_test_debug 



> 在 2017年3月16日，上午3:37，Stephan Ewen  写道：
> 
> Thanks for the update!
> 
> Just merged to 1.2.1 also: [FLINK-5962] [checkpoints] Remove scheduled
> cancel-task from timer queue to prevent memory leaks
> 
> The remaining issue list looks good, but I would say that (5) is optional.
> It is not a critical production bug.
> 
> 
> 
> On Wed, Mar 15, 2017 at 5:38 PM, Tzu-Li (Gordon) Tai 
> wrote:
> 
>> Thanks a lot for the updates so far everyone!
>> 
>> From the discussion so far, the below is the still unfixed pending issues
>> for 1.1.5 / 1.2.1 release.
>> 
>> Since there’s only one backport for 1.1.5 left, I think having an RC for
>> 1.1.5 near the end of this week / early next week is very promising, as
>> basically everything is already in.
>> I’d be happy to volunteer to help manage the release for 1.1.5, and
>> prepare the RC when it’s ready :)
>> 
>> For 1.2.1, we can leave the pending list here for tracking, and come back
>> to update it in the near future.
>> 
>> If there’s anything I missed, please let me know!
>> 
>> 
>> === Still pending for Flink 1.1.5 ===
>> 
>> (1) https://issues.apache.org/jira/browse/FLINK-5701
>> Broken at-least-once Kafka producer.
>> Status: backport PR pending - https://github.com/apache/flink/pull/3549.
>> Since it is a relatively self-contained change, I expect this to be a fast
>> fix.
>> 
>> 
>> 
>> === Still pending for Flink 1.2.1 ===
>> 
>> (1) https://issues.apache.org/jira/browse/FLINK-5808
>> Fix Missing verification for setParallelism and setMaxParallelism
>> Status: PR - https://github.com/apache/flink/pull/3509, review in progress
>> 
>> (2) https://issues.apache.org/jira/browse/FLINK-5713
>> Protect against NPE in WindowOperator window cleanup
>> Status: PR - https://github.com/apache/flink/pull/3535, review pending
>> 
>> (3) https://issues.apache.org/jira/browse/FLINK-6044
>> TypeSerializerSerializationProxy.read() doesn't verify the read buffer
>> length
>> Status: Fixed for master, 1.2 backport pending
>> 
>> (4) https://issues.apache.org/jira/browse/FLINK-5985
>> Flink treats every task as stateful (making topology changes impossible)
>> Status: PR - https://github.com/apache/flink/pull/3543, review in progress
>> 
>> (5) https://issues.apache.org/jira/browse/FLINK-5650
>> Flink-python tests taking up too much time
>> Status: I think Chesnay currently has some progress with this one, we can
>> see if we want to make this a blocker
>> 
>> 
>> Cheers,
>> Gordon
>> 
>> On March 15, 2017 at 7:16:53 PM, Jinkui Shi (shijinkui...@163.com) wrote:
>> 
>> Can we fix this issue in the 1.2.1:
>> 
>> Flink-python tests cost too long time
>> https://issues.apache.org/jira/browse/FLINK-5650 <
>> https://issues.apache.org/jira/browse/FLINK-5650>
>> 
>>> 在 2017年3月15日，下午6:29，Vladislav Pernin  写道：
>>> 
>>> I just tested in in my reproducer. It works.
>>> 
>>> 2017-03-15 11:22 GMT+01:00 Aljoscha Krettek :
>>> 
 I did in fact just open a PR for
> https://issues.apache.org/jira/browse/FLINK-6001
> NPE on TumblingEventTimeWindows with ContinuousEventTimeTrigger and
> allowedLateness
 
 
 On Tue, Mar 14, 2017, at 18:20, Vladislav Pernin wrote:
> Hi,
> 
> I would also include the following (not yet resolved) issue in the
>> 1.2.1
> scope :
> 
> https://issues.apache.org/jira/browse/FLINK-6001
> NPE on TumblingEventTimeWindows with ContinuousEventTimeTrigger and
> allowedLateness
> 
> 2017-03-14 17:34 GMT+01:00 Ufuk Celebi :
> 
>> Big +1 Gordon!
>> 
>> I think (10) is very critical to have in 1.2.1.
>> 
>> – Ufuk
>> 
>> 
>> On Tue, Mar 14, 2017 at 3:37 PM, Stefan Richter
>>  wrote:
>>> Hi,
>>> 
>>> I would suggest to also include in 1.2.1:
>>> 
>>> (9) https://issues.apache.org/jira/browse/FLINK-6044 <
>> https://issues.apache.org/jira/browse/FLINK-6044>
>>> Replaces unintentional calls to InputStream#read(…) with the intended
>>> and correct InputStream#readFully(…)
>>> Status: PR
>>> 
>>> (10) https://issues.apache.org/jira/browse/FLINK-5985 <
>> https://issues.apache.org/jira/browse/FLINK-5985>
>>> Flink 1.2 was creating state handles for stateless tasks which caused
>> trouble
>>> at restore time for users that wanted to do some changes that only
>> include
>>> stateless operators to their topology.
>>> Status: PR
>>> 
>>> 
 Am 14.03.2017 um 15:15 schrieb Till Rohrmann  :
 
 Thanks for kicking off the discussion Tzu-Li. I'd like to add the
>> following
 issues 

[jira] [Created] (FLINK-6070) Suggestion: add ComparableTuple types

[Luke Hutchison (JIRA)]

Luke Hutchison created FLINK-6070:
-

 Summary: Suggestion: add ComparableTuple types
 Key: FLINK-6070
 URL: https://issues.apache.org/jira/browse/FLINK-6070
 Project: Flink
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.2.0
Reporter: Luke Hutchison
Priority: Minor


Since Java doesn't have built-in tuple types, I find myself using Flink tuples 
for a lot of tasks in Flink programs. One downside is that these tuples are not 
inherently comparable, so when you want to sort a collection of tuples, you 
have to provide a custom comparator.

I created a ComparableTuple2 type, as follows. I wanted to get feedback on 
whether something like this would be considered useful for Flink before I 
submitted a PR. Also, I don't know how high I should go with the field arity 
for a ComparableTuple -- presumably not as high as for non-comparable tuples?

{code}
import org.apache.flink.api.java.tuple.Tuple2;

/** A comparable tuple, consisting of comparable fields that act as primary and 
secondary sort keys. */
public class ComparableTuple2, T1 extends 
Comparable> extends Tuple2
implements Comparable> {
private static final long serialVersionUID = 1L;

private boolean invertSortOrder0;
private boolean invertSortOrder1;

public ComparableTuple2() {
}

/**
 * Create a 2-tuple of comparable elements.
 * 
 * @param f0
 *The first element, which is also the primary sort key, and 
sorts in ascending order.
 * @param f1
 *The second element, which is also the secondary sort key, and 
sorts in ascending order.
 * @param invertSortOrder0
 *If true, invert the sort order for the first field (i.e. sort 
in descending order).
 * @param invertSortOrder1
 *If true, invert the sort order for the second field (i.e. 
sort in descending order).
 */
public ComparableTuple2(T0 f0, T1 f1) {
super(f0, f1);
}

/**
 * Create a comparable 2-tuple out of comparable elements.
 * 
 * @param f0
 *The first element, which is also the primary sort key, and 
sorts in ascending order if
 *invertSortOrder0 == false, else sorts in descending order.
 * @param f1
 *The second element, which is also the secondary sort key, and 
sorts in decending order if
 *invertSortOrder1 == false, else sorts in descending order.
 * @param invertSortOrder0
 *If true, invert the sort order for the first field (i.e. sort 
in descending order).
 * @param invertSortOrder1
 *If true, invert the sort order for the second field (i.e. 
sort in descending order).
 */
public ComparableTuple2(final T0 f0, final T1 f1, final boolean 
invertSortOrder0,
final boolean invertSortOrder1) {
super(f0, f1);
this.invertSortOrder0 = invertSortOrder0;
this.invertSortOrder1 = invertSortOrder1;
}

/**
 * Comparison function that compares first the primary sort key, f0, and 
then if equal, compares the secondary sort
 * key, f1.
 */
@Override
public int compareTo(final Tuple2 o) {
int diff = this.f0.compareTo(o.f0);
if (invertSortOrder0) {
diff = -diff;
}
if (diff != 0) {
return diff;
}
diff = this.f1.compareTo(o.f1);
if (invertSortOrder1) {
diff = -diff;
}
return diff;
}
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [Discuss] Retraction for Flink Streaming

[Shaoxuan Wang]

The doc gets lots of attention. I appreciate everyone for the valuable
comments.

Hi Fabian,
Thanks for your comments.

I agree with you that we should ensure that all operators are running in
the same mode (either turning on retraction globally or not). From my point
of view, I did not see any problem to support the retraction for all the
operators we have so far in Flink master (assuming we keep the window
aggregate in a "without early firing mode" as it is now, and not to handle
the late arrival after window is materialized as we agreed via the
commenting discussions in the google doc).

Can you please create a feature branch. We have a complete design on top of
Flink master. We would like to submit the design to the feature branch
asap, then everyone will get a deeper inside of the design with more
details.

Regards,
Shaoxuan



On Thu, Mar 16, 2017 at 12:54 AM, Fabian Hueske  wrote:

> Hi Shaoxuan,
>
> thanks a lot for this proposal!
> Support for retractions is a super nice and important feature and will
> enable many more use cases for the Table API / SQL.
> I'm really excited to see this happening. I made a first pass over your
> proposal and added a few comments. I'll do another pass soon.
>
> Since it is only 6 weeks left until the feature freeze for Flink 1.3, I
> propose to develop the retraction support in a feature branch.
> IMO, we must make sure that either all operators support retraction or
> none. Otherwise, the behavior of the Table API / SQL will not be
> predictable.
>
> I also think that we should define which operators we want to support in
> Flink 1.3 in order to coordinate the development of retraction support.
>
> What do others think?
>
> Cheers, Fabian
>
>
> 2017-03-14 16:53 GMT+01:00 Shaoxuan Wang :
>
> > Hello everyone,
> >
> > Flink is widely used in Alibaba Group, especially in our Search and
> > Recommendation Infra. Retraction is one of the most important features
> that
> > we needed. We have spent lots of efforts to try to solve this problem,
> and
> > gladly at the end we develop an approach which can address most of
> > retraction problems in our production scenarios. Same as usual, we
> (Alibaba
> > search-data infra team) would like to share our retraction solution to
> the
> > entire Flink community. If you like this proposal, I would also like to
> > make it as one of the FLIPs. I am attaching the design doc of "Retraction
> > for Flink Streaming" as well as the introduction section below. I have
> also
> > created a master jira (FLINK-6047) to track the discussion and design of
> > the Flink retraction. All suggestions and comments are welcome.
> >
> >
> > *Design doc:*
> > https://docs.google.com/document/d/18XlGPcfsGbnPSApRipJDLPg5IFNGT
> > Qjnz7emkVpZlkw
> >
> > *Introduction:*
> >
> > "Retraction" is an important building block for data streaming to refine
> > the early fired results in streaming. “Early firing” are very common and
> > widely used in many streaming scenarios, for instance “window-less” or
> > unbounded aggregate and stream-stream inner join, windowed (with early
> > firing) aggregate and stream-stream inner join. As described in Streaming
> > 102, there are mainly two cases that require retractions: 1) update on
> the
> > keyed table (the key is either a primaryKey (PK) on source table, or a
> > groupKey/partitionKey in an aggregate); 2) When dynamic windows (e.g.,
> > session window) are in use, the new value may be replacing more than one
> > previous window due to window merging.
> >
> > To the best of our knowledge, the retraction for the early fired
> streaming
> > results has never been practically solved before. In this proposal, we
> > develop a retraction solution and explain how it works for the problem of
> > “update on the keyed table”. The same solution can be easily extended for
> > the dynamic windows merging, as the key component of retraction - how to
> > refine an early fired results - is the same across different problems.
> >
> > *Master Jira: *
> > https://issues.apache.org/jira/browse/FLINK-6047
> >
> >
> > Regards,
> > Shaoxuan
> >
>

[jira] [Created] (FLINK-6076) Let the HeartbeatManager interface extend HeartbeatTarget

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6076:


 Summary: Let the HeartbeatManager interface extend HeartbeatTarget
 Key: FLINK-6076
 URL: https://issues.apache.org/jira/browse/FLINK-6076
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination
Affects Versions: 1.3.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 1.3.0


The {{HeartbeatManager}} interface sub classes {{HeartbeatManagerImpl}} and 
{{HeartbeatManagerSenderImpl}} all implement the {{HeartbeatTarget}} interface. 
I think we could let the {{HeartbeatManager}} interface extend the 
{{HeartbeatTarget}} interface. Furthermore, I think it would be nicer to 
instantiate the {{HeartbeatManagerImpls}} directly with a {{HeartbeatListener}} 
instance. That way we could get rid of the {{HeartbeatManager#start}} method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6086) Rework PythonSender/-Streamer generics

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6086:
---

 Summary: Rework PythonSender/-Streamer generics
 Key: FLINK-6086
 URL: https://issues.apache.org/jira/browse/FLINK-6086
 Project: Flink
  Issue Type: Improvement
  Components: Python API
Affects Versions: 1.3.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.3.0


The PythonSender/PythonStreamer classes make heavy use of raw-types or 
wildcards. This is done since both classes deal with the cases of the operation 
having 1 or 2 input types, the differentiation of which is done completely by 
an int argument.

We can clean this rather insane code up a bit by creating dedicated sub-classes 
for 1 or 2 inputs.

The PythonSender also contains multiple unused methods that we can remove while 
we're at it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Machine Learning on Flink - Next steps

[Gábor Hermann]

@Theodore: thanks for bringing the discussion together.
I think it's reasonable to go on all the three directions, just as you 
suggested. I agree we should concentrate our efforts, but we can do a 
low-effort evaluation of all the three.


I would like to volunteer for shepherding *Offline learning on 
Streaming*. I am already working on related issues, and I believe I have 
a fairly good overview on the streaming API and its limitations. 
However, we need to find a good use-case to aim for, and I don't have 
one in mind yet, so please help with that if you can. I absolutely agree 
with Theodore, that setting the scope is the most important here.


We should find a simple use-case for incremental learning. As Flink is 
really strong in low-latency data processing, the best would be a 
use-case where rapidly adapting the model to new data provides a value. 
We should also consider low-latency serving for such a use-case, as 
there is not much use in fast model updates if we cannot serve the 
predictions that fast. Of course, it's okay to simply implement offline 
algorithms, but showcasing would be easier if we could add prediction 
serving for the model in the same system.


What should be the way of work here? We could have sketches for the 
separate projects in Gdocs, then the shepherds could make a proposal out 
of it. Would that be feasible?


@Stephan:
Thanks for your all insights. I also like the approach of aiming for new 
and somewhat unexplored areas. I guess we can do that with both the 
serving/evaluation and incremental training (that should be in scope of 
the offline ML on streaming).


I agree GPU acceleration is an important issue, however it might be 
out-of-scope for the prototypes of these new ML directions. What do you 
think?


Regarding your comments on the other thread, I'm really glad PMC is 
working towards growing the community. This is crucial to have anything 
merged in Flink while keeping the code quality. However, for the 
prototypes, I'd prefer Theodore's suggestion, to do it in a separate 
repository, to make initial development faster. After the prototypes 
have proven their usability we could merge them, and continue working on 
them inside the Flink repository. But we can decide that later.


Cheers,
Gabor


On 2017-03-14 21:04, Stephan Ewen wrote:

Thanks Theo. Just wrote some comments on the other thread, but it looks
like you got it covered already.

Let me re-post what I think may help as input:

*Concerning Model Evaluation / Serving *

- My personal take is that the "model evaluation" over streams will be
happening in any case - there
  is genuine interest in that and various users have built that
themselves already.
  I would be a cool way to do something that has a very high chance of
being productionized by users soon.

- The model evaluation as one step of a streaming pipeline (classifying
events), followed by CEP (pattern detection)
  or anomaly detection is a valuable use case on top of what pure model
serving systems usually do.

- A question I have not yet a good intuition on is whether the "model
evaluation" and the training part are so
 different (one a good abstraction for model evaluation has been built)
that there is little cross coordination needed,
 or whether there is potential in integrating them.


*Thoughts on the ML training library (DataSet API or DataStream API)*

   - I honestly don't quite understand what the big difference will be in
targeting the batch or streaming API. You can use the
 DataSet API in a quite low-level fashion (missing async iterations).

   - There seems especially now to be a big trend towards deep learning (is
it just temporary or will this be the future?) and in
  that space, little works without GPU acceleration.

   - It is always easier to do something new than to be the n-th version of
something existing (sorry for the generic true-ism).
 The later admittedly gives the "all in one integrated framework"
advantage (which can be a very strong argument indeed),
 but the former attracts completely new communities and can often make
more impact with less effort.

   - The "new" is not required to be "online learning", where Theo has
described some concerns well.
 It can also be traditional ML re-imagined for "continuous
applications", as "continuous / incremental re-training" or so.
 Even on the "model evaluation side", there is a lot of interesting
stuff as mentioned already, like ensembles, multi-armed bandits, ...

   - It may be well worth tapping into the work of an existing library (like
tensorflow) for an easy fix to some hard problems (pre-existing
 hardware integration, pre-existing optimized linear algebra solvers,
etc) and think about how such use cases would look like in
 the context of typical Flink applications.


*A bit of engine background information that may help in the planning:*

   - The DataStream API will in the future also support 

[jira] [Created] (FLINK-6087) The file-filter is only applied to directories in ContinuousFileMonitoringFunction

[Yassine Marzougui (JIRA)]

Yassine Marzougui created FLINK-6087:


 Summary: The file-filter is only applied to directories in 
ContinuousFileMonitoringFunction
 Key: FLINK-6087
 URL: https://issues.apache.org/jira/browse/FLINK-6087
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Yassine Marzougui
Assignee: Yassine Marzougui


The file-filter is only applied to directories in 
ContinuousFileMonitoringFunction, therefore filtering individual files when 
enumerateNestedFiles is true is currently not possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] FLIP-17 Side Inputs

[Gábor Hermann]

Regarding the CoFlatMap workaround,
- For keyed streams, do you suggest that having a per-key buffer stored 
as keyed state would have a large memory overhead? That must be true, 
although a workaround could be partitioning the data and using a 
non-keyed stream. Of course that seems hacky, as we have a keyed stream 
abstraction, so I agree with you.
- I agree that keeping a broadcast side-input in the operator state is 
not optimal. That's a good point I have not thought about. First we have 
a separate abstraction for broadcast state, then we can optimize e.g. 
checkpointing it (avoiding checkpointing it at every operator).



Regarding blocking/backpressuring inputs, it should not only be useful 
for static side-input, but also for periodically updated (i.e. slowly 
changing). E.g. when a machine learning model is updated and loaded 
every hour, it make sense to prioritize loading the model on the side 
input. But I see the limitations of the underlying runtime.


Exposing a buffer could be useful for now. Although, the *API* for 
blocking could even be implemented by simply buffering. So the buffering 
could be hidden from the user, and later maybe optimized to not only 
buffer, but also apply backpressure. What do you think? Again, for the 
prototype, exposing the buffer should be fine IMHO. API and 
implementation for blocking inputs could be a separate issue, but let's 
not forget about it.


Cheers,
Gabor


On 2017-03-15 16:14, Aljoscha Krettek wrote:

Hi,
thanks for you input! :-)

Regarding 1)
I don't see the benefit of integrating windowing into the side-input
logic. Windowing can happen upstream and whenever that emits new data
then operator will notice because there is new input. Having windowing
inside the side-input of an operator as well would just make the
implementation more complex without adding benefit, IMHO.

Regarding 2)
That's a very good observation! I think we are fine, though, because
checkpoint barriers never "overtake" elements. It's only elements that
can overtake checkpoint barriers. If the broadcast state on different
parallel instances differs in a checkpoint then it only differs because
some parallel instances have reflected changes in their state from
elements that they shouldn't have "seen" yet in the exactly-once mode.
If we pick the state of an arbitrary instance as the de-facto state we
don't break guarantees any more than turning on at-least-once mode does.

Regarding 3)
We need the special buffer support for keyed operations because there we
need to make sure that data is restored on the correct operator that is
responsible for the key of the data while also allowing us to iterate
over all the buffered data (for when we are ready to process the data).
This iteration over elements is not possible when simply storing data in
keyed state.

What do you think?

On Wed, Mar 15, 2017, at 09:07, wenlong.lwl wrote:

Hi, Aljoscha, I just go through your prototype. I like the design of the
SideInputReader which can make it flexible to determine when we can get
the
side input.

I agree that side inputs are API sugar on the top of the three
components(n-ary
inputs, broadcast state and input buffering), following is some more
thought about the three component:

1. Take both N-ary input operator and windowing/triggers mechanism into
consideration, I think we may need the N-ary input operator supports some
inputs(side inputs) are windowed while the others(main input) are normal
stream. for static/slow-evolving data, we need to use global windows and
for windowed-base join data , we need to use time window or custom
windows.
The window function on the side input can be used to collect or merge the
data to generate the value of the side input(a single value or
list/map).
Once a side input reader window is triggered, the SideInputReader will
return value available, and if a Window is triggered more than once, the
value of side input will be updated and maybe the SideInputReader need a
interface to notice the user that something changed. Besides, I prefer
the
option to make every input of N-ary input operator equal, because user
may
need one side input depends on another side input.

2. Regarding broadcast state, my concern is that how can we merge the
value
of the state from different subtasks. If the job running in at least once
mode, the returned value of broadcast state from different subtasks will
be
different. Is there already any design on broadcast state?

3. Regarding input buffering, I think if we use window/trigger mechanism,
state can be store in the state of window, which may be mostly like what
we
need to do currently in KeyedWindow and AllWindow. We may need to allow
custom merge strategy on all window state data since in side inputs we
may
need to choose data according to broadcast state strategy  while in
normal
windows we can just redistribute the window state data.

What do you think?

Best Regards!

Wenlong

On 14 March 2017 at 01:41, Aljoscha Krettek 

[jira] [Created] (FLINK-6073) Support for SQL inner queries for proctime

[radu (JIRA)]

radu created FLINK-6073:
---

 Summary: Support for SQL inner queries for proctime
 Key: FLINK-6073
 URL: https://issues.apache.org/jira/browse/FLINK-6073
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: radu
Priority: Critical


Time target: Proc Time


**SQL targeted query examples:**
 
Q1) `Select  item, (select item2 from stream2 ) as itemExtern from stream1;`

Comments: This is the main functionality targeted by this JIRA to enable to 
combine in the main query results from an inner query.

Q2) `Select  s1.item, (Select a2 from table as t2 where table.id = s1.id  limit 
1) from s1;`


Comments:
Another equivalent way to write the first example of inner query is with limit 
1. This ensures the equivalency with the SingleElementAggregation used when 
translated the main target syntax for inner query. We must ensure that the 2 
syntaxes are supported and implemented with the same functionality. 
There is the option also to select elements in the inner query from a table not 
just from a different stream. This should be a sub-JIRA issue implement this 
support.


**Description:**
Parsing the SQL inner query via calcite is translated to a join function (left 
join with always true condition) between the output of the query on the main 
stream and the output of a single output aggregation operation on the inner 
query. The translation logic is shown below
```
LogicalJoin [condition=true;type=LEFT]
LogicalSingleValue[type=aggregation]
…logic of inner query (LogicalProject, LogicalScan…)
…logical of main,external query (LogicalProject, LogicalScan…))
```

`LogicalJoin[condition=true;type=LEFT] `– it can be considered as a special 
case operation rather than a proper join to be implemented between 
stream-to-stream. The implementation behavior should attach to the main stream 
output a value from a different query. 

`LogicalSingleValue[type=aggregation]` – it can be interpreted as the holder of 
the single value that results from the inner query. As this operator is the 
guarantee that the inner query will bring to the join no more than one value, 
there are several options on how to consider it’s functionality in the 
streaming context:
1.  Throw an error if the inner query returns more than one result. This 
would be a typical behavior in the case of standard SQL over DB. However, it is 
very unlikely that a stream would only emit a single value. Therefore, such a 
behavior would be very limited for streams in the inner query. However, such a 
behavior might be more useful and common if the inner query is over a table. 
1.  We can interpret the usage of this parameter as the guarantee that at 
one moment only one value is selected. Therefore the behavior would rather be 
as a filter to select one value. This brings the option that the output of this 
operator evolves in time with the second stream that drives the inner query. 
The decision on when to evolve the stream should depend on what marks the 
evolution of the stream (processing time, watermarks/event time, ingestion 
time, window time partitions…).

 In this JIRA issue the evolution would be marked by the processing time. For 
this implementation the operator would work based on option 2. Hence at every 
moment the state of the operator that holds one value can evolve with the last 
elements. In this way the logic of the inner query is to select always the last 
element (fields, or other query related transformations based on the last 
value). This behavior is needed in many scenarios: (e.g., the typical problem 
of computing the total income, when incomes are in multiple currencies and the 
total needs to be computed in one currency by using always the last exchange 
rate).
This behavior is motivated also by the functionality of the 3rd SQL query 
example – Q3  (using inner query as the input source for FROM ). In such 
scenarios, the selection in the main query would need to be done based on 
latest elements. Therefore with such a behavior the 2 types of queries (Q1 and 
Q3) would provide the same, intuitive result.


**Functionality example**
Based on the logical translation plan, we exemplify next the behavior of the 
inner query applied on 2 streams that operate on processing time.
SELECT amount, (SELECT exchange FROM inputstream1) AS field1 FROM inputstream2

 



  
Time
Stream1
Stream2
Output
  
  
T1

1.2

  
  
T2
User1,10

(10,1.2)
  
  
T3
User2,11

(11,1.2)
  
  
T4

1.3

  
  
T5
User3,9

(9,1.3)
  
  
...
  


Note 1. For streams that would operate on event time, at moment T3 we would 
need to retract the previous outputs ((10, 1.2), (11,1.2) ) and reemit them as 
((10,1.3), (11,1.3) ). 

Note 2. Rather than failing when a new value comes in the 

Re: Bumping API stability check version

[Greg Hogan]

Robert,

I was going to file a ticket but posted to the mailing list and was hoping for 
your input.

That may be the wrong configuration option but what I was thinking is that:
- 1.3.0 should have a compatible API relative to 1.2.0 (no modifications, only 
additions)
- 1.2.1+ should have the same API as 1.2.0 (no modifications or additions) 
unless explicitly overridden

Do you think these configuration changes can be scripted as part of the release 
process?

Greg


> On Mar 16, 2017, at 12:29 PM, Robert Metzger  wrote:
> 
> Hi Greg,
> 
> I was not able to update the version to 1.2.0 when updating to
> 1.3-SNAPSHOT, because 1.2.0 was not released at that point in time :)
> But I agree that we should update that version once the release is out. I
> forgot to do it.
> 
> I'm not sure if I completely understand "breakBuildOnModifications". Does
> it fail the build even if somebody did a change that is non API breaking on
> a @Public class?
> 
> On Thu, Mar 16, 2017 at 3:37 PM, Greg Hogan  wrote:
> 
>> Hi,
>> 
>> I see in the parent pom.xml that 1.3-SNAPSHOT is checking for API
>> stability against 1.1.4. Also, that this version was only bumped with
>> FLINK-5617 late in the 1.2 development cycle.
>> 
>> Should we bump this version as part of the release process, i.e. on the
>> 1.2.0 release updating 1.3-SNAPSHOT to verify against 1.2.0? And should we
>> be setting ‘breakBuildOnModifications’ [0] to true for patch releases
>> (overridden when necessary)?
>> 
>> [0] https://siom79.github.io/japicmp/MavenPlugin.html
>> 
>> Greg

[jira] [Created] (FLINK-6071) Savepoints should not count in the number of retained checkpoints

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-6071:
---

 Summary: Savepoints should not count in the number of retained 
checkpoints
 Key: FLINK-6071
 URL: https://issues.apache.org/jira/browse/FLINK-6071
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Affects Versions: 1.2.0
Reporter: Stephan Ewen
 Fix For: 1.3.0


The checkpoint store can retain a number of checkpoints for fallback when one 
of them cannot be restored.

I suggest to not count savepoints among those, as savepoints are assumed to be 
deleted outside the control of the system and may thus not actually be 
available any more, even though referenced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6072) TestCase CheckpointStateRestoreTest::testSetState should be fail

[Guowei Ma (JIRA)]

Guowei Ma created FLINK-6072:


 Summary: TestCase CheckpointStateRestoreTest::testSetState should 
be fail
 Key: FLINK-6072
 URL: https://issues.apache.org/jira/browse/FLINK-6072
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Reporter: Guowei Ma
Assignee: Guowei Ma
Priority: Minor


This case should be fail because three KeyGroupStateHandles which have same  
KeyGroupRange[0,0] are received by 'CheckpointCoordinator'.

1. In this case these ‘statefulExec’s should use three different KeyGroupRanges 
such as [0,0],[1,1],[2,2].

2. Add another test case which can verify 'CheckpointCoordiantor' throws a 
exception when receive overlapped KeyGroupRange.
 
 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6088) java.lang.NoSuchFieldError: INSTANCE Issue with classloading

[prabhat kumar (JIRA)]

prabhat kumar created FLINK-6088:


 Summary: java.lang.NoSuchFieldError: INSTANCE Issue with 
classloading
 Key: FLINK-6088
 URL: https://issues.apache.org/jira/browse/FLINK-6088
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 1.3.0
 Environment: Mac OS, maven 3.3.9
Reporter: prabhat kumar
Priority: Critical


I am at latest code with respect to master and using scala 2.11 so made the 
changes to code using flink tool.
Issue has resurfaced "java.lang.NoSuchFieldError: INSTANCE" for httpcompnent 
httpclient 4.5.2.
I followed the instruction 
mvn clean install -DskipTests at the parent dir
cd flink-dist
mvn clean install -DskipTests 
but still getting the issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6089) Implement decoration phase for rewriting predicated logical plan after volcano optimization phase

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6089:


 Summary: Implement decoration phase for rewriting predicated 
logical plan after volcano optimization phase
 Key: FLINK-6089
 URL: https://issues.apache.org/jira/browse/FLINK-6089
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


At present, there is no chance to modify the DataStreamRel tree after the 
volcano optimization. We consider to add a decoration phase after volcano 
optimization phase. Decoration phase is dedicated for rewriting predicated 
logical plan and is independent of cost module. After decoration phase is 
added, we get the chance to apply retraction rules at this phase.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6090) Implement optimizer for retraction and turn on retraction for over window aggregate

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6090:


 Summary: Implement optimizer for retraction and turn on retraction 
for over window aggregate
 Key: FLINK-6090
 URL: https://issues.apache.org/jira/browse/FLINK-6090
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


Implement optimizer for retraction and turn on the retraction for over window 
as the first prototype example:
  1.Add RetractionRule at the stage of decoration，which can derive the replace 
table/append table, NeedRetraction property.
  2. Match the NeedRetraction and replace table, mark the accumulating mode; 
Add the necessary retract generate function at the replace table, and add the 
retract process logic at the retract consumer
  3. turn on retraction for over window aggregate




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6094) Implement and turn on retraction for stream-stream inner join

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6094:


 Summary: Implement and turn on retraction for stream-stream inner 
join
 Key: FLINK-6094
 URL: https://issues.apache.org/jira/browse/FLINK-6094
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


This includes:
Modify the RetractionRule to consider stream-stream inner join
Implement the retract process logic for join



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6092) Implement and turn on retraction for table source

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6092:


 Summary: Implement and turn on retraction for table source 
 Key: FLINK-6092
 URL: https://issues.apache.org/jira/browse/FLINK-6092
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


Add the Primary Key and replace/append properties for table source, and 
consider table source in optimizer RetractionRule 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6093) Implement and turn on retraction for table sink

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6093:


 Summary: Implement and turn on retraction for table sink 
 Key: FLINK-6093
 URL: https://issues.apache.org/jira/browse/FLINK-6093
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


Add sink tableInsert and NeedRetract property, and consider table sink in 
optimizer RetractionRule



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6091) Implement and turn on the retraction for grouping window aggregate

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6091:


 Summary: Implement and turn on the retraction for grouping window 
aggregate
 Key: FLINK-6091
 URL: https://issues.apache.org/jira/browse/FLINK-6091
 Project: Flink
  Issue Type: Sub-task
Reporter: Shaoxuan Wang
Assignee: Shaoxuan Wang


Implement the functions for processing retract message for grouping window 
aggregate. No retract generating function needed as for now, as the current 
grouping window aggregates are all executed at “without early firing mode”.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)