Re: Towards a spec for robust streaming SQL, Part 1

[Shaoxuan Wang]

Tyler,
Yes, dynamic table changes over time. You can find more details about
dynamic table from this Flink blog (
https://flink.apache.org/news/2017/04/04/dynamic-tables.html). Fabian, me
and Xiaowei posted it a week before the flink-forward@SF.
"A dynamic table is a table that is continuously updated and can be queried
like a regular, static table. However, in contrast to a query on a batch
table which terminates and returns a static table as result, a query on a
dynamic table runs continuously and produces a table that is continuously
updated depending on the modification on the input table. Hence, the
resulting table is a dynamic table as well."

Regards,
Shaoxuan


On Sat, May 13, 2017 at 1:05 AM, Tyler Akidau 
wrote:

> Being able to support an EMIT config independent of the query itself sounds
> great for compatible use cases (which should be many :-).
>
> Shaoxuan, can you please refresh my memory what a dynamic table means in
> Flink? It's basically just a state table, right? The "dynamic" part of the
> name is to simply to imply it can evolve and change over time?
>
> -Tyler
>
>
> On Fri, May 12, 2017 at 1:59 AM Shaoxuan Wang  wrote:
>
> > Thanks to Tyler and Fabian for sharing your thoughts.
> >
> > Regarding to the early/late update control of FLINK. IMO, each dynamic
> > table can have an EMIT config. For FLINK table-API, this can be easily
> > implemented in different manners, case by case. For instance, in window
> > aggregate, we could define "when EMIT a result" via a windowConf per each
> > window when we create windows. Unfortunately we do not have such
> > flexibility (as compared with TableAPI) in SQL query, we need find a way
> to
> > embed this EMIT config.
> >
> > Regards,
> > Shaoxuan
> >
> >
> > On Fri, May 12, 2017 at 4:28 PM, Fabian Hueske 
> wrote:
> >
> > > 2017-05-11 7:14 GMT+02:00 Tyler Akidau :
> > >
> > > > On Tue, May 9, 2017 at 3:06 PM Fabian Hueske 
> > wrote:
> > > >
> > > > > Hi Tyler,
> > > > >
> > > > > thank you very much for this excellent write-up and the super nice
> > > > > visualizations!
> > > > > You are discussing a lot of the things that we have been thinking
> > about
> > > > as
> > > > > well from a different perspective.
> > > > > IMO, yours and the Flink model are pretty much aligned although we
> > use
> > > a
> > > > > different terminology (which is not yet completely established). So
> > > there
> > > > > should be room for unification ;-)
> > > > >
> > > >
> > > > Good to hear, thanks for giving it a look. :-)
> > > >
> > > >
> > > > > Allow me a few words on the current state in Flink. In the upcoming
> > > 1.3.0
> > > > > release, we will have support for group window (TUMBLE, HOP,
> > SESSION),
> > > > OVER
> > > > > RANGE/ROW window (without FOLLOWING), and non-windowed GROUP BY
> > > > > aggregations. The group windows are triggered by watermark and the
> > over
> > > > > window and non-windowed aggregations emit for each input record
> > > > > (AtCount(1)). The window aggregations do not support early or late
> > > firing
> > > > > (late records are dropped), so no updates here. However, the
> > > non-windowed
> > > > > aggregates produce updates (in acc and acc/retract mode). Based on
> > this
> > > > we
> > > > > will work on better control for late updates and early firing as
> well
> > > as
> > > > > joins in the next release.
> > > > >
> > > >
> > > > Impressive. At this rate there's a good chance we'll just be doing
> > > catchup
> > > > and thanking you for building everything. ;-) Do you have ideas for
> > what
> > > > you want your early/late updates control to look like? That's one of
> > the
> > > > areas I'd like to see better defined for us. And how deep are you
> going
> > > > with joins?
> > > >
> > >
> > > Right now (well actually I merged the change 1h ago) we are using a
> > > QueryConfig object to specify state retention intervals to be able to
> > clean
> > > up state for inactive keys.
> > > A running a query looks like this:
> > >
> > > // ---
> > > val env = StreamExecutionEnvironment.getExecutionEnvironment
> > > val tEnv = TableEnvironment.getTableEnvironment(env)
> > >
> > > val qConf = tEnv.queryConfig.withIdleStateRetentionTime(
> Time.hours(12))
> > //
> > > state of inactive keys is kept for 12 hours
> > >
> > > val t: Table = tEnv.sql("SELECT a, COUNT(*) AS cnt FROM MyTable GROUP
> BY
> > > a")
> > > val stream: DataStream[(Boolean, Row)] = t.toRetractStream[Row](qConf)
> //
> > > boolean flag for acc/retract
> > >
> > > env.execute()
> > > // ---
> > >
> > > We plan to use the QueryConfig also to specify early/late updates. Our
> > main
> > > motivation is to have a uniform and standard SQL for batch and
> streaming.
> > > Hence, we have to move the configuration out of the query. But I agree,
> > > that it would be very nice to be able to include it in the query. I
> think
> > > it should 

[jira] [Created] (FLINK-6574) External catalog should support a single level catalog

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-6574:
-

 Summary: External catalog should support a single level catalog
 Key: FLINK-6574
 URL: https://issues.apache.org/jira/browse/FLINK-6574
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: 1.3.0


We found out that the current external catalog requires three layers of 
references for any tables. For example, the SQL would look like the following 
when referencing external table:

{noformat}
SELECT * FROM catalog.db.table
{noformat}

It would be great to support only two layers of indirections which is closer to 
many of the deployment on Presto / Hive today.

{noformat}
SELECT * FROM db.table
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6573) Flink MongoDB Connector

[Nagamallikarjuna (JIRA)]

Nagamallikarjuna created FLINK-6573:
---

 Summary: Flink MongoDB Connector
 Key: FLINK-6573
 URL: https://issues.apache.org/jira/browse/FLINK-6573
 Project: Flink
  Issue Type: New Feature
  Components: Batch Connectors and Input/Output Formats
Affects Versions: 1.2.0
 Environment: Linux Operating System, Mongo DB
Reporter: Nagamallikarjuna
Priority: Minor
 Fix For: 2.0.0


Hi Community,

Currently we are using Flink in the current Project. We have huge amount of 
data to process using Flink which resides in Mongo DB. We have a requirement of 
parallel data connectivity in between Flink and Mongo DB for both reads/writes. 
Currently we are planning to create this connector and contribute to the 
Community.

I will update the further details once I receive your feedback 

Please let us know if you have any concerns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Towards a spec for robust streaming SQL, Part 1

[Tyler Akidau]

Being able to support an EMIT config independent of the query itself sounds
great for compatible use cases (which should be many :-).

Shaoxuan, can you please refresh my memory what a dynamic table means in
Flink? It's basically just a state table, right? The "dynamic" part of the
name is to simply to imply it can evolve and change over time?

-Tyler


On Fri, May 12, 2017 at 1:59 AM Shaoxuan Wang  wrote:

> Thanks to Tyler and Fabian for sharing your thoughts.
>
> Regarding to the early/late update control of FLINK. IMO, each dynamic
> table can have an EMIT config. For FLINK table-API, this can be easily
> implemented in different manners, case by case. For instance, in window
> aggregate, we could define "when EMIT a result" via a windowConf per each
> window when we create windows. Unfortunately we do not have such
> flexibility (as compared with TableAPI) in SQL query, we need find a way to
> embed this EMIT config.
>
> Regards,
> Shaoxuan
>
>
> On Fri, May 12, 2017 at 4:28 PM, Fabian Hueske  wrote:
>
> > 2017-05-11 7:14 GMT+02:00 Tyler Akidau :
> >
> > > On Tue, May 9, 2017 at 3:06 PM Fabian Hueske 
> wrote:
> > >
> > > > Hi Tyler,
> > > >
> > > > thank you very much for this excellent write-up and the super nice
> > > > visualizations!
> > > > You are discussing a lot of the things that we have been thinking
> about
> > > as
> > > > well from a different perspective.
> > > > IMO, yours and the Flink model are pretty much aligned although we
> use
> > a
> > > > different terminology (which is not yet completely established). So
> > there
> > > > should be room for unification ;-)
> > > >
> > >
> > > Good to hear, thanks for giving it a look. :-)
> > >
> > >
> > > > Allow me a few words on the current state in Flink. In the upcoming
> > 1.3.0
> > > > release, we will have support for group window (TUMBLE, HOP,
> SESSION),
> > > OVER
> > > > RANGE/ROW window (without FOLLOWING), and non-windowed GROUP BY
> > > > aggregations. The group windows are triggered by watermark and the
> over
> > > > window and non-windowed aggregations emit for each input record
> > > > (AtCount(1)). The window aggregations do not support early or late
> > firing
> > > > (late records are dropped), so no updates here. However, the
> > non-windowed
> > > > aggregates produce updates (in acc and acc/retract mode). Based on
> this
> > > we
> > > > will work on better control for late updates and early firing as well
> > as
> > > > joins in the next release.
> > > >
> > >
> > > Impressive. At this rate there's a good chance we'll just be doing
> > catchup
> > > and thanking you for building everything. ;-) Do you have ideas for
> what
> > > you want your early/late updates control to look like? That's one of
> the
> > > areas I'd like to see better defined for us. And how deep are you going
> > > with joins?
> > >
> >
> > Right now (well actually I merged the change 1h ago) we are using a
> > QueryConfig object to specify state retention intervals to be able to
> clean
> > up state for inactive keys.
> > A running a query looks like this:
> >
> > // ---
> > val env = StreamExecutionEnvironment.getExecutionEnvironment
> > val tEnv = TableEnvironment.getTableEnvironment(env)
> >
> > val qConf = tEnv.queryConfig.withIdleStateRetentionTime(Time.hours(12))
> //
> > state of inactive keys is kept for 12 hours
> >
> > val t: Table = tEnv.sql("SELECT a, COUNT(*) AS cnt FROM MyTable GROUP BY
> > a")
> > val stream: DataStream[(Boolean, Row)] = t.toRetractStream[Row](qConf) //
> > boolean flag for acc/retract
> >
> > env.execute()
> > // ---
> >
> > We plan to use the QueryConfig also to specify early/late updates. Our
> main
> > motivation is to have a uniform and standard SQL for batch and streaming.
> > Hence, we have to move the configuration out of the query. But I agree,
> > that it would be very nice to be able to include it in the query. I think
> > it should not be difficult to optionally support an EMIT clause as well.
> >
> >
> > >
> > > > Reading the document, I did not find any major difference in our
> > > concepts.
> > > > In fact, we are aiming to support the cases you are describing as
> well.
> > > > I have a question though. Would you classify an OVER aggregation as a
> > > > stream -> stream or stream -> table operation? It collects records to
> > > > aggregate them, but emits one record for each input row. Depending on
> > the
> > > > window definition (i.e., with FOLLOWING CURRENT ROW), you can compute
> > and
> > > > emit the result record when the input record is received.
> > > >
> > >
> > > I would call it a composite stream → stream operation (since SQL, like
> > the
> > > standard Beam/Flink APIs, is a higher level set of constructs than raw
> > > streams/tables operations) consisting of a stream → table windowed
> > grouping
> > > followed by a table → stream triggering on every element, basically as
> > you
> > > 

[jira] [Created] (FLINK-6572) Make Flink configuration properties available as env vars in Docker image

[Joshua Griffith (JIRA)]

Joshua Griffith created FLINK-6572:
--

 Summary: Make Flink configuration properties available as env vars 
in Docker image
 Key: FLINK-6572
 URL: https://issues.apache.org/jira/browse/FLINK-6572
 Project: Flink
  Issue Type: Improvement
  Components: Docker
Affects Versions: 1.2.1
Reporter: Joshua Griffith
Priority: Minor


We run Flink in containers in production and it has been convenient to have all 
of Flink's configuration options available as environment variables so a 
separate configuration volume is not required. In our image we [grab the 
options from the config 
documentation|https://github.com/orgsync/docker-flink/blob/2abc0bf2b93238e4d1895d4ada0e8f6a72b577cf/Dockerfile#L33-L40]
 and [load them into the configuration file in the 
entrypoint|https://github.com/orgsync/docker-flink/blob/2abc0bf2b93238e4d1895d4ada0e8f6a72b577cf/entrypoint.sh#L32-L50].
 While this method is not ideal, I think this feature would be useful in the 
official Docker image.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6571) InfiniteSource in SourceStreamOperatorTest should deal with InterruptedExceptions

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6571:
---

 Summary: InfiniteSource in SourceStreamOperatorTest should deal 
with InterruptedExceptions
 Key: FLINK-6571
 URL: https://issues.apache.org/jira/browse/FLINK-6571
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.3.0, 1.4.0
Reporter: Chesnay Schepler
Priority: Minor


So this is a new one: i got a failing test ({{testNoMaxWatermarkOnAsyncStop}}) 
due to an uncatched InterruptedException.

{code}
[00:28:15] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
0.828 sec <<< FAILURE! - in 
org.apache.flink.streaming.runtime.operators.StreamSourceOperatorTest
[00:28:15] 
testNoMaxWatermarkOnAsyncStop(org.apache.flink.streaming.runtime.operators.StreamSourceOperatorTest)
  Time elapsed: 0 sec  <<< ERROR!
[00:28:15] java.lang.InterruptedException: sleep interrupted
[00:28:15]  at java.lang.Thread.sleep(Native Method)
[00:28:15]  at 
org.apache.flink.streaming.runtime.operators.StreamSourceOperatorTest$InfiniteSource.run(StreamSourceOperatorTest.java:343)
[00:28:15]  at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
[00:28:15]  at 
org.apache.flink.streaming.runtime.operators.StreamSourceOperatorTest.testNoMaxWatermarkOnAsyncStop(StreamSourceOperatorTest.java:176)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6570) QueryableStateClient constructor in documentation doesn't match actual signature

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-6570:
-

 Summary: QueryableStateClient constructor in documentation doesn't 
match actual signature
 Key: FLINK-6570
 URL: https://issues.apache.org/jira/browse/FLINK-6570
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Queryable State
Reporter: Robert Metzger


The queryable state documentation states to use the following ctor

{code}
QueryableStateClient client = new QueryableStateClient(config);
{code}

However, this ctor doesn't exist anymore (since 1.3)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Operator name

[Flavio Pompermaier]

Hi to all,
in many of my Flink job it is helpful to give a name to operators in order
to make JobManager UI simpler to read.
Currently, to give a name to operators it is necessary to specify it of the
operator everytime it is used, for example:

   - env.readAsCsv().map().name("My map function")

Wouldn't be more convinient to allow an operator to override the default
name (at least for rich ones) by adding to *RichFunction* something like

   - public abstract String getOperatorName()

What do you think?

Best,
Flavio
-- 
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 1823908

[jira] [Created] (FLINK-6569) flink-table KafkaJsonTableSource example doesn't work

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-6569:
-

 Summary: flink-table KafkaJsonTableSource example doesn't work
 Key: FLINK-6569
 URL: https://issues.apache.org/jira/browse/FLINK-6569
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Table API & SQL
Affects Versions: 1.3.0
Reporter: Robert Metzger


The code example uses 
{code}
TypeInformation typeInfo = Types.ROW(
  new String[] { "id", "name", "score" },
  new TypeInformation[] { Types.INT(), Types.STRING(), Types.DOUBLE() }
);
{code}

the correct way of using it is something like
{code}
TypeInformation typeInfo = Types.ROW_NAMED(
new String[] { "id", "zip", "date" },
Types.LONG, Types.INT, Types.SQL_DATE);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

[Robert Metzger]

I'm considering to do another non voting RC1 on Monday with the incremental
checkpointing and maven fixes in (and of course as many other things as
possible). Once the other critical fixes are in (ideally Tuesday or
Wednesday), I'll create the first voting one.

The current RC0 is almost unusable because of the incorrect shading.

On Fri, May 12, 2017 at 2:09 PM, Greg Hogan  wrote:

> +1 for sticking to the code freeze deadline and building a new release
> candidate but since the release is still two weeks off (5/26) I think it
> better to delay voting to give time for additional bug fixes.
>
>
> > On May 11, 2017, at 10:19 AM, Robert Metzger 
> wrote:
> >
> > It seems that we found quite a large number of critical issues in the
> first
> > RC.
> >
> > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan
> has
> > a PR open to fix the critical ones)
> > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has a
> > pending PR)
> > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> > issues (status unknown)
> > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan is on
> > it)
> > - FLINK-6508 Include license files of packaged dependencies (Stephan is
> on
> > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR files
> > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > ZooKeeperCompletedCheckpointStore (unknown)
> >
> > I would like to get these issues fixed by end of this week (Sunday), so
> > that I can create the first voting RC on Monday morning.
> > Please reject if you think we will not manage to get the stuff fixed
> until
> > then.
> >
> >
> >
> > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann 
> > wrote:
> >
> >> Unfortunately, it won't be fully functional in 1.3.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, May 11, 2017 at 10:45 AM, Renjie Liu 
> >> wrote:
> >>
> >>> @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
> >>>
> >>> On Thu, May 11, 2017 at 4:12 PM Gyula Fóra 
> wrote:
> >>>
>  Thanks Stefan!
>  Gyula
> 
>  Stefan Richter  ezt írta (időpont: 2017.
> >>> máj.
>  11., Cs, 10:04):
> 
> >
> > Hi,
> >
> > Thanks for reporting this. I found a couple of issues yesterday and I
> >>> am
> > currently working on a bundle of fixes. I will take a look at this
>  problem,
> > and if it is already covered.
> >
> > Best,
> > Stefan
> >
> >> Am 11.05.2017 um 09:47 schrieb Gyula Fóra :
> >>
> >> Hi,
> >> I am not sure if this belong to this thread, but while trying to
> >> run
> >>> a
> > job
> >> with rocks incremental backend I ran into 2 issues:
> >>
> >> One with savepoints, I can't figure out because I can't make sense
> >> of
>  the
> >> error or how it happenned:
> >> The error stack trace is here:
> >> https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239
> >> This happens on every savepoint attempt and seems to be related to
> >>> the
> >> kafka source. Interestingly other tasks succeed in writing data to
>  hdfs.
> >>
> >> The other one is covered by
> > https://issues.apache.org/jira/browse/FLINK-6531 I
> >> guess. I am not sure if the first one is related though.
> >>
> >> Thank you!
> >> Gyula
> >>
> >> Till Rohrmann  ezt írta (időpont: 2017. máj.
>  11.,
> > Cs,
> >> 9:14):
> >>
> >>> Hi Renjie,
> >>>
> >>> 1.3 already contains some Flip-6 code. However, it is not yet
> >> fully
> >>> functional. What you already can do is to run local jobs on the
> >>> Flip-6
> > code
> >>> base by instantiating a MiniCluster and then using the
> >>> Flip6LocalStreamEnvironment.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Thu, May 11, 2017 at 6:00 AM, Renjie Liu <
> >>> liurenjie2...@gmail.com>
> >>> wrote:
> >>>
>  Hi, all:
>  Will the FLIP 6 be included in release 1.3?
> 
>  On Wed, May 10, 2017 at 9:48 PM Gyula Fóra  >>>
> > wrote:
> 
> > Thanks you! :)
> >
> > Chesnay Schepler  ezt írta (időpont: 2017.
> >>> máj.
> >>> 10.,
> > Sze, 15:44):
> >
> >> I guess it is related to this one
> >> https://issues.apache.org/jira/browse/FLINK-6514 ?
> >>
> >> On 10.05.2017 15:34, Gyula Fóra wrote:
> >>> Hi,
> >>>
> >>> I tried to run an application on 1.3 but I keep getting the
> >>> following
> >> error:
> >>> java.lang.NoClassDefFoundError: Could not initialize class
> >>> org.apache.hadoop.security.UserGroupInformation
> >>> at
> >>>
> >>
> 

[jira] [Created] (FLINK-6568) flink-table doesn't work without flink-streaming-scala dependency

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-6568:
-

 Summary: flink-table doesn't work without flink-streaming-scala 
dependency
 Key: FLINK-6568
 URL: https://issues.apache.org/jira/browse/FLINK-6568
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Table API & SQL
Affects Versions: 1.3.0
Reporter: Robert Metzger


In my user application, I got errors because I didn't have the 
flink-streaming-scala dependency defined in my user code (and I'm using Java).

The documentation should be updated or flink-streaming-scala should not be a 
provided dependency.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6567) ExecutionGraphMetricsTest fails on Windows CI

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6567:
---

 Summary: ExecutionGraphMetricsTest fails on Windows CI
 Key: FLINK-6567
 URL: https://issues.apache.org/jira/browse/FLINK-6567
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.3.0, 1.4.0
Reporter: Chesnay Schepler
Priority: Minor


The {{testExecutionGraphRestartTimeMetric}} fails every time i run it on 
AppVeyor. It also very rarely failed for me locally.

The test fails at Line 235 if the RUNNING timestamp is equal to the RESTARTING 
timestamp, which may happen when combining a fast test with a low resolution 
clock.

A simple fix would be to increase the timestamp between RUNNING and RESTARTING 
by adding a 50ms sleep timeout into the {{TestingRestartStrategy#canRestart()}} 
method, as this one is called before transitioning to the RESTARTING state.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

[Greg Hogan]

+1 for sticking to the code freeze deadline and building a new release 
candidate but since the release is still two weeks off (5/26) I think it better 
to delay voting to give time for additional bug fixes.


> On May 11, 2017, at 10:19 AM, Robert Metzger  wrote:
> 
> It seems that we found quite a large number of critical issues in the first
> RC.
> 
> - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan has
> a PR open to fix the critical ones)
> - FLINK-6531 Deserialize checkpoint hooks with user classloader (has a
> pending PR)
> - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> issues (status unknown)
> - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan is on
> it)
> - FLINK-6508 Include license files of packaged dependencies (Stephan is on
> it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR files
> - FLINK-6284 Incorrect sorting of completed checkpoints in
> ZooKeeperCompletedCheckpointStore (unknown)
> 
> I would like to get these issues fixed by end of this week (Sunday), so
> that I can create the first voting RC on Monday morning.
> Please reject if you think we will not manage to get the stuff fixed until
> then.
> 
> 
> 
> On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann 
> wrote:
> 
>> Unfortunately, it won't be fully functional in 1.3.
>> 
>> Cheers,
>> Till
>> 
>> On Thu, May 11, 2017 at 10:45 AM, Renjie Liu 
>> wrote:
>> 
>>> @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
>>> 
>>> On Thu, May 11, 2017 at 4:12 PM Gyula Fóra  wrote:
>>> 
 Thanks Stefan!
 Gyula
 
 Stefan Richter  ezt írta (időpont: 2017.
>>> máj.
 11., Cs, 10:04):
 
> 
> Hi,
> 
> Thanks for reporting this. I found a couple of issues yesterday and I
>>> am
> currently working on a bundle of fixes. I will take a look at this
 problem,
> and if it is already covered.
> 
> Best,
> Stefan
> 
>> Am 11.05.2017 um 09:47 schrieb Gyula Fóra :
>> 
>> Hi,
>> I am not sure if this belong to this thread, but while trying to
>> run
>>> a
> job
>> with rocks incremental backend I ran into 2 issues:
>> 
>> One with savepoints, I can't figure out because I can't make sense
>> of
 the
>> error or how it happenned:
>> The error stack trace is here:
>> https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239
>> This happens on every savepoint attempt and seems to be related to
>>> the
>> kafka source. Interestingly other tasks succeed in writing data to
 hdfs.
>> 
>> The other one is covered by
> https://issues.apache.org/jira/browse/FLINK-6531 I
>> guess. I am not sure if the first one is related though.
>> 
>> Thank you!
>> Gyula
>> 
>> Till Rohrmann  ezt írta (időpont: 2017. máj.
 11.,
> Cs,
>> 9:14):
>> 
>>> Hi Renjie,
>>> 
>>> 1.3 already contains some Flip-6 code. However, it is not yet
>> fully
>>> functional. What you already can do is to run local jobs on the
>>> Flip-6
> code
>>> base by instantiating a MiniCluster and then using the
>>> Flip6LocalStreamEnvironment.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Thu, May 11, 2017 at 6:00 AM, Renjie Liu <
>>> liurenjie2...@gmail.com>
>>> wrote:
>>> 
 Hi, all:
 Will the FLIP 6 be included in release 1.3?
 
 On Wed, May 10, 2017 at 9:48 PM Gyula Fóra >> 
> wrote:
 
> Thanks you! :)
> 
> Chesnay Schepler  ezt írta (időpont: 2017.
>>> máj.
>>> 10.,
> Sze, 15:44):
> 
>> I guess it is related to this one
>> https://issues.apache.org/jira/browse/FLINK-6514 ?
>> 
>> On 10.05.2017 15:34, Gyula Fóra wrote:
>>> Hi,
>>> 
>>> I tried to run an application on 1.3 but I keep getting the
>>> following
>> error:
>>> java.lang.NoClassDefFoundError: Could not initialize class
>>> org.apache.hadoop.security.UserGroupInformation
>>> at
>>> 
>> 
> org.apache.flink.runtime.security.modules.HadoopModule.
 install(HadoopModule.java:45)
>>> at
>>> 
>> 
> org.apache.flink.runtime.security.SecurityUtils.
 install(SecurityUtils.java:78)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.
>>> java:1128)
>>> 
>>> Even after adding hadoop-common to the lib manually (which I
>>> guess
> should
>>> not be necessary).
>>> 
>>> Any idea what might cause this?
>>> 
>>> Thanks,
>>> Gyula
>>> 
>>> Chesnay Schepler  ezt írta (időpont:
>> 

RE: [DISCUSS] Expose State Backend Interface for UDAGG

[Radu Tudoran]

Hi,

In general I believe it is a good idea to expose the state backend to the 
functions. You can always optimize the data processing based on the data 
storage. Hence, as the level of the processing (aggregation here) you would be 
able to control the access to data, you can implement this in a smart way. 
Moreover, we can also construct different data organizations/partition 
strategy/etc based on the specific computation. I understand that this would be 
quite an effort, but at some point it is worth making it.

Meanwhile if it would not be possible to have the aggregation function 
extending the rich interface, wouldn't we be able to supplement this with some 
extra logic in the process function that would provide the aggregates the 
needed data or at least pointers to the required state?

As far as I know it would be legal now to have something like:

ProcessFunction () {

ValueState state = ...

processElement(newElement) {

acc.accumulate(newElement, state)

}
}

WeightedAvgAccum {

public void accumulate(Row newElement, ValueState state) {

state.value
}
}
Would something like this at least partially solve the problem? ...it would 
allow you to manage the intermediate data directly in the state instead of the 
memory


-Original Message-
From: Shaoxuan Wang [mailto:wshaox...@gmail.com] 
Sent: Friday, May 12, 2017 1:20 PM
To: Dev
Cc: Stephan Ewen
Subject: Re: [DISCUSS] Expose State Backend Interface for UDAGG

Fabian,
Thanks for your quick reply.
The goal of "FLINK-6544" is not to expose state backend in all UDAGG cases.
It is designed to provide an interface which provides an ability for user to 
access state backend when it is allowed (yes, right now this is only allowed by 
ProcessFunction).  This interface itself does not make the things better. 
Instead, it provides a generic interface for the future adoption of exposing 
backend state in all different UDAGG cases, and the current over Aggregate and 
unbounded group aggregate can enjoy the benefits of accessing state backend.

In the meanwhile, I am also curious why we cannot build AggregateFunction on 
RichFunction. We will lose lots of benefit of having state backend for window 
Aggregate if it does not provide runtime context.
@Stephan It is really appreciate if you can share the concerns or blocking 
reasons of not having AggregateFunction designed on top of RichFunction.

Regards,
Shaoxuan


On Fri, May 12, 2017 at 6:21 PM, Fabian Hueske  wrote:

> Hi, thanks for the proposal.
>
> I think exposing state to UDAGGs would be very difficult and a lot of work.
>
> UDAGGs are called from ProcessFunctions (stream, OVER window and 
> non-window aggs), AggregateFunctions (stream, group-window aggs), 
> CombineFunctions
> (batch) and GroupReduceFunctions (batch). The batch functions do not 
> support state backends at all, ProcessFunctions can register state, 
> and AggregateFunction cannot.
> Even when putting the batch case aside this is very hard.
>
> AggregateFunctions support merging of windows. Right now, this only 
> involves merging of accumulators. If we allow AggregateFunctions to 
> have state, we would also need to provide logic to merge the state. 
> Moreover, it is not clearly defined when AggregateFunctions are called 
> (similar to Combiners in MapReduce) which would make state handling very 
> complex.
> Changing this would be a major effort in the DataStream API.
>
> An alternative would be to reimplement the group-window logic in the 
> Table API, but this will he a huge effort as well (maybe we have to do 
> it anyway at some point though).
>
> @Stephan knows more about the implications of allowing state in 
> AggregateFunctions.
>
> Best, Fabian
>
> 2017-05-12 11:53 GMT+02:00 Shaoxuan Wang :
>
> > Hi everyone,
> >
> > We made some progress in the implementation of UDAGG (FLINK-5564).
> However,
> > we realized that there are cases where users may want to use state
> backend
> > to store the data. For instance, the built-in 
> > MaxWithRetractAggFunction currently create a hashMap to store the 
> > historical data. It will have problem when the # of keys are huge enough, 
> > thereby leading to OOM.
> >
> > In FLINK-6544, we have proposed an approach to expose State Backend 
> > Interface for UDAGG. A brief design doc can be found in
> > https://docs.google.com/document/d/1g-wHOuFj5pMaYMJg90kVIO2IHbiQiO26
> > nWscLIOn50c/edit
> >
> > I am opening this discussion thread, as I realized there are 
> > recently
> some
> > open jiras which are towards to implement some special aggregators, 
> > such
> as
> > "count distinct". IMO, "count distinct" is just an UDAGG. With the 
> > new proposed FLINK-6544, we can just make it as a built-in agg 
> > without
> changing
> > the current UDAGG framework.
> >
> > @Fabian, @Haohui, @Radu, please take a look at FLINK-6544 and let me 
> > know what you think.
> > Btw, we do not need include this change for release 1.3 in our opinion.
> >
> 

Re: waiting on assignment for FLINK-1750

[Alex De Castro]

Thanks!




Alex

On 12 May 2017 at 12:34, Fabian Hueske  wrote:

> Done! :-)
>
> 2017-05-12 11:48 GMT+02:00 Alex De Castro :
>
> > Thanks!
> >
>

Re: waiting on assignment for FLINK-1750

[Fabian Hueske]

Done! :-)

2017-05-12 11:48 GMT+02:00 Alex De Castro :

> Thanks!
>

Re: git overwritten by checkout

[Fabian Hueske]

Sorry, that was me :-/

Thanks for sharing how you resolved the issue.


2017-05-12 12:38 GMT+02:00 Greg Hogan :

> The following file in the following commit throw the following error when
> rebasing to master. I agree that case insensitive filesystems are an
> abomination and hopefully never supported by APFS but just wanted to note
> the situation. With a clean working directory I solved this with `git fetch
> apache master` and `git reset --hard FETCH_HEAD`.
>
> Greg
>
> .../scala/org/apache/flink/table/sources/{DefinedTimeAttributes.scala →
> definedTimeAttributes.scala}
>
> https://github.com/apache/flink/commit/b50ef4b8de73e0e19df154d87ea588
> 236e3ccb43  b50ef4b8de73e0e19df154d87ea588236e3ccb43>
>
> git rebase apache/master
> First, rewinding head to replay your work on top of it...
> error: The following untracked working tree files would be overwritten by
> checkout:
> flink-libraries/flink-table/src/main/scala/org/apache/
> flink/table/sources/definedTimeAttributes.scala
> Please move or remove them before you switch branches.
> Aborting
> could not detach HEAD
>

Re: [DISCUSS] Expose State Backend Interface for UDAGG

[Shaoxuan Wang]

Fabian,
Thanks for your quick reply.
The goal of "FLINK-6544" is not to expose state backend in all UDAGG cases.
It is designed to provide an interface which provides an ability for user
to access state backend when it is allowed (yes, right now this is only
allowed by ProcessFunction).  This interface itself does not make the
things better. Instead, it provides a generic interface for the future
adoption of exposing backend state in all different UDAGG cases, and the
current over Aggregate and unbounded group aggregate can enjoy the benefits
of accessing state backend.

In the meanwhile, I am also curious why we cannot build AggregateFunction
on RichFunction. We will lose lots of benefit of having state backend for
window Aggregate if it does not provide runtime context.
@Stephan It is really appreciate if you can share the concerns or blocking
reasons of not having AggregateFunction designed on top of RichFunction.

Regards,
Shaoxuan


On Fri, May 12, 2017 at 6:21 PM, Fabian Hueske  wrote:

> Hi, thanks for the proposal.
>
> I think exposing state to UDAGGs would be very difficult and a lot of work.
>
> UDAGGs are called from ProcessFunctions (stream, OVER window and non-window
> aggs), AggregateFunctions (stream, group-window aggs), CombineFunctions
> (batch) and GroupReduceFunctions (batch). The batch functions do not
> support state backends at all, ProcessFunctions can register state, and
> AggregateFunction cannot.
> Even when putting the batch case aside this is very hard.
>
> AggregateFunctions support merging of windows. Right now, this only
> involves merging of accumulators. If we allow AggregateFunctions to have
> state, we would also need to provide logic to merge the state. Moreover, it
> is not clearly defined when AggregateFunctions are called (similar to
> Combiners in MapReduce) which would make state handling very complex.
> Changing this would be a major effort in the DataStream API.
>
> An alternative would be to reimplement the group-window logic in the Table
> API, but this will he a huge effort as well (maybe we have to do it anyway
> at some point though).
>
> @Stephan knows more about the implications of allowing state in
> AggregateFunctions.
>
> Best, Fabian
>
> 2017-05-12 11:53 GMT+02:00 Shaoxuan Wang :
>
> > Hi everyone,
> >
> > We made some progress in the implementation of UDAGG (FLINK-5564).
> However,
> > we realized that there are cases where users may want to use state
> backend
> > to store the data. For instance, the built-in MaxWithRetractAggFunction
> > currently create a hashMap to store the historical data. It will have
> > problem when the # of keys are huge enough, thereby leading to OOM.
> >
> > In FLINK-6544, we have proposed an approach to expose State Backend
> > Interface for UDAGG. A brief design doc can be found in
> > https://docs.google.com/document/d/1g-wHOuFj5pMaYMJg90kVIO2IHbiQiO26
> > nWscLIOn50c/edit
> >
> > I am opening this discussion thread, as I realized there are recently
> some
> > open jiras which are towards to implement some special aggregators, such
> as
> > "count distinct". IMO, "count distinct" is just an UDAGG. With the new
> > proposed FLINK-6544, we can just make it as a built-in agg without
> changing
> > the current UDAGG framework.
> >
> > @Fabian, @Haohui, @Radu, please take a look at FLINK-6544 and let me know
> > what you think.
> > Btw, we do not need include this change for release 1.3 in our opinion.
> >
> > Regards,
> > Shaoxuan
> >
>

git overwritten by checkout

[Greg Hogan]

The following file in the following commit throw the following error when 
rebasing to master. I agree that case insensitive filesystems are an 
abomination and hopefully never supported by APFS but just wanted to note the 
situation. With a clean working directory I solved this with `git fetch apache 
master` and `git reset --hard FETCH_HEAD`.

Greg

.../scala/org/apache/flink/table/sources/{DefinedTimeAttributes.scala → 
definedTimeAttributes.scala}

https://github.com/apache/flink/commit/b50ef4b8de73e0e19df154d87ea588236e3ccb43 


git rebase apache/master
First, rewinding head to replay your work on top of it...
error: The following untracked working tree files would be overwritten by 
checkout:

flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/definedTimeAttributes.scala
Please move or remove them before you switch branches.
Aborting
could not detach HEAD

Re: [DISCUSS] Expose State Backend Interface for UDAGG

[Fabian Hueske]

Hi, thanks for the proposal.

I think exposing state to UDAGGs would be very difficult and a lot of work.

UDAGGs are called from ProcessFunctions (stream, OVER window and non-window
aggs), AggregateFunctions (stream, group-window aggs), CombineFunctions
(batch) and GroupReduceFunctions (batch). The batch functions do not
support state backends at all, ProcessFunctions can register state, and
AggregateFunction cannot.
Even when putting the batch case aside this is very hard.

AggregateFunctions support merging of windows. Right now, this only
involves merging of accumulators. If we allow AggregateFunctions to have
state, we would also need to provide logic to merge the state. Moreover, it
is not clearly defined when AggregateFunctions are called (similar to
Combiners in MapReduce) which would make state handling very complex.
Changing this would be a major effort in the DataStream API.

An alternative would be to reimplement the group-window logic in the Table
API, but this will he a huge effort as well (maybe we have to do it anyway
at some point though).

@Stephan knows more about the implications of allowing state in
AggregateFunctions.

Best, Fabian

2017-05-12 11:53 GMT+02:00 Shaoxuan Wang :

> Hi everyone,
>
> We made some progress in the implementation of UDAGG (FLINK-5564). However,
> we realized that there are cases where users may want to use state backend
> to store the data. For instance, the built-in MaxWithRetractAggFunction
> currently create a hashMap to store the historical data. It will have
> problem when the # of keys are huge enough, thereby leading to OOM.
>
> In FLINK-6544, we have proposed an approach to expose State Backend
> Interface for UDAGG. A brief design doc can be found in
> https://docs.google.com/document/d/1g-wHOuFj5pMaYMJg90kVIO2IHbiQiO26
> nWscLIOn50c/edit
>
> I am opening this discussion thread, as I realized there are recently some
> open jiras which are towards to implement some special aggregators, such as
> "count distinct". IMO, "count distinct" is just an UDAGG. With the new
> proposed FLINK-6544, we can just make it as a built-in agg without changing
> the current UDAGG framework.
>
> @Fabian, @Haohui, @Radu, please take a look at FLINK-6544 and let me know
> what you think.
> Btw, we do not need include this change for release 1.3 in our opinion.
>
> Regards,
> Shaoxuan
>

[DISCUSS] Expose State Backend Interface for UDAGG

[Shaoxuan Wang]

Hi everyone,

We made some progress in the implementation of UDAGG (FLINK-5564). However,
we realized that there are cases where users may want to use state backend
to store the data. For instance, the built-in MaxWithRetractAggFunction
currently create a hashMap to store the historical data. It will have
problem when the # of keys are huge enough, thereby leading to OOM.

In FLINK-6544, we have proposed an approach to expose State Backend
Interface for UDAGG. A brief design doc can be found in
https://docs.google.com/document/d/1g-wHOuFj5pMaYMJg90kVIO2IHbiQiO26nWscLIOn50c/edit

I am opening this discussion thread, as I realized there are recently some
open jiras which are towards to implement some special aggregators, such as
"count distinct". IMO, "count distinct" is just an UDAGG. With the new
proposed FLINK-6544, we can just make it as a built-in agg without changing
the current UDAGG framework.

@Fabian, @Haohui, @Radu, please take a look at FLINK-6544 and let me know
what you think.
Btw, we do not need include this change for release 1.3 in our opinion.

Regards,
Shaoxuan

waiting on assignment for FLINK-1750

[Alex De Castro]

Thanks!

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

[Till Rohrmann]

I would like to get https://issues.apache.org/jira/browse/FLINK-6020 also
in the 1.3.0 release, because multiple job submissions to the same cluster
can corrupt the stored blobs.

Cheers,
Till

On Fri, May 12, 2017 at 9:52 AM, Fabian Hueske  wrote:

> Hi Haohui,
>
> open issue for 1.3 are tracked in JIRA.
> You can specify "Fix version: 1.3.0" and an appropriate priority.
>
> I updated a couple of issues to have the fix version tag.
> Any bug that is found while release testing should be added to this list by
> setting fix version, priority, and component.
>
> Thanks,
> Fabian
>
> PS.: This query will list the open Table API issues for 1.3:
>
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20fixVersion%20%3D%201.3.0%20AND%20component%20%3D%20%
> 22Table%20API%20%26%20SQL%22%20ORDER%20BY%20priority%20DESC
>
> 2017-05-12 0:50 GMT+02:00 Haohui Mai :
>
> > Hi,
> >
> > Just curious -- do we have anything to track the blockers on the table /
> > SQL side? For example, it seems to me that FLINK-6563 is a blocker
> because
> > all group windows in SQL will not work without it. I think that Fabian /
> > Timo might want to get several jiras into 1.3.
> >
> > Regards,
> > Haohui
> >
> > On Thu, May 11, 2017 at 12:33 PM Stephan Ewen  wrote:
> >
> > > From the above list, I addressed a bunch of issues.
> > >
> > > - FLINK-6514 Cannot start Flink Cluster in standalone mode has an open
> > Pull
> > > Request:
> > >- master: https://github.com/apache/flink/pull/3876
> > >- 1.3: https://github.com/apache/flink/pull/3877
> > >
> > > Left are:
> > >
> > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan
> > has
> > > PR
> > > open to fix the critical ones)
> > >
> > > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > > ZooKeeperCompletedCheckpointStore
> > >
> > >
> > > *I think that FLINK-6284 is totally a blocker for the release!*
> > >
> > >
> > > On Thu, May 11, 2017 at 4:19 PM, Robert Metzger 
> > > wrote:
> > >
> > > > It seems that we found quite a large number of critical issues in the
> > > first
> > > > RC.
> > > >
> > > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots
> (Stefan
> > > has
> > > > a PR open to fix the critical ones)
> > > > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has
> a
> > > > pending PR)
> > > > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> > > > issues (status unknown)
> > > > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan
> is
> > on
> > > > it)
> > > > - FLINK-6508 Include license files of packaged dependencies (Stephan
> is
> > > on
> > > > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR
> > files
> > > > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > > > ZooKeeperCompletedCheckpointStore (unknown)
> > > >
> > > > I would like to get these issues fixed by end of this week (Sunday),
> so
> > > > that I can create the first voting RC on Monday morning.
> > > > Please reject if you think we will not manage to get the stuff fixed
> > > until
> > > > then.
> > > >
> > > >
> > > >
> > > > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann <
> trohrm...@apache.org>
> > > > wrote:
> > > >
> > > > > Unfortunately, it won't be fully functional in 1.3.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, May 11, 2017 at 10:45 AM, Renjie Liu <
> > liurenjie2...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
> > > > > >
> > > > > > On Thu, May 11, 2017 at 4:12 PM Gyula Fóra  >
> > > > wrote:
> > > > > >
> > > > > > > Thanks Stefan!
> > > > > > > Gyula
> > > > > > >
> > > > > > > Stefan Richter  ezt írta
> (időpont:
> > > > 2017.
> > > > > > máj.
> > > > > > > 11., Cs, 10:04):
> > > > > > >
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Thanks for reporting this. I found a couple of issues
> yesterday
> > > > and I
> > > > > > am
> > > > > > > > currently working on a bundle of fixes. I will take a look at
> > > this
> > > > > > > problem,
> > > > > > > > and if it is already covered.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stefan
> > > > > > > >
> > > > > > > > > Am 11.05.2017 um 09:47 schrieb Gyula Fóra <
> > > gyula.f...@gmail.com
> > > > >:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I am not sure if this belong to this thread, but while
> trying
> > > to
> > > > > run
> > > > > > a
> > > > > > > > job
> > > > > > > > > with rocks incremental backend I ran into 2 issues:
> > > > > > > > >
> > > > > > > > > One with savepoints, I can't figure out because I can't
> make
> > > > sense
> > > > > of
> > > > > > > the
> > > > > > > > > error or how it happenned:
> > > > > > > > > The error stack trace is here:
> > > > > > > > 

Re: Towards a spec for robust streaming SQL, Part 1

[Shaoxuan Wang]

Thanks to Tyler and Fabian for sharing your thoughts.

Regarding to the early/late update control of FLINK. IMO, each dynamic
table can have an EMIT config. For FLINK table-API, this can be easily
implemented in different manners, case by case. For instance, in window
aggregate, we could define "when EMIT a result" via a windowConf per each
window when we create windows. Unfortunately we do not have such
flexibility (as compared with TableAPI) in SQL query, we need find a way to
embed this EMIT config.

Regards,
Shaoxuan


On Fri, May 12, 2017 at 4:28 PM, Fabian Hueske  wrote:

> 2017-05-11 7:14 GMT+02:00 Tyler Akidau :
>
> > On Tue, May 9, 2017 at 3:06 PM Fabian Hueske  wrote:
> >
> > > Hi Tyler,
> > >
> > > thank you very much for this excellent write-up and the super nice
> > > visualizations!
> > > You are discussing a lot of the things that we have been thinking about
> > as
> > > well from a different perspective.
> > > IMO, yours and the Flink model are pretty much aligned although we use
> a
> > > different terminology (which is not yet completely established). So
> there
> > > should be room for unification ;-)
> > >
> >
> > Good to hear, thanks for giving it a look. :-)
> >
> >
> > > Allow me a few words on the current state in Flink. In the upcoming
> 1.3.0
> > > release, we will have support for group window (TUMBLE, HOP, SESSION),
> > OVER
> > > RANGE/ROW window (without FOLLOWING), and non-windowed GROUP BY
> > > aggregations. The group windows are triggered by watermark and the over
> > > window and non-windowed aggregations emit for each input record
> > > (AtCount(1)). The window aggregations do not support early or late
> firing
> > > (late records are dropped), so no updates here. However, the
> non-windowed
> > > aggregates produce updates (in acc and acc/retract mode). Based on this
> > we
> > > will work on better control for late updates and early firing as well
> as
> > > joins in the next release.
> > >
> >
> > Impressive. At this rate there's a good chance we'll just be doing
> catchup
> > and thanking you for building everything. ;-) Do you have ideas for what
> > you want your early/late updates control to look like? That's one of the
> > areas I'd like to see better defined for us. And how deep are you going
> > with joins?
> >
>
> Right now (well actually I merged the change 1h ago) we are using a
> QueryConfig object to specify state retention intervals to be able to clean
> up state for inactive keys.
> A running a query looks like this:
>
> // ---
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(env)
>
> val qConf = tEnv.queryConfig.withIdleStateRetentionTime(Time.hours(12)) //
> state of inactive keys is kept for 12 hours
>
> val t: Table = tEnv.sql("SELECT a, COUNT(*) AS cnt FROM MyTable GROUP BY
> a")
> val stream: DataStream[(Boolean, Row)] = t.toRetractStream[Row](qConf) //
> boolean flag for acc/retract
>
> env.execute()
> // ---
>
> We plan to use the QueryConfig also to specify early/late updates. Our main
> motivation is to have a uniform and standard SQL for batch and streaming.
> Hence, we have to move the configuration out of the query. But I agree,
> that it would be very nice to be able to include it in the query. I think
> it should not be difficult to optionally support an EMIT clause as well.
>
>
> >
> > > Reading the document, I did not find any major difference in our
> > concepts.
> > > In fact, we are aiming to support the cases you are describing as well.
> > > I have a question though. Would you classify an OVER aggregation as a
> > > stream -> stream or stream -> table operation? It collects records to
> > > aggregate them, but emits one record for each input row. Depending on
> the
> > > window definition (i.e., with FOLLOWING CURRENT ROW), you can compute
> and
> > > emit the result record when the input record is received.
> > >
> >
> > I would call it a composite stream → stream operation (since SQL, like
> the
> > standard Beam/Flink APIs, is a higher level set of constructs than raw
> > streams/tables operations) consisting of a stream → table windowed
> grouping
> > followed by a table → stream triggering on every element, basically as
> you
> > described in the previous paragraph.
> >
> >
> That makes sense. Thanks :-)
>
>
> > -Tyler
> >
> >
> > >
> > > I'm looking forward to the second part.
> > >
> > > Cheers, Fabian
> > >
> > >
> > >
> > > 2017-05-09 0:34 GMT+02:00 Tyler Akidau :
> > >
> > > > Any thoughts here Fabian? I'm planning to start sending out some more
> > > > emails towards the end of the week.
> > > >
> > > > -Tyler
> > > >
> > > >
> > > > On Wed, Apr 26, 2017 at 8:18 AM Tyler Akidau 
> > wrote:
> > > >
> > > > > No worries, thanks for the heads up. Good luck wrapping all that
> > stuff
> > > > up.
> > > > >
> > > > > -Tyler
> > > > >
> > > > > 

Re: Towards a spec for robust streaming SQL, Part 1

[Fabian Hueske]

2017-05-11 7:14 GMT+02:00 Tyler Akidau :

> On Tue, May 9, 2017 at 3:06 PM Fabian Hueske  wrote:
>
> > Hi Tyler,
> >
> > thank you very much for this excellent write-up and the super nice
> > visualizations!
> > You are discussing a lot of the things that we have been thinking about
> as
> > well from a different perspective.
> > IMO, yours and the Flink model are pretty much aligned although we use a
> > different terminology (which is not yet completely established). So there
> > should be room for unification ;-)
> >
>
> Good to hear, thanks for giving it a look. :-)
>
>
> > Allow me a few words on the current state in Flink. In the upcoming 1.3.0
> > release, we will have support for group window (TUMBLE, HOP, SESSION),
> OVER
> > RANGE/ROW window (without FOLLOWING), and non-windowed GROUP BY
> > aggregations. The group windows are triggered by watermark and the over
> > window and non-windowed aggregations emit for each input record
> > (AtCount(1)). The window aggregations do not support early or late firing
> > (late records are dropped), so no updates here. However, the non-windowed
> > aggregates produce updates (in acc and acc/retract mode). Based on this
> we
> > will work on better control for late updates and early firing as well as
> > joins in the next release.
> >
>
> Impressive. At this rate there's a good chance we'll just be doing catchup
> and thanking you for building everything. ;-) Do you have ideas for what
> you want your early/late updates control to look like? That's one of the
> areas I'd like to see better defined for us. And how deep are you going
> with joins?
>

Right now (well actually I merged the change 1h ago) we are using a
QueryConfig object to specify state retention intervals to be able to clean
up state for inactive keys.
A running a query looks like this:

// ---
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = TableEnvironment.getTableEnvironment(env)

val qConf = tEnv.queryConfig.withIdleStateRetentionTime(Time.hours(12)) //
state of inactive keys is kept for 12 hours

val t: Table = tEnv.sql("SELECT a, COUNT(*) AS cnt FROM MyTable GROUP BY a")
val stream: DataStream[(Boolean, Row)] = t.toRetractStream[Row](qConf) //
boolean flag for acc/retract

env.execute()
// ---

We plan to use the QueryConfig also to specify early/late updates. Our main
motivation is to have a uniform and standard SQL for batch and streaming.
Hence, we have to move the configuration out of the query. But I agree,
that it would be very nice to be able to include it in the query. I think
it should not be difficult to optionally support an EMIT clause as well.


>
> > Reading the document, I did not find any major difference in our
> concepts.
> > In fact, we are aiming to support the cases you are describing as well.
> > I have a question though. Would you classify an OVER aggregation as a
> > stream -> stream or stream -> table operation? It collects records to
> > aggregate them, but emits one record for each input row. Depending on the
> > window definition (i.e., with FOLLOWING CURRENT ROW), you can compute and
> > emit the result record when the input record is received.
> >
>
> I would call it a composite stream → stream operation (since SQL, like the
> standard Beam/Flink APIs, is a higher level set of constructs than raw
> streams/tables operations) consisting of a stream → table windowed grouping
> followed by a table → stream triggering on every element, basically as you
> described in the previous paragraph.
>
>
That makes sense. Thanks :-)


> -Tyler
>
>
> >
> > I'm looking forward to the second part.
> >
> > Cheers, Fabian
> >
> >
> >
> > 2017-05-09 0:34 GMT+02:00 Tyler Akidau :
> >
> > > Any thoughts here Fabian? I'm planning to start sending out some more
> > > emails towards the end of the week.
> > >
> > > -Tyler
> > >
> > >
> > > On Wed, Apr 26, 2017 at 8:18 AM Tyler Akidau 
> wrote:
> > >
> > > > No worries, thanks for the heads up. Good luck wrapping all that
> stuff
> > > up.
> > > >
> > > > -Tyler
> > > >
> > > > On Tue, Apr 25, 2017 at 12:07 AM Fabian Hueske 
> > > wrote:
> > > >
> > > >> Hi Tyler,
> > > >>
> > > >> thanks for pushing this effort and including the Flink list.
> > > >> I haven't managed to read the doc yet, but just wanted to thank you
> > for
> > > >> the
> > > >> write-up and let you know that I'm very interested in this
> discussion.
> > > >>
> > > >> We are very close to the feature freeze of Flink 1.3 and I'm quite
> > busy
> > > >> getting as many contributions merged before the release is forked
> off.
> > > >> When that happened, I'll have more time to read and comment.
> > > >>
> > > >> Thanks,
> > > >> Fabian
> > > >>
> > > >>
> > > >> 2017-04-22 0:16 GMT+02:00 Tyler Akidau  >:
> > > >>
> > > >> > Good point, when you start talking about anything less than a full
> > > 

[jira] [Created] (FLINK-6566) Narrow down interface for compatibility hook method in VersionedIOReadableWritable

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-6566:
--

 Summary: Narrow down interface for compatibility hook method in 
VersionedIOReadableWritable
 Key: FLINK-6566
 URL: https://issues.apache.org/jira/browse/FLINK-6566
 Project: Flink
  Issue Type: Improvement
  Components: Core, Type Serialization System
Reporter: Tzu-Li (Gordon) Tai


The {{VersionedIOReadableWritable}} provides a {{isCompatibleVersion}} method 
that allows users to override to resolve older versions.

This method allows too much space for the user to mess up the implementation, 
and is much more relevant now because we have an user-facing class 
{{TypeSerializerConfigSnapshot}} which extends {{VersionedIOReadableWritable}}.

Instead of a {{isCompatibleVersion}} method, it should only expose a narrower 
{{int[] getCompatibleVersions}} that the internal version check uses.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6565) Improve error messages for state restore failures

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-6565:
--

 Summary: Improve error messages for state restore failures
 Key: FLINK-6565
 URL: https://issues.apache.org/jira/browse/FLINK-6565
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Affects Versions: 1.3.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Priority: Critical


The error messages thrown when state restore fails needs to be more explicit 
and clear of the actual reason.

At least 2 cases we've seen so far:

1.
For example, currently, when restoring an operator state or memory-backed keyed 
state, the previous serializer must exist. If it doesn't exist, currently only 
a vague NPE is thrown, without a clear message of the actual reason.

2.
If the restore failure was due to an incompatible version of a serializer's 
config snapshot, then it should report something more informative then: 
"Incompatible version: found 1, required 1."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

[Fabian Hueske]

Hi Haohui,

open issue for 1.3 are tracked in JIRA.
You can specify "Fix version: 1.3.0" and an appropriate priority.

I updated a couple of issues to have the fix version tag.
Any bug that is found while release testing should be added to this list by
setting fix version, priority, and component.

Thanks,
Fabian

PS.: This query will list the open Table API issues for 1.3:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.3.0%20AND%20component%20%3D%20%22Table%20API%20%26%20SQL%22%20ORDER%20BY%20priority%20DESC

2017-05-12 0:50 GMT+02:00 Haohui Mai :

> Hi,
>
> Just curious -- do we have anything to track the blockers on the table /
> SQL side? For example, it seems to me that FLINK-6563 is a blocker because
> all group windows in SQL will not work without it. I think that Fabian /
> Timo might want to get several jiras into 1.3.
>
> Regards,
> Haohui
>
> On Thu, May 11, 2017 at 12:33 PM Stephan Ewen  wrote:
>
> > From the above list, I addressed a bunch of issues.
> >
> > - FLINK-6514 Cannot start Flink Cluster in standalone mode has an open
> Pull
> > Request:
> >- master: https://github.com/apache/flink/pull/3876
> >- 1.3: https://github.com/apache/flink/pull/3877
> >
> > Left are:
> >
> > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan
> has
> > PR
> > open to fix the critical ones)
> >
> > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > ZooKeeperCompletedCheckpointStore
> >
> >
> > *I think that FLINK-6284 is totally a blocker for the release!*
> >
> >
> > On Thu, May 11, 2017 at 4:19 PM, Robert Metzger 
> > wrote:
> >
> > > It seems that we found quite a large number of critical issues in the
> > first
> > > RC.
> > >
> > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan
> > has
> > > a PR open to fix the critical ones)
> > > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has a
> > > pending PR)
> > > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> > > issues (status unknown)
> > > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan is
> on
> > > it)
> > > - FLINK-6508 Include license files of packaged dependencies (Stephan is
> > on
> > > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR
> files
> > > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > > ZooKeeperCompletedCheckpointStore (unknown)
> > >
> > > I would like to get these issues fixed by end of this week (Sunday), so
> > > that I can create the first voting RC on Monday morning.
> > > Please reject if you think we will not manage to get the stuff fixed
> > until
> > > then.
> > >
> > >
> > >
> > > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann 
> > > wrote:
> > >
> > > > Unfortunately, it won't be fully functional in 1.3.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, May 11, 2017 at 10:45 AM, Renjie Liu <
> liurenjie2...@gmail.com>
> > > > wrote:
> > > >
> > > > > @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
> > > > >
> > > > > On Thu, May 11, 2017 at 4:12 PM Gyula Fóra 
> > > wrote:
> > > > >
> > > > > > Thanks Stefan!
> > > > > > Gyula
> > > > > >
> > > > > > Stefan Richter  ezt írta (időpont:
> > > 2017.
> > > > > máj.
> > > > > > 11., Cs, 10:04):
> > > > > >
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Thanks for reporting this. I found a couple of issues yesterday
> > > and I
> > > > > am
> > > > > > > currently working on a bundle of fixes. I will take a look at
> > this
> > > > > > problem,
> > > > > > > and if it is already covered.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stefan
> > > > > > >
> > > > > > > > Am 11.05.2017 um 09:47 schrieb Gyula Fóra <
> > gyula.f...@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > > I am not sure if this belong to this thread, but while trying
> > to
> > > > run
> > > > > a
> > > > > > > job
> > > > > > > > with rocks incremental backend I ran into 2 issues:
> > > > > > > >
> > > > > > > > One with savepoints, I can't figure out because I can't make
> > > sense
> > > > of
> > > > > > the
> > > > > > > > error or how it happenned:
> > > > > > > > The error stack trace is here:
> > > > > > > >
> > https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239
> > > > > > > > This happens on every savepoint attempt and seems to be
> related
> > > to
> > > > > the
> > > > > > > > kafka source. Interestingly other tasks succeed in writing
> data
> > > to
> > > > > > hdfs.
> > > > > > > >
> > > > > > > > The other one is covered by
> > > > > > > https://issues.apache.org/jira/browse/FLINK-6531 I
> > > > > > > > guess. I am not sure if the first one is related though.
> > > > > > > >
> > > > > > > > Thank you!
> > > > > > > > Gyula
> > > > > > > >
> > > > > > > >