[jira] [Created] (FLINK-12269) Support Temporal Table Join in blink planner

[Jark Wu (JIRA)]

Jark Wu created FLINK-12269:
---

 Summary: Support Temporal Table Join in blink planner
 Key: FLINK-12269
 URL: https://issues.apache.org/jira/browse/FLINK-12269
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Jark Wu


Support translate following "FOR SYSTEM_TIME AS OF" query into 
{{StreamExecTemporalTableJoin}}.

{code:sql}
SELECT
  o.amout, o.currency, r.rate, o.amount * r.rate
FROM
  Orders AS o
  JOIN LatestRates FOR SYSTEM_TIME AS OF o.proctime AS r
  ON r.currency = o.currency
{code}

This is an extension to current temporal join (FLINK-9738) using a standard 
syntax introduced in Calcite 1.19.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re:Hello, I want to subscribe Flink dev mail group

[shengjk1]

Welcome.  please send email to dev-subscr...@flink.apache.org


More details about flink's email  : 
https://flink.apache.org/community.html#mailing-lists


Best,
Shengjk1




On 04/20/2019 10:32，Armstrong wrote：
Thanks for your time.

hello

[Armstrong]

hello

Hello, I want to subscribe Flink dev mail group

[Armstrong]

Thanks for your time.

[jira] [Created] (FLINK-12268) Port SharedSlotsTest to new code base

[leesf (JIRA)]

leesf created FLINK-12268:
-

 Summary: Port SharedSlotsTest to new code base
 Key: FLINK-12268
 URL: https://issues.apache.org/jira/browse/FLINK-12268
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: leesf
Assignee: leesf
 Fix For: 1.9.0


Get rid of Instance.

Port SchedulerTestUtils#getRandomInstance to new code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12267) Port SimpleSlotTest to new code base

[leesf (JIRA)]

leesf created FLINK-12267:
-

 Summary: Port SimpleSlotTest to new code base
 Key: FLINK-12267
 URL: https://issues.apache.org/jira/browse/FLINK-12267
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination, Tests
Reporter: leesf
Assignee: leesf
 Fix For: 1.9.0


Mainly get rid of `Instance`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12266) add NOTICE file for dependencies that are newly introduced in flink-connector-hive

[Bowen Li (JIRA)]

Bowen Li created FLINK-12266:


 Summary: add NOTICE file for dependencies that are newly 
introduced in flink-connector-hive
 Key: FLINK-12266
 URL: https://issues.apache.org/jira/browse/FLINK-12266
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12265) link-table-planner-blink UT Failure: SortDistinctAggregateITCase.testSomeColumnsBothInDistinctAggAndGroupBy

[Bowen Li (JIRA)]

Bowen Li created FLINK-12265:


 Summary: link-table-planner-blink UT Failure: 
SortDistinctAggregateITCase.testSomeColumnsBothInDistinctAggAndGroupBy
 Key: FLINK-12265
 URL: https://issues.apache.org/jira/browse/FLINK-12265
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Bowen Li
Assignee: Jingsong Lee
 Fix For: 1.9.0


Seems to be a memory allocation issue but would be worth checking out if that's 
caused by the planner itself taking too much memory

https://travis-ci.org/apache/flink/jobs/522051049

11:41:37.461 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 2, Time 
elapsed: 73.859 s <<< FAILURE! - in 
org.apache.flink.table.runtime.batch.sql.agg.SortDistinctAggregateITCase
11:41:37.461 [ERROR] 
testSomeColumnsBothInDistinctAggAndGroupBy(org.apache.flink.table.runtime.batch.sql.agg.SortDistinctAggregateITCase)
  Time elapsed: 6.274 s  <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
Caused by: java.lang.RuntimeException: 
org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate 
64 pages. Only 0 pages are remaining.
Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could not 
allocate 64 pages. Only 0 pages are remaining.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12264) Port ExecutionGraphTestUtils to new code base

[leesf (JIRA)]

leesf created FLINK-12264:
-

 Summary: Port ExecutionGraphTestUtils to new code base
 Key: FLINK-12264
 URL: https://issues.apache.org/jira/browse/FLINK-12264
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: leesf
Assignee: leesf
 Fix For: 1.9.0


Mainly get rid of Instance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Apache Flink at Season of Docs

[Konstantin Knauf]

Hi everyone,

I have just submitted our application :)

@Stephan Ewen , @David Anderson :
Please remember to register as mentor by Tuesday:
https://docs.google.com/forms/d/e/1FAIpQLSe-JjGvaKKGWZOXxrorONhB8qN3mjPrB9ZVkcsntR73Cv_K7g/viewform

Cheers,

Konstantin

On Thu, Apr 18, 2019 at 1:04 PM jincheng sun 
wrote:

> Hi Konstantin,
>
> The  blog post is released, please see
> https://flink.apache.org/news/2019/04/17/sod.html
>
> And I think is better to spread the word by
> https://twitter.com/ApacheFlink,
> but I found that only PMC can manage and publish messages.
>
> Help spread the word here:
> https://twitter.com/sunjincheng121/status/1118831783762481152
>
> Best,
> Jincheng
>
> jincheng sun  于2019年4月18日周四 上午11:01写道：
>
> > Thanks Konstantin!
> >
> > I registered as an alternative administrator,  and have had left few
> > comments at the blog post.
> >
> > Best,
> > Jincheng
> >
> > Fabian Hueske  于2019年4月18日周四 上午12:36写道：
> >
> >> Thanks Konstantin!
> >>
> >> I registered as a mentor and will have a look at the blog post.
> >>
> >> Best, Fabian
> >>
> >> Am Mi., 17. Apr. 2019 um 18:14 Uhr schrieb Konstantin Knauf <
> >> konstan...@ververica.com>:
> >>
> >> > Hi everyone,
> >> >
> >> > a few updates on our application:
> >> >
> >> > 1. As Aizhamal (Thanks!) has suggested I also added the information to
> >> > submit during our application to the Google doc [1] and Fabian added a
> >> > description for the SQL project idea (Thanks!).
> >> >
> >> > 2. I had a quick chat with Fabian offline and we concluded, that the
> >> "Flink
> >> > Internals" project might not be good fit for Season of Docs after all,
> >> > because, we think, the amount of mentoring by core developers that
> >> would be
> >> > necessary to produce such a documentation could not be guaranteed. Any
> >> > opinions?
> >> >
> >> > 3. To submit our application, we need to publish our project ideas
> list.
> >> > For this I have just opened a PR to add a small blog post about Season
> >> of
> >> > Docs [2]. Please have a look and provide feedback.
> >> >
> >> > 4. For mentors (Stephan, Fabian, Jark, David) please complete the
> mentor
> >> > registration [3] by next Tuesday (*Application Deadline*)
> >> >
> >> > Cheers,
> >> >
> >> > Konstantin
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://docs.google.com/document/d/1Up53jNsLztApn-mP76AB6xWUVGt3nwS9p6xQTiceKXo/edit?usp=sharing
> >> > [2] https://github.com/apache/flink-web/pull/202
> >> > [3]
> >> >
> >> >
> >>
> https://docs.google.com/forms/d/e/1FAIpQLSe-JjGvaKKGWZOXxrorONhB8qN3mjPrB9ZVkcsntR73Cv_K7g/viewform
> >> >
> >> > On Mon, Apr 15, 2019 at 9:12 PM Aizhamal Nurmamat kyzy <
> >> > aizha...@google.com>
> >> > wrote:
> >> >
> >> > > +Konstantin Knauf  this is looking good,
> >> > thanks
> >> > > for sharing!
> >> > >
> >> > > I also created a similar doc for Apache Airflow [1]. It is a bit
> >> messy,
> >> > > but it has questions from the application form that you can work
> with.
> >> > >
> >> > > Cheers,
> >> > > Aizhamal
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1HoL_yjNYiTAP9IxSlhx3EUnPFU4l9WOT9EnwBZjCZo0/edit#
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2019 at 2:24 AM Robert Metzger  >
> >> > > wrote:
> >> > >
> >> > >> Hi all,
> >> > >> I'm very happy to see this project happening!
> >> > >>
> >> > >> Thank you for the proposal Konstantin! One idea for the "related
> >> > >> material": we could also link to talks or blog posts about
> concepts /
> >> > >> monitoring / operations. Potential writers could feel overwhelmed
> by
> >> our
> >> > >> demand for improvements, without any additional material.
> >> > >>
> >> > >>
> >> > >> On Mon, Apr 15, 2019 at 10:16 AM Konstantin Knauf <
> >> > >> konstan...@ververica.com> wrote:
> >> > >>
> >> > >>> Hi everyone,
> >> > >>>
> >> > >>> thanks @Aizhamal Nurmamat kyzy . As we only
> >> have
> >> > >>> one
> >> > >>> week left until the application deadline, I went ahead and
> created a
> >> > >>> document for the project ideas [1]. I have added the description
> for
> >> > the
> >> > >>> "stream processing concepts" as well as the "deployment &
> operations
> >> > >>> documentation" project idea. Please let me know what you think,
> >> edit &
> >> > >>> comment. We also need descriptions for the other two projects
> (Table
> >> > >>> API/SQL & Flink Internals). @Fabian/@Jark/@Stephan can you chime
> in?
> >> > >>>
> >> > >>> Any more project ideas?
> >> > >>>
> >> > >>> Best,
> >> > >>>
> >> > >>> Konstantin
> >> > >>>
> >> > >>>
> >> > >>> [1]
> >> > >>>
> >> > >>>
> >> >
> >>
> https://docs.google.com/document/d/1Up53jNsLztApn-mP76AB6xWUVGt3nwS9p6xQTiceKXo/edit?usp=sharing
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Fri, Apr 12, 2019 at 6:50 PM Aizhamal Nurmamat kyzy <
> >> > >>> aizha...@google.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Hello everyone,
> >> > >>> >
> >> > >>> > @Konstantin Knauf  - yes, you are
> >> correct.
> >> > >>> > Between steps 1 and 2 though, the open source 

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

[Jeff Zhang]

>>>  The ExecutionEnvironment is usually used by the user who writes the
code and this person (I assume) would not be really interested in these
callbacks.

Usually ExecutionEnvironment is used by the user who write the code, but it
doesn't needs to be created and configured by this person. e.g. in Zeppelin
notebook, ExecutionEnvironment is created by Zeppelin, user just use
ExecutionEnvironment to write flink program.  You are right that the end
user would not be interested in these callback, but the third party library
that integrate with zeppelin would be interested in these callbacks.

>>> In your case, it could be sufficient to offer some hooks for the
ClusterClient or being able to provide a custom ClusterClient.

Actually in my initial PR (https://github.com/apache/flink/pull/8190), I do
pass JobListener to ClusterClient and invoke it there.
But IMHO, ClusterClient is not supposed be a public api for users. Instead
JobClient is the public api that user should use to control job. So adding
hooks to ClusterClient directly and provide a custom ClusterClient doesn't
make sense to me. IIUC, you are suggesting the following approach
 env.getClusterClient().addJobListener(jobListener)
but I don't see its benefit compared to this.
 env.addJobListener(jobListener)

Overall, I think adding hooks is orthogonal with fine grained job control.
And I agree that we should refactor the flink client component, but I don't
think it would affect the JobListener interface. What do you think ?




Till Rohrmann  于2019年4月18日周四 下午8:57写道：

> Thanks for starting this discussion Jeff. I can see the need for
> additional hooks for third party integrations.
>
> The thing I'm wondering is whether we really need/want to expose a
> JobListener via the ExecutionEnvironment. The ExecutionEnvironment is
> usually used by the user who writes the code and this person (I assume)
> would not be really interested in these callbacks. If he would, then one
> should rather think about a better programmatic job control where the
> `ExecutionEnvironment#execute` call returns a `JobClient` instance.
> Moreover, we would effectively make this part of the public API and every
> implementation would need to offer it.
>
> In your case, it could be sufficient to offer some hooks for the
> ClusterClient or being able to provide a custom ClusterClient. The
> ClusterClient is the component responsible for the job submission and
> retrieval of the job result and, hence, would be able to signal when a job
> has been submitted or completed.
>
> Cheers,
> Till
>
> On Thu, Apr 18, 2019 at 8:57 AM vino yang  wrote:
>
>> Hi Jeff,
>>
>> I personally like this proposal. From the perspective of programmability,
>> the JobListener can make the third program more appreciable.
>>
>> The scene where I need the listener is the Flink cube engine for Apache
>> Kylin. In the case, the Flink job program is embedded into the Kylin's
>> executable context.
>>
>> If we could have this listener, it would be easier to integrate with
>> Kylin.
>>
>> Best,
>> Vino
>>
>> Jeff Zhang  于2019年4月18日周四 下午1:30写道：
>>
>>>
>>> Hi All,
>>>
>>> I created FLINK-12214
>>>  for adding
>>> JobListener (hook) in flink job lifecycle. Since this is a new public api
>>> for flink, so I'd like to discuss it more widely in community to get more
>>> feedback.
>>>
>>> The background and motivation is that I am integrating flink into apache
>>> zeppelin (which is a notebook in case you
>>> don't know). And I'd like to capture some job context (like jobId) in the
>>> lifecycle of flink job (submission, executed, cancelled) so that I can
>>> manipulate job in more fined grained control (e.g. I can capture the jobId
>>> when job is submitted, and then associate it with one paragraph, and when
>>> user click the cancel button, I can call the flink cancel api to cancel
>>> this job)
>>>
>>> I believe other projects which integrate flink would need similar
>>> mechanism. I plan to add api addJobListener in
>>> ExecutionEnvironment/StreamExecutionEnvironment so that user can add
>>> customized hook in flink job lifecycle.
>>>
>>> Here's draft interface JobListener.
>>>
>>> public interface JobListener {
>>>
>>> void onJobSubmitted(JobID jobId);
>>>
>>> void onJobExecuted(JobExecutionResult jobResult);
>>>
>>> void onJobCanceled(JobID jobId, String savepointPath);
>>> }
>>>
>>> Let me know your comment and concern, thanks.
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

-- 
Best Regards

Jeff Zhang

[jira] [Created] (FLINK-12263) Remove SINGLE_VALUE aggregate function from physical plan

[Jark Wu (JIRA)]

Jark Wu created FLINK-12263:
---

 Summary: Remove SINGLE_VALUE aggregate function from physical plan
 Key: FLINK-12263
 URL: https://issues.apache.org/jira/browse/FLINK-12263
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Jark Wu


 SINGLE_VALUE is an aggregate function which only accepts one row, and throws 
exception when received more than one row.

 

For example: 
{code:sql}
SELECT a2, SUM(a1) FROM A GROUP BY a2 HAVING SUM(a1) > (SELECT SUM(a1) * 0.1 
FROM A)
{code}
will get a physical plan contains SINGLE_VALUE:
{code:sql}
+- NestedLoopJoin(joinType=[InnerJoin], where=[>(EXPR$1, $f0)], select=[a2, 
EXPR$1, $f0], build=[right], singleRowJoin=[true])
   :- HashAggregate(isMerge=[true], groupBy=[a2], select=[a2, Final_SUM(sum$0) 
AS EXPR$1])
   :  +- Exchange(distribution=[hash[a2]])
   : +- LocalHashAggregate(groupBy=[a2], select=[a2, Partial_SUM(a1) AS 
sum$0])
   :+- TableSourceScan(table=[[A, source: [TestTableSource(a1, a2)]]], 
fields=[a1, a2])
   +- Exchange(distribution=[broadcast])
  +- HashAggregate(isMerge=[true], select=[Final_SINGLE_VALUE(value$0, 
count$1) AS $f0])
 +- Exchange(distribution=[single])
+- LocalHashAggregate(select=[Partial_SINGLE_VALUE(EXPR$0) AS 
(value$0, count$1)])
   +- Calc(select=[*($f0, 0.1) AS EXPR$0])
  +- HashAggregate(isMerge=[true], select=[Final_SUM(sum$0) AS 
$f0])
 +- Exchange(distribution=[single])
+- LocalHashAggregate(select=[Partial_SUM(a1) AS sum$0])
   +- Calc(select=[a1])
  +- TableSourceScan(table=[[A, source: 
[TestTableSource(a1, a2)]]], fields=[a1, a2])
{code}
But SINGLE_VALUE is a bit wired in physical plan because the logical plan can 
make sure there is only one input row. Moreover it it also introduces 
additional overhead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)