[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760474#comment-16760474 ] Dian Fu commented on FLINK-11447: - [~twalthr] Thanks a lot for offering the help. I have uploaded the PR and also updated the documentation and the java docs according to your suggestions. Looking forward to your feedback. I'll keep updating the PR as soon as possible to make sure it can be committed in this week. :) > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Blocker > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759990#comment-16759990 ] Timo Walther commented on FLINK-11447: -- Happy Chinese New Year everyone :) [~dian.fu] thanks for working on this issue. It would be great to get this issue done as quickly as possible. I will be out of office next week, so it would be great to merge it even this week. Let me know if I can help. I can also implement it myself if you don't have capacities. I will make this issue a blocker for the next Flink release. > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759943#comment-16759943 ] sunjincheng commented on FLINK-11447: - I am very happy that everyone has reached an agreement. Looking forward the PR! And Happy Chinese New Year ! > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759831#comment-16759831 ] Dian Fu commented on FLINK-11447: - Thanks [~twalthr] [~dawidwys] [~hequn8128] [~sunjincheng121] [~jark] for your inputs. Very appreciated! I'm fine with joinLateral too as we have also saw some issues for the Table.create these two days. Besides the issues [~twalthr] mentioned, there are two places to create a table for UDTF: Table.create and TableEnvironment.create(). If we want to take this approach, may be we should use TableEnvironment.create instead of Table.create(). Anyway, I will take the joinLateral approach and provide a PR ASAP. Thanks again. > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759825#comment-16759825 ] Hequn Cheng commented on FLINK-11447: - Hi all, thank you all for your valuable suggestions. [~twalthr] [~dawidwys] OK, I think your concerns also make sense. Although the `create` approach hasn't introduced a new operator it may also bring some other problems. It may not very mature now. Considering this, I'm fine that we can start with the `joinLateral` method. What do you guys think? Best, Hequn > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759742#comment-16759742
]
Timo Walther commented on FLINK-11447:
--
A {{Table.create()}} would still not solve the issues that I mentioned before:
- Users need to pass a table environment.
- A lateral table should not exist on its own and this is also expressed in the
SQL standard.
User should not be able to write a program like:
{code}
val table = Table.create("udtf")
table.filter(...).print();
{code}
What are the semantics for this example?
Furthermore, for table environments we will perform a discovery but how do we
instantiate the table?
Another point is usability, especially when you start with the API, a user will
use {{TableEnvironment.create}} and will look for the same mechanism for table.
If we offer a {{Table.create()}}, I'm sure users will ask how to create a table
with this method instead of using the methods provided in table environment.
We could also rename {{joinLateral}} to something completely different like
{{joinFunction(udtf)}}. I think would make it very clear.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758846#comment-16758846
]
Jark Wu commented on FLINK-11447:
-
Jump in the discussion ;-)
That's right {{Table.create}} will not work if Table is only an interface in
flink-table-api.
But from a user perspective, {{.join(Table.create("udtf"))}} is more clear than
{{.joinLateral(udtf)}}. Btw, just as a reference, we renamed {{crossApply}} to
{{join}} for udtf years ago, see FLINK-5304.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758530#comment-16758530
]
Dawid Wysakowicz commented on FLINK-11447:
--
I'm still +1 for the {{joinLateral}}
No static method will work ATM, as there is no Table implementation in API to
be instantiated. Implementation of Table comes only from Planner.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758487#comment-16758487 ] Dian Fu commented on FLINK-11447: - Personally I like the approach suggested by [~hequn8128] to add a static method Table.create to create a table to represent UDTF and still use the join interface for joining UDTF. As it only adds one method Table.create to replace the constructor to create a Table for UDTF. All other things remains the same as before. Regarding to adding an interface such as joinLateral, may be we should start a discussion in the dev mail list and hear the opinions from more people and only adds it if the majority think it OK. > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758450#comment-16758450
]
sunjincheng commented on FLINK-11447:
-
Thanks for the [~twalthr] [~dawidwys] [~hequn8128] lively discussion！
Hi [~hequn8128] Thank you very much for reviewing the information of many
lateralTables!
That's true, there are clear definitions of lateral in several commonly used
databases, e.g.:
* Postresql Lateral will join a subquery, e.g.:
{code:java}
SELECT * FROM tab, lateral (select ...) alias ...`{code}
* Hive Lateral will join the VIEW keyword, then UDTF, e.g:
{code:java}
SELECT * FROM tab LATERAL VIEW udtf(..) ...{code}
* SQL Server uses Cross Apply, but essentially adds a subquery, e.g.:
{code:java}
SELECT * FROM tableA a CROSS APPLY (select ...) alias{code}
Although there are certain differences in grammar, here is essentially the
right side parameter of join is Table.
So，I think the method `join(String)` should not be defined in TableAPI which we
have reached a consensus
Then we have two better approach:
Approach 1: Add the new operator `JoinLateral` and `leftOuterJoinLateral`
Approach 2: Add the static method `create(..)` (maybe `createLateralTable(..)`
is better) in table interface;
>From the points of my view, both solutions can solve the UDTF problem of java
>users. About #1 we introduced joinLateral, which shows that the user's
>argument is a udtf (String or Expression), but the semantics are still JOIN.
>There is a little doubt which [~hequn8128] mentioned above.
About #2: we do not need introduce the new operator, but there is also a
problem here that the fieldRefrence of table.create(tenv,
"udtf(fieldRefrence")), i.e: `table.create()` cannot be used independently
(currently new Table("udtf(fieldRefrence") also exists same problem).
I think both of the approach may be the temporary solution to complete the
`flip32`, we may further discuss in the future to find a better solution, such
as we may discuss Table classification, Table, TemporalTable, LateralTable,
etc. So I suggest that we can choose one of them now As soon as possible to
advance the progress of FLIP-32, maybe we can discuss the long-term planning in
the new thread.
What do you think? [~twalthr] [~hequn8128] [~dawidwys] [~dian.fu]
Best, Jincheng
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758307#comment-16758307
]
Hequn Cheng commented on FLINK-11447:
-
Hi, [~twalthr]. I agree that we don't need to complete in sync with SQL in
TableAPI. And yes, it is very unlikely that we will ever support correlated
subqueries in Table API.
Hi, [~dawidwys]. I agree with you that we would better not to introduce the
Lateral.subquery concept for now. Such a feature would definitely need another
big discussion.
Thanks again for your nice comments. I joined the discussion as I saw some good
points raised by the second option listed by [~sunjincheng121]:
- Don't need to add another kind of join.
- The right side of join should be a Table.
I would like to share more thoughts for these two points.
1. We don't need to add another kind of join.
The semantic of `joinLateral` is exactly same to `inner join`, i.e., it is not
a left join, nor a right join. It is more clear to use `join`. We only need
another kind of join if it acts differently. For example, we need to add
antiJoin/semiJoin as they join two tables differently.
2. The right side of the join should be a Table.
We can only join two tables. I think this is the problem we need to solve when
we make Table interface. And I'd like to propose a solution as follows.
{code:java}
object Table {
def create(tEnv: TableEnvironment, expr: String): Table = {
tEnv.createTable(expr)
}
}
{code}
Similar to {{TableEnvironment.create()}}, we can also add a create method for
Table. The query written by user would be {{table.join(Table.create(tEnv,
"xxx"))}}, similar to the query before, {{table.join(new Table(tEnv, "xxx"))}}
What do you guys think?
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757126#comment-16757126
]
Timo Walther commented on FLINK-11447:
--
Thanks for joining the discussion [~hequn8128].
First of all, in my opinion, we don't need to be completely in sync with SQL in
the Table API. The goal should be to have a clear and fluent API. The more
classes we introduce the more import issues occur and the more people miss
interesting features. With a {{joinLateral}} method, users get this feature
suggested in their IDE.
The {{Lateral}} keyword "allows the sub-SELECT to refer to columns of FROM
items that appear before it in the FROM list", however, it is very unlikely
that we will ever support correlated subqueries in Table API. We will only use
it for table functions.
If we were aiming for being in sync with SQL, we would need something like:
{code}
myTable.join(Lateral(Table("split(a)").as("word, length")))
myTable.join(Lateral(tableEnv.scan("OtherTable"
myTable.join(Lateral(tableEnv.sql("SELECT correlatedSubQuery? FROM..."
{code}
Because lateral itself can be used for both SQL subqueries or expressions with
a {{TABLE}} keyword.
Why should users be able to filter, groupBy, or window on a lateral table? A
lateral table should not exist on its own.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757113#comment-16757113
]
Dawid Wysakowicz commented on FLINK-11447:
--
Hi [~hequn8128],
We thought about the approach you mention. The reason why I prefer the
{{lateralJoin}} approach is that it simplifies things. Lateral makes sense only
in case of joins (if it is not in join, it must be uncorrelated query, which
is equal to select from a regular table). It is highly improbable that we will
support join with correlated queries in the near future, so it does not make
sense to introduce such a complex concept for feature that we don't even know
if it will make it into the codebase.
The solution with {{joinLateral}} does not block us from switching to something
similar to your solution in the future, if we introduce support for lateral
subqueries/correlated tables. We could deprecate those methods then.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757095#comment-16757095
]
Hequn Cheng commented on FLINK-11447:
-
Hi all, thanks for the nice discussion! I would like to share some thoughts of
mine. I'm not sure if we have to add another kind of join. From my point of
view, the difference between table-join-table and table-join-lateralTable is
the type of right Table instead of the join operator. This would be much clear
if we take a look at the SQL:
{code:java}
"SELECT c, s FROM MyTable LEFT JOIN LATERAL TABLE(func1(c)) AS T(s) ON TRUE"
{code}
We may need another way to define a (Lateral) Table rather than define a new
Join. For example,
{code:java}
myTable.join(Lateral.subquery("split(a)").as("word, length"))
{code}
And we can have the following advantages:
- We haven't introduced a new kind of join operator. The join behavior would
be very clear. Everyone knows how inner or outer join works. However, he(she)
would be confused about the joinLateral.
- It is consistent to join, i.e, table join table.
- As Lateral.subquery("xxx") returns a Table, we can also perform select or
filter on it.
- In the long term, we would support select from a Lateral Table, so this
would be "select * from Lateral.subquery("xxx")", i.e., Lateral Table should be
orthogonal to join.
What do you guys think?
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756311#comment-16756311 ] sunjincheng commented on FLINK-11447: - Hi guys, adding a `JoinLateral` operator is indeed a good way to find a way out. In this way we don't break the semantics of traditional JOIN. And introduce the a new operator to handle the TableFunction problem, so we have more self-definition space. currently I like the solution which [~twalthr] and [~pnowojski] mentioned above. Please give me some time to check out more information， so that to make our decision more confident. will feedback later(Before Friday night). Thanks, Jincheng > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756227#comment-16756227 ] Dawid Wysakowicz commented on FLINK-11447: -- What I understand is: * yes lateral join will not use join/leftOuterJoin anymore. * we don't need implict that converts from TableFunction to Table, but we need an implicit that converts TableFunction => `TableFunctionCall` expression > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756199#comment-16756199
]
Dawid Wysakowicz commented on FLINK-11447:
--
I guess you meant those signatures:
{code}
joinLateral(Expression)
joinLateral(String)
joinLateral(Expression, Expression)
joinLateral(String, String)
leftOuterJoinLateral(Expression)
leftOuterJoinLateral(String)
leftOuterJoinLateral(Expression, Expression)
leftOuterJoinLateral(String, String)
{code}
I also still prefer this option it clearly states what is supported, and that
it cannot exist on its own, but is only applicable in case of those kind of
joins.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756205#comment-16756205 ] Dian Fu commented on FLINK-11447: - If so, we need to remove the implicit which converts TableFunction to Table and all literal join will not use the join/leftOuterJoin interface any more, right? > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756079#comment-16756079
]
Timo Walther commented on FLINK-11447:
--
[~sunjincheng121] I understand your concerns regarding {{join(String)}}. People
could misunderstand it as a table name. However, my concern is that table
environment must not be part of this operation. This object is actually not
necessary and a {{tableEnv.lateralTable("split(a)”).as (“word, length")}} has
no meaning on its own. If you check the explanation of the lateral key word, it
means that "This allows the sub-SELECT to refer to columns of FROM items that
appear before it in the FROM list". So it has nothing to do with a table
environment.
[~pnowojski] suggested another solution:
{code}
Table result = orders.lateralJoin("rates(o_proctime)", "o_currency =
r_currency")
{code}
What do you think?
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Dian Fu
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755930#comment-16755930 ] sunjincheng commented on FLINK-11447: - Yes, when we discussed the UDTF, we discussed this problem. From the semantics of JOIN, the right side of the join should be a Table, that is, the method of `join(String)` should not be defined. What do you think? [~twalthr] Hi, [~dian.fu] glad to hear that you can take this JIRA. and assigned it to you :-) > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755785#comment-16755785
]
Dian Fu commented on FLINK-11447:
-
Thanks [~dawidwys] for the sharing your thoughts. If we choose option 3, we
need to add the following new interfaces to Table (maybe more in the future if
more join type is supported):
{code:java}
join(udtf: String)
join(udtf: String, joinPredicate: String)
join(udtf: String, joinPredicate: Expression)
leftOuterJoin(udtf: String)
leftOuterJoin(udtf: String, joinPredicate: String)
leftOuterJoin(udtf: String, joinPredicate: Expression){code}
Actually there are such kinds of interfaces in Table previously and these
interfaces are removed in FLINK-6334 as to left UDTF use table.join(table)
interface. Personally I prefer the table.join(table) as it's more straight
forward in the semantics. Thoughts?
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755068#comment-16755068
]
Dawid Wysakowicz commented on FLINK-11447:
--
I am very much in favor of option 3. AFAIK not every type of join is supported
for Temporal Tables (not sure if it will change). Also the 3rd option unifies
the semantics between scala and java.
In scala we use: `sTab.join(func1('c, "$") as 's)` already via implicit
conversions.
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[ https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754956#comment-16754956 ] Dian Fu commented on FLINK-11447: - Thanks [~twalthr] for creating the ticket and [~sunjincheng121] for the detailed solution. Personally I like the approach 2 for the following reasons: 1) Regarding to approach 1, personally I think *tableEnv.scan* is more suitable for a source table from the literal point of view. It's not straight forward for *TableFunction.* 2) Regarding to approach 3, table join table may be more straight forward compared to approach 2. > Deprecate "new Table(TableEnvironment, String)" > --- > > Key: FLINK-11447 > URL: https://issues.apache.org/jira/browse/FLINK-11447 > Project: Flink > Issue Type: New Feature > Components: Table API SQL >Reporter: Timo Walther >Assignee: sunjincheng >Priority: Major > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Once table is an interface we can easily replace the underlying > implementation at any time. The constructor call prevents us from converting > it into an interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"
[
https://issues.apache.org/jira/browse/FLINK-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754893#comment-16754893
]
sunjincheng commented on FLINK-11447:
-
Hi, Thanks [~twalthr] I want to share how to implement this JIRA, detail as
follows: [Google
doc|https://docs.google.com/document/d/1rEeEKwEhw2S2l92hWqmOSn-PyV7mMnHcnBaqsBtjma0/edit?usp=sharing]
Java users should use `new Table(udf(..))` when call the lateral join. Users
cannot use `new Table()` If the implement of Table do not in the same module
with Table Interface. So to extract the table interface, we should deprecate
`new Table()`.
Currently to get a Lateral Table, the usage of SQL and tableAPI is as follows
(java):
* SQL - SELECT c, s FROM MyTable, *LATERAL TABLE(func1(c))* AS T(s)
* TableAPI - left.join(*new Table(tableEnv, "split(c)"*).as("s", "t", "v"))
To deprecate the `new Table()`, we have three ways as follows:
# TableEnvironment.scan - Currently scan a table name will return a Table
instance. And we can expand scan a udtf to get a Table. e.g.:
{code:java}
myTable.join(tableEnv.scan("split(a) as (word, length)")){code}
Of course we have to check the legality of the UDTF input parameters.
# TableEnvironment.lateralTable - We can add a new method for create a Table
by a UDTF parameter. which consistent with the syntax of sql. e.g.:
{code:java}
myTable.join(tableEnv.lateralTable("split(a)”).as (“word, length")){code}
# We can add the “join(udtf: String)” to table which will make java and scala
look more consistent. e.g:
{code:java}
myTable.join("split(a)”).as (“word, length)"){code}
But from the semantic perspective of join, the join parameter should be a Table.
>From the points of my view, I like the second approach.
What do you think?
Best, Jincheng
> Deprecate "new Table(TableEnvironment, String)"
> ---
>
> Key: FLINK-11447
> URL: https://issues.apache.org/jira/browse/FLINK-11447
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Once table is an interface we can easily replace the underlying
> implementation at any time. The constructor call prevents us from converting
> it into an interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
