Hi all,
thanks for the great feedback! I see a clear majority for a new Java DSL.
I agree with Dawid that we should aim for 1.9. Compared to other
efforts, this is a relatively small change ;-)
I will further develop my protoype and come back to you once I have a
production-ready solution.
Regards,
Timo
Am 26.03.19 um 11:06 schrieb Elias Saalmann:
Hi all,
I'm investigating using Table-API as an abstraction for Batch-API in
our GRADOOP project [1]. As a user of Flink I like to give you some
feedback from the user perspective.
I really like Timo's ideas - since we built another framework which is
based on Apache Flink we are just providing another API for our users.
Using java string literals (or building boolean expressions via string
builder, for instance) in our code isn't convenient. Therefore I built
little helpers for our special use cases:
- A builder [2] to build a single Expression, e.g.
ExpressionBuilder builder = new ExpressionBuilder();
Expression e = builder
.field("foo")
.as("bar")
.toExpression();
- A builder [3] to build a Seq<Expression>, e.g.
ExpressionSeqBuilder builder = new ExpressionSeqBuilder();
Seq<Expression> seq = builder
.field("foo")
.as("a")
.field("bar")
.as("b")
.buildSeq()
Using those builders we make use of those "select(Expression)" API
methods. Of course, we would appreciate a DSL provided by Flink!
Best,
Elias
[1] https://dbs.uni-leipzig.de/en/research/projects/gradoop
[2]
https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionBuilder.java
[3]
https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionSeqBuilder.java
Am 26.03.19 um 10:22 schrieb jincheng sun:
Thanks for bringing up this DISCUSS Timo!
Java Expression DSL is pretty useful for java user. When we have the
Java
Expression DSL, Java API will become very rich and easy to use!
+1 from my side.
Best,
Jincheng
Dawid Wysakowicz <dwysakow...@apache.org> 于2019年3月26日周二
下午5:08写道：
Hi,
I really like the idea of introducing Java Expression DSL. I think this
will solve many problems e.g. right now it's quite tricky how string
literals work in scala (sometimes it might go through the
ExpressionParser
and it will end up as an UnresolvedFieldReference), another important
problem we could solve with this is the need for unique column names in
tables right now. We could at some point introduce sth like:
Table table = ...
table.field("fieldName")
and etc. A common "entry point" to expressions should simplify a lot.
Therefore I am strongly +1 for introducing this feature.
@Jark I think we could aim to introduce the new Java DSL API in 1.9 and
once we do that we could deprecate the string approach.
Best,
Dawid
On 22/03/2019 03:36, Jark Wu wrote:
Hi Timo,
Sounds good to me.
Do you want to deprecate the string-based API in 1.9 or make the
decision
in 1.10 after some feedbacks ?
On Thu, 21 Mar 2019 at 21:32, Timo Walther <twal...@apache.org> wrote:
Thanks for your feedback Rong and Jark.
@Jark: Yes, you are right that the string-based API is used quite a
lot.
On the other side, the potential user base in the future is still
bigger
than our current user base. Because the Table API will become equally
important as the DataStream API, we really need to fix some crucial
design
decisions before it is too late. I would suggest to introduce the
new DSL
in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
developement point of view, I think we can handle the overhead to
maintain
3 APIs until then because 2 APIs will share the same code base +
expression
parser.
Regards,
Timo
Am 21.03.19 um 05:21 schrieb Jark Wu:
Hi Timo,
I'm +1 on the proposal. I like the idea to provide a Java DSL which is
more friendly than string-based approach in programming.
My concern is if/when we can drop the string-based expression
parser. If
it takes a very long time, we have to paid more development
cost on the three Table APIs. As far as I know, the string-based
API is
used in many companies.
We should also get some feedbacks from users. So I'm CCing this
email to
user mailing list.
Best,
Jark
On Wed, 20 Mar 2019 at 08:51, Rong Rong <walter...@gmail.com> wrote:
Thanks for sharing the initiative of improving Java side Table
expression
DSL.
I agree as in the doc stated that Java DSL was always a "3rd class
citizen"
and we've run into many hand holding scenarios with our Flink
developers
trying to get the Stringify syntax working.
Overall I am a +1 on this, it also help reduce the development
cost of
the
Table API so that we no longer need to maintain different DSL and
documentations.
I left a few comments in the doc. and also some features that I think
will
be beneficial to the final outcome. Please kindly take a look @Timo.
Many thanks,
Rong
On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <twal...@apache.org>
wrote:
Hi everyone,
some of you might have already noticed the JIRA issue that I opened
recently [1] about introducing a proper Java expression DSL for the
Table API. Instead of using string-based expressions, we should
aim for
a unified, maintainable, programmatic Java DSL.
Some background: The Blink merging efforts and the big
refactorings as
part of FLIP-32 have revealed many shortcomings in the current
Table &
SQL API design. Most of these legacy issues cause problems
nowadays in
making the Table API a first-class API next to the DataStream
API. An
example is the ExpressionParser class[2]. It was implemented in the
early days of the Table API using Scala parser combinators.
During the
last years, this parser caused many JIRA issues and user
confusion on
the mailing list. Because the exceptions and syntax might not be
straight forward.
For FLINK-11908, we added a temporary bridge instead of
reimplementing
the parser in Java for FLIP-32. However, this is only a intermediate
solution until we made a final decision.
I would like to propose a new, parser-free version of the Java Table
API:
https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
I already implemented an early protoype that shows that such a
DSL is
not much implementation effort and integrates nicely with all
existing
API methods.
What do you think?
Thanks for your feedback,
Timo
[1] https://issues.apache.org/jira/browse/FLINK-11890
[2]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
