[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-08 Thread haohui 
Github user haohui closed the pull request at:

https://github.com/apache/flink/pull/4046


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749951
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
--- End diff --

Add a link to 
http://localhost:4000/dev/table/common.html#register-an-external-catalog


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753514
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
--- End diff --

indention. I think we should format most following queries (everything 
that's not `SELECT FROM`) in multiple lines, for example like:
```
SELECT ...
FROM ...
GROUP BY ...
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750870
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
 
-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not yet feature complete. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
 
 * This will be replaced by the TOC
 {:toc}
 
 Specifying a Query
 ---
 
+Here are a few examples on specifying a DataStream / DataSet program using 
SQL:
--- End diff --

I would not mention that SQL queries are translated into DataStream / 
DataSet programs. This is useful information for some users and it should be 
mentioned in an internals section. I would assume that most users do not care 
about this and might be even confused.

What do you think @haohui ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755990
  
--- Diff: docs/dev/table/sql.md ---
@@ -217,7 +444,7 @@ Group windows are defined in the `GROUP BY` clause of a 
SQL query. Just like que
   
 
 
-For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must be one of the `rowtime()` or `proctime()` time-indicators, 
which distinguish between event or processing time, respectively. For SQL on 
batch tables, the `time_attr` argument of the group window function must be an 
attribute of type `TIMESTAMP`. 
+For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must refer to the virtual column that specifies the processing 
time or the event time. For SQL on batch tables, the `time_attr` argument of 
the group window function must be an attribute of type `TIMESTAMP`.
--- End diff --

Add a link to the time attribute section on the Table API Streaming doc 
page.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754779
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

`TableFunction` is not nicely rendered in my setup. Can you 
check on yours?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755168
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that Flink does not optimize join order yet and joins in the 
same order as specified in the query.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120756834
  
--- Diff: docs/dev/table/sql.md ---
@@ -340,18 +567,14 @@ val result4 = tableEnv.sql(
 
 {% top %}
 
-### Limitations
-
-**TODO: Integrate this with the examples**
-
  Batch
--- End diff --

I would remove the `Batch` and `Streaming` limitations sections. 
Instead, I would integrate the relevant information with the `Operations` 
sections, e.g., add a comments to UNNEST that WITH ORDINALITY is not supported, 
etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754991
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

TableFunctions do also accept multiple values. Add a link to the UDF docs 
page.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755459
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
--- End diff --

Add a line for `User defined aggregation function`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749760
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

Add links to the relevant sections of the Overview page:
* http://localhost:4000/dev/table/common.html#mixing-table-api-and-sql
* 
http://localhost:4000/dev/table/common.html#integration-with-datastream-and-dataset-api
* http://localhost:4000/dev/table/common.html#emit-a-table


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752386
  
--- Diff: docs/dev/table/sql.md ---
@@ -82,14 +84,12 @@ val result2 = tableEnv.sql(
 
 
 
-**TODO: Add some intro.**
-
 {% top %}
 
 Supported Syntax
 
 
-Flink uses [Apache 
Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. 
Currently, Flink SQL only supports query-related SQL syntax and only a subset 
of the comprehensive SQL standard. The following BNF-grammar describes the 
supported SQL features:
+Flink parses SQL using [Apache 
Calcite](https://calcite.apache.org/docs/reference.html). Flink supports 
standard ANSI SQL, but it provides no supports for DML and DDL. The following 
BNF-grammar describes the supported SQL features:
--- End diff --

We should mention that not all features are supported by batch and 
streaming. The `Operations` section will show which features are supported by 
batch and streaming.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752941
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
--- End diff --

User Defined Scalar Functions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754137
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Add a brief description that only all aggregates must be defined over the 
same window, i.e., we do not support aggregates over different windows.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750248
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
--- End diff --

`... will automatically register the table under a unique name in its 
TableEnvironment and ...`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753745
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Over windows are only supported for streaming


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753703
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
--- End diff --

Add a space between `Distinct` and `(Batch only)`. Same for the following 
`(Batch only)` labels.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754480
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that join must have at least one conjunctive equality 
predicate. CROSS or Theta joins are not supported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752761
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
--- End diff --

Markdown highlighting does not seem to work here. We should also add a link 
to the UDF docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-07 Thread fhueske 
Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120751070
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

I would not mention the translation into DataStream and DataSet programs 
(see another comment below).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119893904
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, 
including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the 
`TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can 
also register external catalogs in the `TableEnvironment` to specify the 
location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique 
table name under the `Table`'s `TableEnvironment` and return the table name. So 
it allows to call SQL directly on tables in a string concatenation (see 
examples below).
--- End diff --

... a unique table name ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119894456
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, 
including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the 
`TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can 
also register external catalogs in the `TableEnvironment` to specify the 
location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique 
table name under the `Table`'s `TableEnvironment` and return the table name. So 
it allows to call SQL directly on tables in a string concatenation (see 
examples below).
 
-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
 
 * This will be replaced by the TOC
 {:toc}
 
 Specifying a Query
 ---
 
+Here are a few examples on how to specify a DataStream / DataSet program 
using SQL:
--- End diff --

examples of how to


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119894379
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, 
including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the 
`TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can 
also register external catalogs in the `TableEnvironment` to specify the 
location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique 
table name under the `Table`'s `TableEnvironment` and return the table name. So 
it allows to call SQL directly on tables in a string concatenation (see 
examples below).
 
-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
--- End diff --

Flink's SQL support is not yet feature complete.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119896766
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, 
including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the 
`TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can 
also register external catalogs in the `TableEnvironment` to specify the 
location of the data sources.
--- End diff --

This reads a bit awkwardly. How about this?

Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include `Table`s, `DataSet`s, `DataStream`s, and external `TableSource`s. 
Alternatively, ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119894264
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, 
including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the 
`TableEnvironment` (see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can 
also register external catalogs in the `TableEnvironment` to specify the 
location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique 
table name under the `Table`'s `TableEnvironment` and return the table name. So 
it allows to call SQL directly on tables in a string concatenation (see 
examples below).
--- End diff --

This allows SQL to be called directly on tables in a string concatenation 
(see examples below).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119893598
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in the subsequent 
SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in 
subsequent Table API queries or written to a `TableSink` (see [Writing Tables 
to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries 
can seamlessly mixed and are holistically optimized and translated into a 
single program.
--- End diff --

A `Table` can be used in subsequent SQL / Table API queries, be converted 
into a `DataSet` or `DataStream`, or written to a `TableSink` ...

_(drop "the" and the redundant "used in subsequent Table API queries")_




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-02 Thread alpinegizmo 
Github user alpinegizmo commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r119894751
  
--- Diff: docs/dev/table/sql.md ---
@@ -82,14 +84,12 @@ val result2 = tableEnv.sql(
 
 
 
-**TODO: Add some intro.**
-
 {% top %}
 
 Supported Syntax
 
 
-Flink uses [Apache 
Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. 
Currently, Flink SQL only supports query-related SQL syntax and only a subset 
of the comprehensive SQL standard. The following BNF-grammar describes the 
supported SQL features:
+Flink parses SQL using [Apache 
Calcite](https://calcite.apache.org/docs/reference.html). Flink supports the 
standard ANSI SQL but it provides no supports for DML and DDL. The following 
BNF-grammar describes the supported SQL features:
--- End diff --

Flink supports standard ANSI SQL, but it provides no support for DML or DDL.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

2017-06-01 Thread haohui 
GitHub user haohui opened a pull request:

https://github.com/apache/flink/pull/4046

[FLINK-6749] [table] Table API / SQL Docs: SQL Page

This first iteration of the documentation. Comments are appreciated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haohui/flink FLINK-6749

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4046


commit 15a5654a3143db79d1df5b9d14abaa5540fc58e0
Author: Haohui Mai 
Date:   2017-06-01T22:46:05Z

[FLINK-6749] [table] Table API / SQL Docs: SQL Page.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---