[jira] [Commented] (FLINK-4180) Create a batch SQL example

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390913#comment-15390913
 ] 

ASF GitHub Bot commented on FLINK-4180:
---

Github user smarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2274#discussion_r71981015
  
--- Diff: 
flink-libraries/flink-table/src/main/java/org/apache/flink/examples/java/JavaSQLExample.java
 ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.examples.java;
+
+import org.apache.flink.api.table.Table;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.table.BatchTableEnvironment;
+import org.apache.flink.api.table.TableEnvironment;
+
+/**
+ * Simple example that shows how the Batch SQL used in Java.
+ */
+public class JavaSQLExample {
+
+   public static class WC {
+   public String word;
+   public long frequence;
--- End diff --

knitpick: change that to 'frequency' :)


> Create a batch SQL example
> --
>
> Key: FLINK-4180
> URL: https://issues.apache.org/jira/browse/FLINK-4180
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>  Labels: starter
>
> Currently there is no runnable code example in `flink-table` showing a 
> working batch SQL query with the Table API.
> A Scala and Java example should be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2274: [FLINK-4180] [FLINK-4181] [table] add Batch SQL an...

2016-07-23 Thread smarthi 
Github user smarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2274#discussion_r71981015
  
--- Diff: 
flink-libraries/flink-table/src/main/java/org/apache/flink/examples/java/JavaSQLExample.java
 ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.examples.java;
+
+import org.apache.flink.api.table.Table;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.table.BatchTableEnvironment;
+import org.apache.flink.api.table.TableEnvironment;
+
+/**
+ * Simple example that shows how the Batch SQL used in Java.
+ */
+public class JavaSQLExample {
+
+   public static class WC {
+   public String word;
+   public long frequence;
--- End diff --

knitpick: change that to 'frequency' :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4250) Cannot select other than first column from Table

2016-07-23 Thread Suneel Marthi (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390911#comment-15390911
 ] 

Suneel Marthi commented on FLINK-4250:
--

Also 'position' is a reserved keyword and having a column name as 'position' 
throws a similar exception. Agree that we need to add a 'Reserved SQL Keywords' 
list to documentation.

> Cannot select other than first column from Table
> 
>
> Key: FLINK-4250
> URL: https://issues.apache.org/jira/browse/FLINK-4250
> Project: Flink
>  Issue Type: Bug
>  Components: Scala API, Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Priority: Critical
>
> Using the Scala Table API and the {{CsvTableSource}} I cannot select a column 
> from the csv source. The following code:
> {code}
> package com.dataartisans.batch
> import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, SqlTimeTypeInfo}
> import org.apache.flink.api.scala._
> import org.apache.flink.api.table.sources.CsvTableSource
> import org.apache.flink.api.table.{Row, TableEnvironment, Table}
> object CsvTableAPIJob {
>   def main(args: Array[String]): Unit = {
> val env = ExecutionEnvironment.getExecutionEnvironment
> val csvFilePath = "table-jobs/src/main/resources/input.csv"
> val tblEnv = TableEnvironment.getTableEnvironment(env)
> val csvTS = new CsvTableSource(csvFilePath, Array("key", "user", "value", 
> "timestamp"), Array(BasicTypeInfo.INT_TYPE_INFO, 
> BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.DOUBLE_TYPE_INFO, 
> BasicTypeInfo.STRING_TYPE_INFO))
> tblEnv.registerTableSource("foobar", csvTS)
> val input = tblEnv.sql("SELECT user FROM foobar")
> tblEnv.toDataSet[Row](input).print()
>   }
> }
> {code}
> fails with 
> {code}
> Exception in thread "main" 
> org.apache.flink.api.table.codegen.CodeGenException: Unsupported call: USER
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator$$anonfun$visitCall$3.apply(CodeGenerator.scala:782)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator$$anonfun$visitCall$3.apply(CodeGenerator.scala:782)
>   at scala.Option.getOrElse(Option.scala:120)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:782)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:54)
>   at org.apache.calcite.rex.RexCall.accept(RexCall.java:108)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator.generateExpression(CodeGenerator.scala:168)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator$$anonfun$5.apply(CodeGenerator.scala:286)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator$$anonfun$5.apply(CodeGenerator.scala:286)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> org.apache.flink.api.table.codegen.CodeGenerator.generateResultExpression(CodeGenerator.scala:286)
>   at 
> org.apache.flink.api.table.plan.nodes.FlinkCalc$class.functionBody(FlinkCalc.scala:52)
>   at 
> org.apache.flink.api.table.plan.nodes.dataset.DataSetCalc.functionBody(DataSetCalc.scala:39)
>   at 
> org.apache.flink.api.table.plan.nodes.dataset.DataSetCalc.translateToPlan(DataSetCalc.scala:108)
>   at 
> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableEnvironment.scala:271)
>   at 
> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(BatchTableEnvironment.scala:139)
>   at com.dataartisans.batch.CsvTableAPIJob$.main(CsvTableAPIJob.scala:21)
>   at com.dataartisans.batch.CsvTableAPIJob.main(CsvTableAPIJob.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4242) Improve validation exception messages

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390751#comment-15390751
 ] 

ASF GitHub Bot commented on FLINK-4242:
---

GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/2290

[FLINK-4242] [table] Improve validation exception messages

Improve validation exception messages in table api.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink flink-4242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2290


commit d79e03d3bc8e5d4387e78b1a648a26ca465fb442
Author: gallenvara 
Date:   2016-07-23T15:59:01Z

Improve validation exception messages in table api.




> Improve validation exception messages
> -
>
> Key: FLINK-4242
> URL: https://issues.apache.org/jira/browse/FLINK-4242
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> The Table API's validation exceptions could be improved to be more meaningful 
> for users. For example, the following code snippet:
> {code}
> Table inputTable = tableEnv.fromDataStream(env.fromElements(
> Tuple3.of(1, "a", 1.0),
> Tuple3.of(2, "b", 2.0),
> Tuple3.of(3, "c", 3.0)), "a, b, c");
> inputTable.select("a").where("!a");
> {code}
> fails correctly. However, the validation exception message says "Expression 
> !('a) failed on input check: Not only accepts child of Boolean Type, get 
> Integer". I think it could be changed such that it says: "The not operator 
> requires a boolean input but "a" is of type integer." or something similar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2290: [FLINK-4242] [table] Improve validation exception ...

2016-07-23 Thread gallenvara 
GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/2290

[FLINK-4242] [table] Improve validation exception messages

Improve validation exception messages in table api.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink flink-4242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2290


commit d79e03d3bc8e5d4387e78b1a648a26ca465fb442
Author: gallenvara 
Date:   2016-07-23T15:59:01Z

Improve validation exception messages in table api.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390680#comment-15390680
 ] 

ASF GitHub Bot commented on FLINK-3874:
---

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2244
  
Build is now passing. Could someone please review this PR?


> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2244: [FLINK-3874] Add a Kafka TableSink with JSON serializatio...

2016-07-23 Thread mushketyk 
Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2244
  
Build is now passing. Could someone please review this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4029) Multi-field "sum" function just like "keyBy"

2016-07-23 Thread Ivan Mushketyk (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390679#comment-15390679
 ] 

Ivan Mushketyk commented on FLINK-4029:
---

I am new to Flink, so I am a little bit confused about what would be the best 
way to implement this features.
Should I implement it on top of existing operators/streams or should I add a 
new operator/stream?
Should I add the "sum" method to DataStream class or to one of the specialized 
streams?

> Multi-field "sum" function just like "keyBy"
> 
>
> Key: FLINK-4029
> URL: https://issues.apache.org/jira/browse/FLINK-4029
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Rami
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> I can use keyBy as follows:
> stream.keyBy(“pojo.field1”,”pojo.field2”,…)
> Would make sense that I can use sum for example, to do its job for more than 
> one field:
> stream.sum(“pojo.field1”,”pojo.field2”,…)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3866) StringArraySerializer claims type is immutable; shouldn't

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390674#comment-15390674
 ] 

ASF GitHub Bot commented on FLINK-3866:
---

GitHub user mushketyk opened a pull request:

https://github.com/apache/flink/pull/2289

[FLINK-3866] StringArraySerializer type should be mutable

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink mutable-array

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2289


commit 4edb12cfc13c06ca1fb19dfb7042795cf883719c
Author: Ivan Mushketyk 
Date:   2016-07-23T09:39:24Z

[FLINK-3866] StringArraySerializer type should be mutable




> StringArraySerializer claims type is immutable; shouldn't
> -
>
> Key: FLINK-3866
> URL: https://issues.apache.org/jira/browse/FLINK-3866
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.3
>Reporter: Tatu Saloranta
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Looking at default `TypeSerializer` instances I noticed what looks like a 
> minor flaw, unless I am missing something.
> Whereas all other array serializers indicate that type is not immutable 
> (since in Java, arrays are not immutable), `StringArraySerializer` has:
> ```
>   @Override
>   public boolean isImmutableType() {
>   return true;
>   }
> ```
> and I think it should instead return `false`. I could create a PR, but seems 
> like a small enough thing that issue report makes more sense.
> I tried looking for deps to see if there's a test for this, but couldn't find 
> one; otherwise could submit a test fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2289: [FLINK-3866] StringArraySerializer type should be ...

2016-07-23 Thread mushketyk 
GitHub user mushketyk opened a pull request:

https://github.com/apache/flink/pull/2289

[FLINK-3866] StringArraySerializer type should be mutable

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink mutable-array

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2289


commit 4edb12cfc13c06ca1fb19dfb7042795cf883719c
Author: Ivan Mushketyk 
Date:   2016-07-23T09:39:24Z

[FLINK-3866] StringArraySerializer type should be mutable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...

2016-07-23 Thread radekg 
Github user radekg commented on the issue:

https://github.com/apache/flink/pull/2231
  
Tests are failing for random setups on travis. Seems to be something scala 
related.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390636#comment-15390636
 ] 

ASF GitHub Bot commented on FLINK-4035:
---

Github user radekg commented on the issue:

https://github.com/apache/flink/pull/2231
  
Tests are failing for random setups on travis. Seems to be something scala 
related.


> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2282: [FLINK-3940] [table] Add support for ORDER BY OFFSET FETC...

2016-07-23 Thread gallenvara 
Github user gallenvara commented on the issue:

https://github.com/apache/flink/pull/2282
  
@wuchong thanks for your advice and i have addressed your comments in the 
new commit. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3940) Add support for ORDER BY OFFSET FETCH

2016-07-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390600#comment-15390600
 ] 

ASF GitHub Bot commented on FLINK-3940:
---

Github user gallenvara commented on the issue:

https://github.com/apache/flink/pull/2282
  
@wuchong thanks for your advice and i have addressed your comments in the 
new commit. :)


> Add support for ORDER BY OFFSET FETCH
> -
>
> Key: FLINK-3940
> URL: https://issues.apache.org/jira/browse/FLINK-3940
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Fabian Hueske
>Assignee: GaoLun
>Priority: Minor
>
> Currently only ORDER BY without OFFSET and FETCH are supported.
> This issue tracks the effort to add support for OFFSET and FETCH and involves:
> - Implementing the execution strategy in `DataSetSort`
> - adapting the `DataSetSortRule` to support OFFSET and FETCH
> - extending the Table API and validation to support OFFSET and FETCH and 
> generate a corresponding RelNode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)