[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723939#comment-16723939 ] sunjincheng commented on FLINK-9555: I think you are right! (y) > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723910#comment-16723910 ] Chesnay Schepler commented on FLINK-9555: - Personally I wouldn't merge it to 1.7. It is a new feature after all, that, as we have already shown, may impact existing features. Given that this issue went through without being detected it also seems like there's not enough test coverage for the scala-shell in general. > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723896#comment-16723896 ] Chesnay Schepler commented on FLINK-9555: - Yes, I reverted it due to FLINK-11161. [~sunjincheng121] What do you mean? > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723899#comment-16723899 ] sunjincheng commented on FLINK-9555: I suggest both FLINK-9555 and FLINK-11161 should push into the release-1.7 branch. It's that make sense to you? > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722166#comment-16722166 ] sunjincheng commented on FLINK-9555: [~Zentol] Why don't we solve it by file a new jira? > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722161#comment-16722161 ] Jeff Zhang commented on FLINK-9555: --- [~Zentol] Do you mean the bug of FLINK-11161 ? > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710993#comment-16710993
]
ASF GitHub Bot commented on FLINK-9555:
---
asfgit closed pull request #7121: [FLINK-9555]Support table api in scala shell
URL: https://github.com/apache/flink/pull/7121
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-scala-shell/pom.xml b/flink-scala-shell/pom.xml
index 3c9a563b30f..e7d31355b06 100644
--- a/flink-scala-shell/pom.xml
+++ b/flink-scala-shell/pom.xml
@@ -78,6 +78,13 @@ under the License.
${scala.version}
+
+ org.apache.flink
+
flink-table_${scala.binary.version}
+ ${project.version}
+ provided
+
+
diff --git
a/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
b/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
index 4b6e886994a..c124d8ea8a3 100644
---
a/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
+++
b/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
@@ -23,6 +23,8 @@ import java.io.{BufferedReader, File, FileOutputStream}
import org.apache.flink.api.java.{JarHelper, ScalaShellRemoteEnvironment,
ScalaShellRemoteStreamEnvironment}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.configuration.Configuration
+import org.apache.flink.table.api.TableEnvironment
+import org.apache.flink.table.api.scala.{BatchTableEnvironment,
StreamTableEnvironment}
import org.apache.flink.util.AbstractID
import scala.tools.nsc.interpreter._
@@ -90,10 +92,17 @@ class FlinkILoop(
}
// local environment
- val (scalaBenv: ExecutionEnvironment, scalaSenv: StreamExecutionEnvironment)
= {
+ val (
+scalaBenv: ExecutionEnvironment,
+scalaSenv: StreamExecutionEnvironment,
+scalaBTEnv: BatchTableEnvironment,
+scalaSTEnv: StreamTableEnvironment
+) = {
val scalaBenv = new ExecutionEnvironment(remoteBenv)
val scalaSenv = new StreamExecutionEnvironment(remoteSenv)
-(scalaBenv,scalaSenv)
+val scalaBTEnv = TableEnvironment.getTableEnvironment(scalaBenv)
+val scalaSTEnv = TableEnvironment.getTableEnvironment(scalaSenv)
+(scalaBenv,scalaSenv,scalaBTEnv,scalaSTEnv)
}
/**
@@ -139,7 +148,10 @@ class FlinkILoop(
"org.apache.flink.api.scala._",
"org.apache.flink.api.scala.utils._",
"org.apache.flink.streaming.api.scala._",
-"org.apache.flink.streaming.api.windowing.time._"
+"org.apache.flink.streaming.api.windowing.time._",
+"org.apache.flink.table.api._",
+"org.apache.flink.table.api.scala._",
+"org.apache.flink.types.Row"
)
override def createInterpreter(): Unit = {
@@ -152,6 +164,8 @@ class FlinkILoop(
// set execution environment
intp.bind("benv", this.scalaBenv)
intp.bind("senv", this.scalaSenv)
+ intp.bind("btenv", this.scalaBTEnv)
+ intp.bind("stenv", this.scalaSTEnv)
}
}
@@ -243,22 +257,29 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
+*
+* val batchTable = btenv.fromDataSet(dataSet)
+* btenv.registerTable("tableName", batchTable)
+* val result = btenv.sqlQuery("SELECT * FROM tableName").collect
+HINT: You can use print() on a DataSet to print the contents or collect()
+a sql query result back to the shell.
-HINT: You can use print() on a DataSet to print the contents to the shell.
-
- Streaming - Use the 'senv' variable
+ Streaming - Use the 'senv' and 'stenv' variable
* val dataStream = senv.fromElements(1, 2, 3, 4)
* dataStream.countWindowAll(2).sum(0).print()
+*
+* val streamTable = stenv.fromDataStream(dataStream, 'num)
+* val resultTable = streamTable.select('num).where('num % 2 === 1 )
+* resultTable.toAppendStream[Row].print()
* senv.execute("My streaming program")
-
HINT: You can only print a DataStream to the shell in local mode.
"""
// scalastyle:on
diff --git
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710906#comment-16710906 ] ASF GitHub Bot commented on FLINK-9555: --- sunjincheng121 commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-444726128 Thanks for the update！ @shuiqiangchen Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709623#comment-16709623
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r238935139
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -243,22 +257,29 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
+*
+* val batchTable = btenv.fromDataSet(dataSet)
+* btenv.registerTable("tableName", batchTable)
+* val result = btenv.sqlQuery("SELECT * FROM tableName").collect
+HINT: You can use print() on a DataSet to print the contents and collect()
+a sql query result back to the shell.
Review comment:
`contents and collect()` -> `contents or collect()` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709582#comment-16709582 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-444350634 Hi all, I have updated some codes following your suggestions. Please review it in your convenience. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708389#comment-16708389 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-444015671 > @shuiqiangchen Travis is failed, could you check that ? Thanks Thanks! It's the matter of checkStyle. I have fixed it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708356#comment-16708356 ] ASF GitHub Bot commented on FLINK-9555: --- zjffdu commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-444009960 @shuiqiangchen Travis is failed, could you check that ? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707964#comment-16707964 ] ASF GitHub Bot commented on FLINK-9555: --- sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r238475160 ## File path: flink-scala-shell/start-script/start-scala-shell.sh ## @@ -52,6 +52,13 @@ bin=`cd "$bin"; pwd` FLINK_CLASSPATH=`constructFlinkClassPath` +#adding flink table jar into class path Review comment: `#adding` -> `# adding` `# Append flink-table jar into class path` ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707961#comment-16707961
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r238474488
##
File path:
flink-scala-shell/src/test/scala/org/apache/flink/api/scala/ScalaShellITCase.scala
##
@@ -168,6 +168,56 @@ class ScalaShellITCase extends TestLogger {
Assert.assertTrue(output.contains("WC(world,10)"))
}
+ @Test
+ def testSimpleSelectWithFilterBatchTableAPIQuery: Unit = {
+val input =
+ """
+|import _root_.scala.collection.mutable
+|val data = new mutable.MutableList[(Int, Long, String)]
+|data.+=((1, 1L, "Hi"))
+|data.+=((2, 2L, "Hello"))
+|data.+=((3, 2L, "Hello world"))
+|val t = benv.fromCollection(data).toTable(btenv, 'a, 'b,
'c).select("*").where('a % 2 === 1 )
+|val results = t.toDataSet[Row].collect()
+|results.foreach(println)
+|:q
+ """.stripMargin
+val output = processInShell(input)
+Assert.assertFalse(output.contains("failed"))
+Assert.assertFalse(output.contains("error"))
+Assert.assertFalse(output.contains("Exception"))
+ }
+
+ @Test
+ def testGroupedAggregationStreamTableAPIQuery: Unit = {
+val input =
+ """
+| val data = List(
+|("Hello", 1),
+|("word", 1),
+|("Hello", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("flink", 1)
+| )
+| val stream = senv.fromCollection(data)
+| val table = stream.toTable(stenv, 'word, 'num)
+| val resultTable = table.groupBy('word).select('num.sum as
'count).groupBy('count).select(
+| 'count,'count.count as 'frequency)
+| val results = resultTable.toRetractStream[Row]
+| results.print
+| senv.execute
+ """.stripMargin
+val output = processInShell(input)
+Assert.assertFalse(output.contains("failed"))
+Assert.assertFalse(output.contains("error"))
+Assert.assertFalse(output.contains("Exception"))
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707098#comment-16707098 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r238244696 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +148,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" Review comment: Sorry, I misunderstood, it should be import initially to support window operations. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704735#comment-16704735 ] ASF GitHub Bot commented on FLINK-9555: --- twalthr commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r237859698 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +146,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.streaming.api.windowing.time._", Review comment: This is important to define windows or interval joins in DataStream API. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704733#comment-16704733 ] ASF GitHub Bot commented on FLINK-9555: --- twalthr commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r237859231 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +148,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" Review comment: Why did you remove this line? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704207#comment-16704207
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r237735688
##
File path:
flink-scala-shell/src/test/scala/org/apache/flink/api/scala/ScalaShellITCase.scala
##
@@ -168,6 +168,56 @@ class ScalaShellITCase extends TestLogger {
Assert.assertTrue(output.contains("WC(world,10)"))
}
+ @Test
+ def testSimpleSelectWithFilterBatchTableAPIQuery: Unit = {
+val input =
+ """
+|import _root_.scala.collection.mutable
+|val data = new mutable.MutableList[(Int, Long, String)]
+|data.+=((1, 1L, "Hi"))
+|data.+=((2, 2L, "Hello"))
+|data.+=((3, 2L, "Hello world"))
+|val t = benv.fromCollection(data).toTable(btenv, 'a, 'b,
'c).select("*").where('a % 2 === 1 )
Review comment:
MutableList is not necessary here. Use this instead
```
val data = Seq(
(1, 1L, "Hi),
(2, 2L, "Hello"),
(3, 2L, "Hello world")
)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704209#comment-16704209
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r237735988
##
File path:
flink-scala-shell/src/test/scala/org/apache/flink/api/scala/ScalaShellITCase.scala
##
@@ -168,6 +168,56 @@ class ScalaShellITCase extends TestLogger {
Assert.assertTrue(output.contains("WC(world,10)"))
}
+ @Test
+ def testSimpleSelectWithFilterBatchTableAPIQuery: Unit = {
+val input =
+ """
+|import _root_.scala.collection.mutable
+|val data = new mutable.MutableList[(Int, Long, String)]
+|data.+=((1, 1L, "Hi"))
+|data.+=((2, 2L, "Hello"))
+|data.+=((3, 2L, "Hello world"))
+|val t = benv.fromCollection(data).toTable(btenv, 'a, 'b,
'c).select("*").where('a % 2 === 1 )
+|val results = t.toDataSet[Row].collect()
+|results.foreach(println)
+|:q
+ """.stripMargin
+val output = processInShell(input)
+Assert.assertFalse(output.contains("failed"))
+Assert.assertFalse(output.contains("error"))
+Assert.assertFalse(output.contains("Exception"))
Review comment:
We also need to verify the output contains "Hi", "Hello world" but not
"Hello".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704210#comment-16704210
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r237736038
##
File path:
flink-scala-shell/src/test/scala/org/apache/flink/api/scala/ScalaShellITCase.scala
##
@@ -168,6 +168,56 @@ class ScalaShellITCase extends TestLogger {
Assert.assertTrue(output.contains("WC(world,10)"))
}
+ @Test
+ def testSimpleSelectWithFilterBatchTableAPIQuery: Unit = {
+val input =
+ """
+|import _root_.scala.collection.mutable
+|val data = new mutable.MutableList[(Int, Long, String)]
+|data.+=((1, 1L, "Hi"))
+|data.+=((2, 2L, "Hello"))
+|data.+=((3, 2L, "Hello world"))
+|val t = benv.fromCollection(data).toTable(btenv, 'a, 'b,
'c).select("*").where('a % 2 === 1 )
+|val results = t.toDataSet[Row].collect()
+|results.foreach(println)
+|:q
+ """.stripMargin
+val output = processInShell(input)
+Assert.assertFalse(output.contains("failed"))
+Assert.assertFalse(output.contains("error"))
+Assert.assertFalse(output.contains("Exception"))
+ }
+
+ @Test
+ def testGroupedAggregationStreamTableAPIQuery: Unit = {
+val input =
+ """
+| val data = List(
+|("Hello", 1),
+|("word", 1),
+|("Hello", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("bark", 1),
+|("flink", 1)
+| )
+| val stream = senv.fromCollection(data)
+| val table = stream.toTable(stenv, 'word, 'num)
+| val resultTable = table.groupBy('word).select('num.sum as
'count).groupBy('count).select(
+| 'count,'count.count as 'frequency)
+| val results = resultTable.toRetractStream[Row]
+| results.print
+| senv.execute
+ """.stripMargin
+val output = processInShell(input)
+Assert.assertFalse(output.contains("failed"))
+Assert.assertFalse(output.contains("error"))
+Assert.assertFalse(output.contains("Exception"))
Review comment:
We also need to verify the output contains the correct word count result.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704205#comment-16704205
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r237735688
##
File path:
flink-scala-shell/src/test/scala/org/apache/flink/api/scala/ScalaShellITCase.scala
##
@@ -168,6 +168,56 @@ class ScalaShellITCase extends TestLogger {
Assert.assertTrue(output.contains("WC(world,10)"))
}
+ @Test
+ def testSimpleSelectWithFilterBatchTableAPIQuery: Unit = {
+val input =
+ """
+|import _root_.scala.collection.mutable
+|val data = new mutable.MutableList[(Int, Long, String)]
+|data.+=((1, 1L, "Hi"))
+|data.+=((2, 2L, "Hello"))
+|data.+=((3, 2L, "Hello world"))
+|val t = benv.fromCollection(data).toTable(btenv, 'a, 'b,
'c).select("*").where('a % 2 === 1 )
Review comment:
MutableList is not necessary here.
```
val data = Seq(
(1, 1L, "Hi),
(2, 2L, "Hello"),
(3, 2L, "Hello world")
)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701788#comment-16701788 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-442425068 Hi All, Thank you for your attention! I have pushed new codes according to your suggestions. Looking forward to your guidance. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699898#comment-16699898 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r236513423 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +147,8 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.table.api._", +"org.apache.flink.table.api.scala._" Review comment: Agree +1，I will add it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699894#comment-16699894 ] ASF GitHub Bot commented on FLINK-9555: --- zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r236512499 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +147,8 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.table.api._", +"org.apache.flink.table.api.scala._" Review comment: Could we also import `org.apache.flink.types.Row` which is used very frequently I believe ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699856#comment-16699856 ] ASF GitHub Bot commented on FLINK-9555: --- zjffdu commented on issue #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#issuecomment-441880219 Thanks @shuiqiangchen , I left a few comments. And agree with @sunjincheng121 , we need to add test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699855#comment-16699855
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236504524
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -243,22 +254,22 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
-
+* val batchTable = btenv.sqlQuery("SELECT * FROM tableName")
HINT: You can use print() on a DataSet to print the contents to the shell.
- Streaming - Use the 'senv' variable
+ Streaming - Use the 'senv' and 'stenv' variable
* val dataStream = senv.fromElements(1, 2, 3, 4)
* dataStream.countWindowAll(2).sum(0).print()
* senv.execute("My streaming program")
-
+* val streamTable = stenv.sqlQuery("SELECT * FROM tableName")
HINT: You can only print a DataStream to the shell in local mode.
Review comment:
Add hint of how to see the table output in scala-shell
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699853#comment-16699853
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236504436
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -243,22 +254,22 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
-
+* val batchTable = btenv.sqlQuery("SELECT * FROM tableName")
HINT: You can use print() on a DataSet to print the contents to the shell.
- Streaming - Use the 'senv' variable
+ Streaming - Use the 'senv' and 'stenv' variable
* val dataStream = senv.fromElements(1, 2, 3, 4)
* dataStream.countWindowAll(2).sum(0).print()
* senv.execute("My streaming program")
-
+* val streamTable = stenv.sqlQuery("SELECT * FROM tableName")
Review comment:
Add example of registering table via `senv` then query it via `stenv`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699854#comment-16699854
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236504513
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -243,22 +254,22 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
-
+* val batchTable = btenv.sqlQuery("SELECT * FROM tableName")
HINT: You can use print() on a DataSet to print the contents to the shell.
Review comment:
Add hint of how to see the table output in scala-shell
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699851#comment-16699851
]
ASF GitHub Bot commented on FLINK-9555:
---
zjffdu commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236504392
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -243,22 +254,22 @@ class FlinkILoop(
F L I N K - S C A L A - S H E L L
-NOTE: Use the prebound Execution Environments to implement batch or streaming
programs.
+NOTE: Use the prebound Execution Environments and Table Environment to
implement batch or streaming programs.
- Batch - Use the 'benv' variable
+ Batch - Use the 'benv' and 'btenv' variable
* val dataSet = benv.readTextFile("/path/to/data")
* dataSet.writeAsText("/path/to/output")
* benv.execute("My batch program")
-
+* val batchTable = btenv.sqlQuery("SELECT * FROM tableName")
Review comment:
Add example of registering table via `benv` then query it via `btenv`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699850#comment-16699850
]
ASF GitHub Bot commented on FLINK-9555:
---
shuiqiangchen commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236503717
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -96,6 +97,12 @@ class FlinkILoop(
(scalaBenv,scalaSenv)
}
+ val (scalaBTEnv: BatchTableEnvironment, scalaSTEnv: StreamTableEnvironment)
= {
Review comment:
Thanks! It seems much more well organized.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699848#comment-16699848
]
ASF GitHub Bot commented on FLINK-9555:
---
shuiqiangchen commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r236503478
##
File path: flink-dist/pom.xml
##
@@ -319,7 +319,6 @@ under the License.
org.apache.flink
flink-table_${scala.binary.version}
${project.version}
- provided
Review comment:
Thank you for your valuable suggestions! I have modified the scala shell
start script to add the flink-table_*.jar to FLINK_CLASSPATH. In this way, we
have no need to change the flink-dist module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691556#comment-16691556
]
ASF GitHub Bot commented on FLINK-9555:
---
zentol commented on a change in pull request #7121: [FLINK-9555]Support table
api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234572365
##
File path: flink-dist/pom.xml
##
@@ -319,7 +319,6 @@ under the License.
org.apache.flink
flink-table_${scala.binary.version}
${project.version}
- provided
Review comment:
I agree that this change should be reverted but flink-table should remain in
/opt. If a user wants to use flink-table (or any connector for that matter) the
required jars should be copied manually to /lib.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691205#comment-16691205
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234481358
##
File path: flink-dist/pom.xml
##
@@ -319,7 +319,6 @@ under the License.
org.apache.flink
flink-table_${scala.binary.version}
${project.version}
- provided
Review comment:
I suggest that revert this change, because `flink-dist` is the whole release
strategy of flink. At present, we package the `flink-table` module to the `opt`
directory. The minimum change I suggest is to modify the scala-shell CLASSPATH
and include `opt/opt/flink-tableXXX.jar` in the CLASSPATH， to support tables in
`scala-shell`, What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691206#comment-16691206
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234478589
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -96,6 +97,12 @@ class FlinkILoop(
(scalaBenv,scalaSenv)
}
+ val (scalaBTEnv: BatchTableEnvironment, scalaSTEnv: StreamTableEnvironment)
= {
Review comment:
I suggest that merge two local environment init logic as follows:
```
// local environment
val (
scalaBenv: ExecutionEnvironment,
scalaSenv: StreamExecutionEnvironment,
scalaBTabEnv: BatchTableEnvironment,
scalaSTabEnv: StreamTableEnvironment) = {
val scalaBenv = new ExecutionEnvironment(remoteBenv)
val scalaSenv = new StreamExecutionEnvironment(remoteSenv)
val scalaBTabEnv = TableEnvironment.getTableEnvironment(scalaBenv)
val scalaSTabEnv = TableEnvironment.getTableEnvironment(scalaSenv)
(scalaBenv, scalaSenv, scalaBTabEnv, scalaSTabEnv)
}```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691207#comment-16691207 ] ASF GitHub Bot commented on FLINK-9555: --- sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r234478740 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +146,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.streaming.api.windowing.time._", Review comment: Why we should import `windowing.time`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689126#comment-16689126 ] ASF GitHub Bot commented on FLINK-9555: --- shuiqiangchen opened a new pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121 ## What is the purpose of the change make table api available in scala shell so that user can experience table api in interactive way. ## Brief change log - add flink-table dependency to flink-scala-shell module. - The scala shell provides a BatchTableEnvironment and a StreamTableEnvironment initially when starting ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) yes - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) no - The serializers: (yes / no / don't know) no - The runtime per-record code paths (performance sensitive): (yes / no / don't know) no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) no - The S3 file system connector: (yes / no / don't know) no ## Documentation - Does this pull request introduce a new feature? (yes / no) yes - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) not documented This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688986#comment-16688986 ] Jeff Zhang commented on FLINK-9555: --- Thanks [~csq] Basically, this is what I'd like to achieve, could you create a PR for it so that others can review it for details ? > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688983#comment-16688983 ] shuiqiangchen commented on FLINK-9555: -- Hi [~zjffdu] and [~sunjincheng121], I have enabled Table API in scala shell by adding flink-table dependency to flink-scala-shell module. The scala shell provides a BatchTableEnvironment and a StreamTableEnvironment initially when starting. How do you think of it? Any good advice and suggestions will be highly appreciated. > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684704#comment-16684704 ] sunjincheng commented on FLINK-9555: Thanks for report this issue Jeff Zhang ！ +1 to make TableApi supported by default in Scala-shell. > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505791#comment-16505791 ] Jeff Zhang commented on FLINK-9555: --- I mean to support it natively so that user don't need to specify table api as dependency and create TableEnviroment explicitly. > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505785#comment-16505785 ] Dawid Wysakowicz commented on FLINK-9555: - I think you can use table API already. Just add the table api dependency to scala-shell. > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505781#comment-16505781 ] Jeff Zhang commented on FLINK-9555: --- I will work on it > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Priority: Major > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
