Re: Converting a DataStream into a Table throws error

Ok gents thanks for clarification. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com

Re: Converting a DataStream into a Table throws error

Hi Mich, I think this is the behavior of the compiler. When run your job in local, you have to remove the provided or add jar to the lib path. But if run on cluster, you have to add the provided to ignore flink classes, since these classes are already exist in your installation version. Best,

Re: Converting a DataStream into a Table throws error

Thanks Fabian, I looked at the maven and this is what it says *provided* [image: image.png] However, this jar file is not shipped with Flink? Is this deliberate? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Converting a DataStream into a Table throws error

The problem is that you declared it as provided. This means the build tool assumes it will be there and therefore does not include it in the Jar file. By adding it to the lib folder you are providing the dependency. Best, Fabian 2018-08-06 18:58 GMT+02:00 Mich Talebzadeh : > Hi, > > I resolved

Re: Converting a DataStream into a Table throws error

Hi, I resolved this issue of java.lang.NoClassDefFoundError: org/apache/flink/table/api/TableEnvironment By adding the jar file flink-table_2.11-1.5.0.jar To $FLINK_HOME/lib It compiles and run OK now. Rather strange as I had this dependency in my SBT libraryDependencies +=

Re: Converting a DataStream into a Table throws error

Apologies should read Vino and Timo Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Converting a DataStream into a Table throws error

Thanks a lot Timo. I will try the changes suggested. Appreciated Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Converting a DataStream into a Table throws error

Hi Mich, I have reviewed your code in the github you provided. I copied your code to org.apache.flink.table.examples.scala under flink-examples-table. It passed the compilation and didn't report the exception you provided, although there are other exceptions (it's about hdfs, this is because of

Re: Converting a DataStream into a Table throws error

Appreciate if anyone had a chance to look at the Scala code in GitHub and advise https://github.com/michTalebzadeh/Flink Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Converting a DataStream into a Table throws error

Thanks Timo, Did as suggested getting this compilation error [info] Compiling 1 Scala source to /home/hduser/dba/bin/flink/md_streaming/target/scala-2.11/classes... [error] /home/hduser/dba/bin/flink/md_streaming/src/main/scala/myPackage/md_streaming.scala:136: could not find implicit value for

Re: Converting a DataStream into a Table throws error

Whenever you use Scala and there is a Scala specific class use it. remove: import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment add: import org.apache.flink.streaming.api.scala._ This will use org.apache.flink.streaming.api.scala.StreamExecutionEnvironment. Timo Am

Re: Converting a DataStream into a Table throws error

Tremendous. Many thanks. Put the sbt build file and the Scala code here https://github.com/michTalebzadeh/Flink Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Converting a DataStream into a Table throws error

Thanks everyone for the advice This worked and passed the compilation error import org.apache.flink.table.api.TableEnvironment import org.apache.flink.table.api.scala._ import org.apache.flink.api.scala._ …. val dataStream = streamExecEnv .addSource(new

Re: Converting a DataStream into a Table throws error

Hi Jorn, Here you go the dependencies libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.2.1" libraryDependencies += "org.apache.hbase" % "hbase" % "1.2.6" libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.2.6" libraryDependencies += "org.apache.hbase" % "hbase-common"

Re: Converting a DataStream into a Table throws error

How does your build.sbt looks especially dependencies? > On 2. Aug 2018, at 00:44, Mich Talebzadeh wrote: > > Changed as suggested > >val streamExecEnv = StreamExecutionEnvironment.getExecutionEnvironment > val dataStream = streamExecEnv >.addSource(new

Re: Converting a DataStream into a Table throws error

Hi Mich, It seems that the type of your DataStream stream is always wrong. If you want to specify four fields, usually the DataStream type should be similar: DataStream[(Type1, Type2, Type3, Type4)], not DataStream[String], you can try it. Thanks, vino 2018-08-02 6:44 GMT+08:00 Mich Talebzadeh

Re: Converting a DataStream into a Table throws error

Changed as suggested val streamExecEnv = StreamExecutionEnvironment.getExecutionEnvironment val dataStream = streamExecEnv .addSource(new FlinkKafkaConsumer011[String](topicsValue, new SimpleStringSchema(), properties)) val tableEnv =

Re: Converting a DataStream into a Table throws error

Hi, You have to pass the StreamExecutionEnvironment to the getTableEnvironment() method, not the DataStream (or DataStreamSource). Change val tableEnv = TableEnvironment.getTableEnvironment(dataStream) to val tableEnv = TableEnvironment.getTableEnvironment(streamExecEnv) Best, Fabian

Re: Converting a DataStream into a Table throws error

Hi, FYI, these are my imports import java.util.Properties import java.util.Arrays import org.apache.flink.api.common.functions.MapFunction import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment import org.apache.flink.streaming.api.scala import

Re: Converting a DataStream into a Table throws error

Hi I think you are mixing Java and Scala dependencies. org.apache.flink.streaming.api.datastream.DataStream is the DataStream of the Java DataStream API. You should use the DataStream of the Scala DataStream API. Best, Fabian 2018-08-01 14:01 GMT+02:00 Mich Talebzadeh : > Hi, > > I believed I

Re: Converting a DataStream into a Table throws error

Hi, I believed I tried Hequn's suggestion and tried again import org.apache.flink.table.api.Table import org.apache.flink.table.api.TableEnvironment *import org.apache.flink.table.api.scala._* Unfortunately I am still getting the same error! [info] Compiling 1 Scala source to

Re: Converting a DataStream into a Table throws error

If these two imports are the only imports that you added, then you did not follow Hequn's advice or the link that I sent you. You need to add the underscore imports to let Scala do its magic. Timo Am 01.08.18 um 10:28 schrieb Mich Talebzadeh: Hi Timo, These are my two flink table related

Re: Converting a DataStream into a Table throws error

Hi Timo, These are my two flink table related imports import org.apache.flink.table.api.Table import org.apache.flink.table.api.TableEnvironment And these are my dependencies building with SBT libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.2.1" libraryDependencies +=

Re: Converting a DataStream into a Table throws error

Hi Mich, I would check you imports again [1]. This is a pure compiler issue that is unrelated to your actual data stream. Also check your project dependencies. Regards, Timo [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#implicit-conversion-for-scala Am

Re: Converting a DataStream into a Table throws error

Hi both, I added the import as Hequn suggested. My stream is very simple and consists of 4 values separated by "," as below 05521df6-4ccf-4b2f-b874-eb27d461b305,IBM,2018-07-30T19:51:50,190.48 So this is what I have been trying to do Code val dataStream = streamExecEnv

Re: Converting a DataStream into a Table throws error

Hi Mich, The field specified by the fromDataStream API must match the number of fields contained in the DataStream stream object, your DataStream's type is just a string, example is here.[1] [1]:

Re: Converting a DataStream into a Table throws error

Hi, Mich You can try adding "import org.apache.flink.table.api.scala._", so that the Symbol can be recognized as an Expression. Best, Hequn On Wed, Aug 1, 2018 at 6:16 AM, Mich Talebzadeh wrote: > Hi, > > I am following this example > > https://ci.apache.org/projects/flink/flink-docs- >

Converting a DataStream into a Table throws error

Hi, I am following this example https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#integration-with-datastream-and-dataset-api This is my dataStream which is built on a Kafka topic // //Create a Kafka consumer // val dataStream = streamExecEnv