Hi, yes, some things were thrown out while making the APIs equal. What you could do with the delimited input format you can now do with a TextInput, i.e. env.readTextFile() and a mapper. There won't even be an actual map operator since the map is fused (chained) to the data source.
Cheers, Aljoscha On Fri, Sep 26, 2014 at 12:59 PM, Maximilian Alber <[email protected]> wrote: > Hi, > > Thanks. > > Something different regarding file input. > > My former code was: > > val Y = DataSource(config.yFile, DelimitedInputFormat[Vector]{ x: String => > Vector.parseFromString(1, x)}) > > with: > > object Vector { > def parseFromString(s: String) = { > Vector.parseFromString(1, s) > } > > def parseFromString(dimension: Int, s: String) = { > val tokens = s split "\\s" filter { _.trim().size > 0 } > val id = tokens(0).toInt > val values = tokens drop 1 map { _ toFloat } > > assert(values.size == dimension) > > new Vector(id, values) > } > } > > Thus my input is heterogeneous (integer and floats). In the new version the > DataSource class is gone as DelimitedInputFormat is. Is the code just not > added yet or is there another way to do it now? > F.e. I could in imagine to read it first as CSV with Float base type and > afterwards creating the Vector objects with a map function. > > > Cheers, > Max > > > > > On Fri, Sep 26, 2014 at 12:17 PM, Aljoscha Krettek <[email protected]> > wrote: >> >> Hi, >> yes, there are two ways to go. In the past we had PactProgram with the >> getPlan method. This is not necessary anymore in the new version. >> >> With the new version the code is simply put in a main method. Instead >> of creating a Scala Plan you can simply call env.execute() to execute >> the program. >> >> The basic program skeleton is this: >> >> object MyProgram { >> def main(args: Array[String]) { >> val env = ExecutionEnvironment.getExecutionEnvironment >> >> val text = env.readTextFile(...) >> val mapped = text.map {...} >> >> // other operations >> >> mapped.writeAsText() // or writeAsCsv(), or something else >> >> env.execute("My Program") >> } >> >> Cheers, >> Aljoscha >> >> On Fri, Sep 26, 2014 at 12:06 PM, Maximilian Alber >> <[email protected]> wrote: >> > Hi! >> > >> > Thanks for the quick help! >> > Seems I have to change the code a bit for the migration. Or basically >> > substitute new ScalaPlan with env.createProgramPlan :-) >> > >> > Cheers, >> > Max >> > >> > On Fri, Sep 26, 2014 at 11:06 AM, Aljoscha Krettek <[email protected]> >> > wrote: >> >> >> >> Hi, >> >> since the 0.7 is just the latest snapshot release the quickstart >> >> script is not yet updated to create a 0.7 project. >> >> >> >> If you want, you can create a 0.7 quickstart using: >> >> curl >> >> >> >> https://raw.githubusercontent.com/apache/incubator-flink/master/flink-quickstart/quickstart-scala-SNAPSHOT.sh >> >> | bash >> >> >> >> There are complete Scala API examples here: >> >> http://flink.incubator.apache.org/docs/0.7-incubating/examples.html. >> >> Just click the Scala Tab. >> >> >> >> Let us know if you need more information. >> >> >> >> Cheres, >> >> Aljoscha >> >> >> >> On Fri, Sep 26, 2014 at 10:20 AM, Maximilian Alber >> >> <[email protected]> wrote: >> >> > Hi Flinkers, >> >> > >> >> > I tried to migrate to 0.7 for new features aka Broadcast Variables in >> >> > Scala. >> >> > Unfortunately the code structure seemed to have changed: >> >> > >> >> > [INFO] Compiling 2 source files to >> >> > >> >> > >> >> > /media/alber_disk/alber/Arbeit/Uni/Kurse/MA/repo/flink/bump_boost/target/classes >> >> > at 1411719085848 >> >> > [ERROR] >> >> > >> >> > >> >> > /media/alber_disk/alber/Arbeit/Uni/Kurse/MA/repo/flink/bump_boost/src/main/scala/bumpboost/BumpBoost.scala:7: >> >> > error: object TextFile is not a member of package >> >> > org.apache.flink.api.scala >> >> > [ERROR] import org.apache.flink.api.scala.TextFile >> >> > [ERROR] ^ >> >> > [ERROR] >> >> > >> >> > >> >> > /media/alber_disk/alber/Arbeit/Uni/Kurse/MA/repo/flink/bump_boost/src/main/scala/bumpboost/BumpBoost.scala:8: >> >> > error: object ScalaPlan is not a member of package >> >> > org.apache.flink.api.scala >> >> > [ERROR] import org.apache.flink.api.scala.ScalaPlan >> >> > [ERROR] ^ >> >> > [ERROR] >> >> > >> >> > >> >> > /media/alber_disk/alber/Arbeit/Uni/Kurse/MA/repo/flink/bump_boost/src/main/scala/bumpboost/BumpBoost.scala:13: >> >> > error: object functions is not a member of package >> >> > org.apache.flink.api.scala >> >> > [ERROR] import org.apache.flink.api.scala.functions.MapFunction >> >> > >> >> > >> >> > Thus I tried to examine the new quickstart code on >> >> > >> >> > >> >> > https://flink.incubator.apache.org/docs/0.7-incubating/scala_api_quickstart.html >> >> > Unfortunately the quickstart script still creates a project for 0.6 >> >> > and >> >> > using Maven results in an error too: >> >> > >> >> > [INFO] --- maven-archetype-plugin:2.2:generate (default-cli) @ >> >> > standalone-pom --- >> >> > [INFO] Generating project in Interactive mode >> >> > [INFO] Archetype repository missing. Using the one from >> >> > [org.apache.flink:flink-quickstart-scala:0.6.1-incubating] found in >> >> > catalog >> >> > remote >> >> > Downloading: >> >> > >> >> > >> >> > http://repo.maven.apache.org/maven2/org/apache/flink/flink-quickstart-scala/0.7-incubating/flink-quickstart-scala-0.7-incubating.jar >> >> > [INFO] >> >> > >> >> > ------------------------------------------------------------------------ >> >> > [INFO] BUILD FAILURE >> >> > [INFO] >> >> > >> >> > ------------------------------------------------------------------------ >> >> > [INFO] Total time: 18.857s >> >> > [INFO] Finished at: Fri Sep 26 10:17:49 CEST 2014 >> >> > [INFO] Final Memory: 12M/104M >> >> > [INFO] >> >> > >> >> > ------------------------------------------------------------------------ >> >> > [ERROR] Failed to execute goal >> >> > org.apache.maven.plugins:maven-archetype-plugin:2.2:generate >> >> > (default-cli) >> >> > on project standalone-pom: The desired archetype does not exist >> >> > (org.apache.flink:flink-quickstart-scala:0.7-incubating) -> [Help 1] >> >> > >> >> > >> >> > Is there some explanation how to move code from 0.6 to 0.7? Or an >> >> > example >> >> > which creates a Plan? >> >> > >> >> > Thanks! >> >> > Cheers, >> >> > Max >> >> > >> >> > >> > >> > > >
