Yeah, I got that. What I had in mind was something like a variable that can be used as broadcast var, thus at runtime gets supplied by Flink to the function f.e. a map function.
It would be something like a shortcut. Right now I already could use a broadcast variable, and extract inside the open function the only value it is holding and then supplying it to the apply function. Am I right with that? Mit freundlichen Grüßen, Max! On Thu, Aug 14, 2014 at 5:24 PM, Aljoscha Krettek <[email protected]> wrote: > No, unfortunately that's not possible right now because a DataSet only > represents an Execution that is run when the program is executed. So while > building your program by chaining together operations the actual data is > not yet available. > > I hope that helps but the whole thing can be a bit confusing. So just ask > if you need clarification. > > Cheers, > Aljoscha > > > On Thu, Aug 14, 2014 at 3:01 PM, Maximilian Alber < > [email protected]> wrote: > >> Thanks for the quick reply. >> >> Ok, but is there a way to get the only element out of a DataSet into a >> variable? >> >> Mit freundlichen Grüßen, >> Max! >> >> >> On Thu, Aug 14, 2014 at 11:13 AM, Aljoscha Krettek <[email protected]> >> wrote: >> >>> Hi, >>> for the Java API there are the so-called broadcast variables. Those can >>> be used to set the output of an operation as an additional input of another >>> operator. The feature is not available in the Scala API though? Or am I >>> wrong here? >>> >>> I'm right now working on bringing the Scala API to feature parity with >>> the Java API. >>> >>> Aljoscha >>> >>> >>> On Wed, Aug 13, 2014 at 5:51 PM, Maximilian Alber < >>> [email protected]> wrote: >>> >>>> Hi Flinker, >>>> >>>> I try to implement a quadratic distribution i.e. I would like to choose >>>> an element from a dataset with probability proportional to it's squared >>>> value. >>>> >>>> In Python this would look like this: >>>> >>>> s = numpy.cumsum(residual**2) >>>> x = numpy.random.rand() * s[-1] >>>> return residual[numpy.sum(x > s)] >>>> >>>> With Flink it is somewhat more complicated, I gave it a try: >>>> >>>> import util.Random >>>> >>>> val X = DataSource(XFile, CsvInputFormat[Float]) >>>> val Y = DataSource(YFile, CsvInputFormat[Float]) >>>> >>>> // take square of them >>>> val X_2 = X map { x => (x*x, x) } >>>> // calc sum of squares >>>> val X_sum = X_2 reduce { (x1, x2) => (x1._1 + x2._1, 0) } map { x => >>>> x._1 } >>>> // choose random value in our range >>>> val y = X_sum map { Random.nextFloat * _ } >>>> >>>> // make cummulative sum and find value we search for >>>> val center = X_2 map { >>>> x => (0.0f, x._1, x._2) //sum, x^2, x >>>> } reduce { >>>> (x1, x2) => >>>> if(x1._1 > y){// already found value we searched for >>>> x1 >>>> } else { >>>> if(x1._1 + x2._2 > y){// this is the value we search for >>>> (x1._1 + x2._2, x2._2, x2._3) >>>> } else { >>>> (x1._1 + x2._2, x1._2, x2._3) // just go on with cummulative >>>> sum >>>> } >>>> } >>>> } map { _._3 } // we just need the initial value >>>> >>>> val output = center //map { x => println(x); x } >>>> val sink = output.write("/tmp/test", CsvOutputFormat[Float], "Center >>>> output") >>>> >>>> My problem here is now, I need to get the information stored in y into >>>> the reduce statement to gather the center value. Unfortunately I have no >>>> idea how to achieve that. If somebody knows a way I would be rather >>>> thankful. If someone would know a easier way to solve this problem too! >>>> >>>> Many thanks in advance! >>>> >>>> Cheers Max >>>> >>> >>> >> >
