Honestly, we have long ago recommended that users use the Typed API which, once it compiles, almost always works as intended.
Here one issue is that the combined pipe is the merge of pipe with different schemas. I honestly don't know what cascading does in this case, but I guess that is part of the problem. I'd recommend using the Typed API if at all possible. The wiki has documentation on this and the Typed unit tests serve as fairly good examples. On Thu, Aug 18, 2016 at 16:54 Xiaolin Li <[email protected]> wrote: > The other weird thing I found is that, self pipe should have 3 fields, but > after the left join, there were only 2 left... > > > On Thursday, August 18, 2016 at 9:52:58 PM UTC-5, Xiaolin Li wrote: >> >> Hi, >> >> I am new to Scalding and I wrote a snippet of code to unify the >> representations of Country Names. (For instance, convert US, USA to US) >> >> Here is the trait function code I wrote: >> >> >> def convertCountryCode(self : Pipe, cntryPipe : Pipe) : Pipe = { >> >> val cntryMap1 = cntryPipe.project('ll, 'lll).rename('lll -> 'temp) >> >> val cntryMap2 = cntryPipe.project('ll, 'nnn).rename('nnn -> 'temp) >> >> val cntryMap3 = cntryPipe.project('ll).map('ll -> 'temp) { >> x : String => x >> } >> >> val combinedCntryMap = cntryMap1 ++ cntryMap2 ++ cntryMap3 >> >> self.joinWithSmaller('oldCntryCode -> 'temp, combinedCntryMap, joiner >> = new LeftJoin) >> } >> >> >> >> Test code >> it should "unify the country representation" in { >> Given { >> List( >> ("place1", "60201", "US"), >> ("place2", "12345", "USA"), >> ("place3", "123456", "GBR"), >> ("place4", "32145", "UGA"), >> ("place5", "31234", "840"), >> ("place6", "23013", "826"), >> ("place7", "11111", "840"), >> ("place8", "12306", "800"), >> ("place9", "77777", "UG"), >> ("place10", "88888", "826") >> >> ) withSchema ('placeName, 'zipCode, 'oldCntryCode) >> } And { >> List( >> ("United Kingdom", "GB", "GBR", "826", "44"), >> ("United States", "US", "USA", "840", "1"), >> ("Uganda", "UG", "UGA", "800", "256") >> ) withSchema ('cntryName, 'll, 'lll, 'nnn, 'dialang) >> } When { >> (pipe : Pipe, cntryPipe : Pipe) => >> pipe.convertCountryCode(cntryPipe) >> } Then { >> buffer : mutable.Buffer[(String, String)] => >> buffer.toSet should equal (Set( >> ("place1", "60201", "US"), >> ("place2", "12345", "US"), >> ("place3", "123456", "GB"), >> ("place4", "32145", "UG"), >> ("place5", "31234", "US"), >> ("place6", "23013", "GB"), >> ("place7", "11111", "US"), >> ("place8", "12306", "UG"), >> ("place9", "77777", "UG"), >> ("place10", "88888", "GB") >> )) >> } >> } >> >> >> However, the test was failed: >> >> *** FAILED *** >> >> [info] Set((place4,32145), (place3,123456), (place6,23013), >> (place2,12345), (place8,12306), (place7,11111), (place1,60201), >> (place5,31234), (place10,88888), (place9,77777)) did not equal >> Set((place1,60201,US), (place7,11111,US), (place4,32145,UG), >> (place5,31234,US), (place3,123456,GB), (place8,12306,UG), >> (place6,23013,GB), (place9,77777,UG), (place2,12345,US), >> (place10,88888,GB)) (CppPipeTransformationsTestSpec.scala:590) >> >> >> I am confused about why this is failed, and I will appreciate if someone >> can give me some guidance. I really appreciate your helps! >> >> >> Thank you! >> >> -Xiaolin >> > -- > You received this message because you are subscribed to the Google Groups > "Scalding Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Scalding Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
