The other weird thing I found is that, self pipe should have 3 fields, but
after the left join, there were only 2 left...
On Thursday, August 18, 2016 at 9:52:58 PM UTC-5, Xiaolin Li wrote:
>
> Hi,
>
> I am new to Scalding and I wrote a snippet of code to unify the
> representations of Country Names. (For instance, convert US, USA to US)
>
> Here is the trait function code I wrote:
>
>
> def convertCountryCode(self : Pipe, cntryPipe : Pipe) : Pipe = {
>
> val cntryMap1 = cntryPipe.project('ll, 'lll).rename('lll -> 'temp)
>
> val cntryMap2 = cntryPipe.project('ll, 'nnn).rename('nnn -> 'temp)
>
> val cntryMap3 = cntryPipe.project('ll).map('ll -> 'temp) {
> x : String => x
> }
>
> val combinedCntryMap = cntryMap1 ++ cntryMap2 ++ cntryMap3
>
> self.joinWithSmaller('oldCntryCode -> 'temp, combinedCntryMap, joiner
> = new LeftJoin)
> }
>
>
>
> Test code
> it should "unify the country representation" in {
> Given {
> List(
> ("place1", "60201", "US"),
> ("place2", "12345", "USA"),
> ("place3", "123456", "GBR"),
> ("place4", "32145", "UGA"),
> ("place5", "31234", "840"),
> ("place6", "23013", "826"),
> ("place7", "11111", "840"),
> ("place8", "12306", "800"),
> ("place9", "77777", "UG"),
> ("place10", "88888", "826")
>
> ) withSchema ('placeName, 'zipCode, 'oldCntryCode)
> } And {
> List(
> ("United Kingdom", "GB", "GBR", "826", "44"),
> ("United States", "US", "USA", "840", "1"),
> ("Uganda", "UG", "UGA", "800", "256")
> ) withSchema ('cntryName, 'll, 'lll, 'nnn, 'dialang)
> } When {
> (pipe : Pipe, cntryPipe : Pipe) => pipe.convertCountryCode(cntryPipe)
> } Then {
> buffer : mutable.Buffer[(String, String)] =>
> buffer.toSet should equal (Set(
> ("place1", "60201", "US"),
> ("place2", "12345", "US"),
> ("place3", "123456", "GB"),
> ("place4", "32145", "UG"),
> ("place5", "31234", "US"),
> ("place6", "23013", "GB"),
> ("place7", "11111", "US"),
> ("place8", "12306", "UG"),
> ("place9", "77777", "UG"),
> ("place10", "88888", "GB")
> ))
> }
> }
>
>
> However, the test was failed:
>
> *** FAILED ***
>
> [info] Set((place4,32145), (place3,123456), (place6,23013),
> (place2,12345), (place8,12306), (place7,11111), (place1,60201),
> (place5,31234), (place10,88888), (place9,77777)) did not equal
> Set((place1,60201,US), (place7,11111,US), (place4,32145,UG),
> (place5,31234,US), (place3,123456,GB), (place8,12306,UG),
> (place6,23013,GB), (place9,77777,UG), (place2,12345,US),
> (place10,88888,GB)) (CppPipeTransformationsTestSpec.scala:590)
>
>
> I am confused about why this is failed, and I will appreciate if someone
> can give me some guidance. I really appreciate your helps!
>
>
> Thank you!
>
> -Xiaolin
>
--
You received this message because you are subscribed to the Google Groups
"Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
