Hi,
I am new to Scalding and I wrote a snippet of code to unify the
representations of Country Names. (For instance, convert US, USA to US)
Here is the trait function code I wrote:
def convertCountryCode(self : Pipe, cntryPipe : Pipe) : Pipe = {
val cntryMap1 = cntryPipe.project('ll, 'lll).rename('lll -> 'temp)
val cntryMap2 = cntryPipe.project('ll, 'nnn).rename('nnn -> 'temp)
val cntryMap3 = cntryPipe.project('ll).map('ll -> 'temp) {
x : String => x
}
val combinedCntryMap = cntryMap1 ++ cntryMap2 ++ cntryMap3
self.joinWithSmaller('oldCntryCode -> 'temp, combinedCntryMap, joiner =
new LeftJoin)
}
Test code
it should "unify the country representation" in {
Given {
List(
("place1", "60201", "US"),
("place2", "12345", "USA"),
("place3", "123456", "GBR"),
("place4", "32145", "UGA"),
("place5", "31234", "840"),
("place6", "23013", "826"),
("place7", "11111", "840"),
("place8", "12306", "800"),
("place9", "77777", "UG"),
("place10", "88888", "826")
) withSchema ('placeName, 'zipCode, 'oldCntryCode)
} And {
List(
("United Kingdom", "GB", "GBR", "826", "44"),
("United States", "US", "USA", "840", "1"),
("Uganda", "UG", "UGA", "800", "256")
) withSchema ('cntryName, 'll, 'lll, 'nnn, 'dialang)
} When {
(pipe : Pipe, cntryPipe : Pipe) => pipe.convertCountryCode(cntryPipe)
} Then {
buffer : mutable.Buffer[(String, String)] =>
buffer.toSet should equal (Set(
("place1", "60201", "US"),
("place2", "12345", "US"),
("place3", "123456", "GB"),
("place4", "32145", "UG"),
("place5", "31234", "US"),
("place6", "23013", "GB"),
("place7", "11111", "US"),
("place8", "12306", "UG"),
("place9", "77777", "UG"),
("place10", "88888", "GB")
))
}
}
However, the test was failed:
*** FAILED ***
[info] Set((place4,32145), (place3,123456), (place6,23013),
(place2,12345), (place8,12306), (place7,11111), (place1,60201),
(place5,31234), (place10,88888), (place9,77777)) did not equal
Set((place1,60201,US), (place7,11111,US), (place4,32145,UG),
(place5,31234,US), (place3,123456,GB), (place8,12306,UG),
(place6,23013,GB), (place9,77777,UG), (place2,12345,US),
(place10,88888,GB)) (CppPipeTransformationsTestSpec.scala:590)
I am confused about why this is failed, and I will appreciate if someone
can give me some guidance. I really appreciate your helps!
Thank you!
-Xiaolin
--
You received this message because you are subscribed to the Google Groups
"Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
