On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <[email protected]> wrote:
>
> On another thread I’ll send you code that shows A + 1 works with blank
> rows in A.
>
I don't see how that worked for you. See this:
test("DRM addToRowCardinality - will fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBControl = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBControl).norm should be < 1e-3
}
test("DRM addToRowCardinality - wont fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBWrong = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBWrong).norm should be < 1e-3
}
And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
- DRM addToRowCardinality - will fail *** FAILED ***
2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
- DRM addToRowCardinality - wont fail
BTW this implies rbind will not solve the problem, it is firmly in data
> prep. But until I know the rules I won’t know how to do the right thing.
>
Rbind expects both A and B to have their Int row keys filled from 0 to
nrow-1, which is how they should be ideally.
