Using methods instead of symbolic ops returns different types so methods work,
ops don’t. If math is math, they should do the smae thing so I’d like to know
what it is supposed to do, can you please allow me to ask specific people the
question?
test("plus one"){
val a = dense(
(1, 1),
(0, 0))
val drmA1 = drmParallelize(m = a, numPartitions = 2)
// modified to return a new CheckpointedDrm so maintains immutability but
still only increases the row cardinality
// by returning new CheckpointedDrmSpark[K](rdd, nrow + n, ncol,
_cacheStorageLevel ) Hack for now.
val drmABigger1 = drmA1.addToRowCardinality(1)
val drmABiggerPlusOne1 = drmABigger1.plus(1.0) // drmABigger has no row 2
in the rdd but an empty row 1
// drmABiggerPlusOne1 is a dense matrix
println(drmABiggerPlusOne1)
val drmA2 = drmParallelize(m = a, numPartitions = 2)
val drmABigger2 = drmA2.addToRowCardinality(1)
val drmABiggerPlusOne2 = drmABigger2 + 1.0
drmABiggerPlusOne2.writeDRM("tmp/plus-one/drma-bigger-plus-one-ops/")
val bp = 0
}
method #1 works, #2 doesn’t. Even when I create a new CheckpintedDrmSpark with
larger _nrow than is in the data—even without the addToRowCardinality.
I agree that in some cases it doesn’t can you please allow me to ask if it
_should_?
On Jul 19, 2014, at 8:56 PM, Anand Avati <[email protected]> wrote:
On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <[email protected]> wrote:
On another thread I’ll send you code that shows A + 1 works with blank rows in
A.
I don't see how that worked for you. See this:
test("DRM addToRowCardinality - will fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBControl = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBControl).norm should be < 1e-3
}
test("DRM addToRowCardinality - wont fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBWrong = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBWrong).norm should be < 1e-3
}
And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
- DRM addToRowCardinality - will fail *** FAILED ***
2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
- DRM addToRowCardinality - wont fail
BTW this implies rbind will not solve the problem, it is firmly in data prep.
But until I know the rules I won’t know how to do the right thing.
Rbind expects both A and B to have their Int row keys filled from 0 to nrow-1,
which is how they should be ideally.
