[jira] [Commented] (SPARK-1836) REPL $outer type mismatch causes lookup() and equals() problems
[
https://issues.apache.org/jira/browse/SPARK-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011477#comment-14011477
]
Michael Armbrust commented on SPARK-1836:
-
Yeah I think its likely they are related. We can re-open this one later if
fixing the other one doesn't solve your issue.
REPL $outer type mismatch causes lookup() and equals() problems
---
Key: SPARK-1836
URL: https://issues.apache.org/jira/browse/SPARK-1836
Project: Spark
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Michael Malak
Anand Avati partially traced the cause to REPL wrapping classes in $outer
classes. There are at least two major symptoms:
1. equals()
=
In REPL equals() (required in custom classes used as a key for groupByKey)
seems to have to be written using instanceOf[] instead of the canonical
match{}
Spark Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = false
{noformat}
Spark Shell (equals uses isInstanceOf[]):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s =
s) else false
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
Scala Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
2. lookup()
=
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s ==
s else false
override def hashCode = s.hashCode
override def toString = s
}
val r = sc.parallelize(Array((new C(a),11),(new C(a),12)))
r.lookup(new C(a))
console:17: error: type mismatch;
found : C
required: C
r.lookup(new C(a))
^
{noformat}
See
http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1836) REPL $outer type mismatch causes lookup() and equals() problems
[
https://issues.apache.org/jira/browse/SPARK-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998807#comment-13998807
]
Michael Malak commented on SPARK-1836:
--
Michael Ambrust: Indeed. Do you think I should add my additional case of
equals() (and its workaround) as a comment to SPARK-1199 and mark this one as a
duplicate?
REPL $outer type mismatch causes lookup() and equals() problems
---
Key: SPARK-1836
URL: https://issues.apache.org/jira/browse/SPARK-1836
Project: Spark
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Michael Malak
Anand Avati partially traced the cause to REPL wrapping classes in $outer
classes. There are at least two major symptoms:
1. equals()
=
In REPL equals() (required in custom classes used as a key for groupByKey)
seems to have to be written using instanceOf[] instead of the canonical
match{}
Spark Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = false
{noformat}
Spark Shell (equals uses isInstanceOf[]):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s =
s) else false
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
Scala Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
2. lookup()
=
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s ==
s else false
override def hashCode = s.hashCode
override def toString = s
}
val r = sc.parallelize(Array((new C(a),11),(new C(a),12)))
r.lookup(new C(a))
console:17: error: type mismatch;
found : C
required: C
r.lookup(new C(a))
^
{noformat}
See
http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1836) REPL $outer type mismatch causes lookup() and equals() problems
[
https://issues.apache.org/jira/browse/SPARK-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998312#comment-13998312
]
Michael Armbrust commented on SPARK-1836:
-
This sounds like it could be related to [SPARK-1199]
REPL $outer type mismatch causes lookup() and equals() problems
---
Key: SPARK-1836
URL: https://issues.apache.org/jira/browse/SPARK-1836
Project: Spark
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Michael Malak
Anand Avati partially traced the cause to REPL wrapping classes in $outer
classes. There are at least two major symptoms:
1. equals()
=
In REPL equals() (required in custom classes used as a key for groupByKey)
seems to have to be written using instanceOf[] instead of the canonical
match{}
Spark Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = false
{noformat}
Spark Shell (equals uses isInstanceOf[]):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s =
s) else false
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
Scala Shell (equals uses match{}):
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = o match {
case that: C = that.s == s
case _ = false
}
}
val x = new C(a)
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new
java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)
res: Boolean = true
{noformat}
2. lookup()
=
{noformat}
class C(val s:String) extends Serializable {
override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s ==
s else false
override def hashCode = s.hashCode
override def toString = s
}
val r = sc.parallelize(Array((new C(a),11),(new C(a),12)))
r.lookup(new C(a))
console:17: error: type mismatch;
found : C
required: C
r.lookup(new C(a))
^
{noformat}
See
http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
