[jira] [Comment Edited] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt
[
https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606633#comment-16606633
]
Hyukjin Kwon edited comment on SPARK-25036 at 9/7/18 2:15 AM:
--
There are already too many warnings and I assume it's difficult to spot
warnings specific to Scala 2.12 given my resent look
(https://github.com/apache/spark/pull/21975). Am I correct? I would rather
leave this resolved, and fix them in a batch later in a separate ticket.
was (Author: hyukjin.kwon):
There are already too many warnings and I assume it's difficult to spot
warnings specific to Scala 2.12 given my resent look
(https://github.com/apache/spark/pull/21975). Am I correct? I would rather
leave this resolved, and fix them in a batch.
> Scala 2.12 issues: Compilation error with sbt
> -
>
> Key: SPARK-25036
> URL: https://issues.apache.org/jira/browse/SPARK-25036
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Kazuaki Ishizaki
>Assignee: Kazuaki Ishizaki
>Priority: Major
>
> When compiling with sbt, the following errors occur:
> There are -two- three types:
> 1. {{ExprValue.isNull}} is compared with unexpected type.
> 2. {{match may not be exhaustive}} is detected at {{match}}
> 3. discarding unmoored doc comment
> The first one is more serious since it may also generate incorrect code in
> Spark 2.3.
> {code:java}
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:63:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (NumericValueInterval(_, _),
> _), (_, NumericValueInterval(_, _)), (_, _)
> [error] [warn] def isIntersected(r1: ValueInterval, r2: ValueInterval):
> Boolean = (r1, r2) match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:79:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (NumericValueInterval(_, _),
> _), (_, NumericValueInterval(_, _)), (_, _)
> [error] [warn] (r1, r2) match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala:67:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (ArrayType(_, _), _), (_,
> ArrayData()), (_, _)
> [error] [warn] (endpointsExpression.dataType, endpointsExpression.eval())
> match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:470:
> match may not be exhaustive.
> [error] It would fail on the following inputs: NewFunctionSpec(_, None,
> Some(_)), NewFunctionSpec(_, Some(_), None)
> [error] [warn] newFunction match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:94:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely always compare unequal
> [error] [warn] if (eval.isNull != "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:126:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely never compare equal
> [error] [warn] if (eval.isNull == "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:133:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely never compare equal
> [error] [warn] if (eval.isNull == "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:709:
> match may not be exhaustive.
> [error] It would fail on the following input: Schema((x:
> org.apache.spark.sql.types.DataType forSome x not in
> org.apache.spark.sql.types.StructType), _)
> [error] [warn] def attributesFor[T: TypeTag]: Seq[Attribute] = schemaFor[T]
> match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala:90:
>
[jira] [Comment Edited] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt
[
https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575137#comment-16575137
]
Kazuaki Ishizaki edited comment on SPARK-25036 at 8/9/18 5:05 PM:
--
Another type of compilation error is found. Added the log to the description
was (Author: kiszk):
Another type of compilation error is found
> Scala 2.12 issues: Compilation error with sbt
> -
>
> Key: SPARK-25036
> URL: https://issues.apache.org/jira/browse/SPARK-25036
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Kazuaki Ishizaki
>Assignee: Kazuaki Ishizaki
>Priority: Major
> Fix For: 2.4.0
>
>
> When compiling with sbt, the following errors occur:
> There are two types:
> 1. {{ExprValue.isNull}} is compared with unexpected type.
> 1. {{match may not be exhaustive}} is detected at {{match}}
> The first one is more serious since it may also generate incorrect code in
> Spark 2.3.
> {code}
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:63:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (NumericValueInterval(_, _),
> _), (_, NumericValueInterval(_, _)), (_, _)
> [error] [warn] def isIntersected(r1: ValueInterval, r2: ValueInterval):
> Boolean = (r1, r2) match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:79:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (NumericValueInterval(_, _),
> _), (_, NumericValueInterval(_, _)), (_, _)
> [error] [warn] (r1, r2) match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala:67:
> match may not be exhaustive.
> [error] It would fail on the following inputs: (ArrayType(_, _), _), (_,
> ArrayData()), (_, _)
> [error] [warn] (endpointsExpression.dataType, endpointsExpression.eval())
> match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:470:
> match may not be exhaustive.
> [error] It would fail on the following inputs: NewFunctionSpec(_, None,
> Some(_)), NewFunctionSpec(_, Some(_), None)
> [error] [warn] newFunction match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:94:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely always compare unequal
> [error] [warn] if (eval.isNull != "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:126:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely never compare equal
> [error] [warn] if (eval.isNull == "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:133:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely never compare equal
> [error] [warn] if (eval.isNull == "true") {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:709:
> match may not be exhaustive.
> [error] It would fail on the following input: Schema((x:
> org.apache.spark.sql.types.DataType forSome x not in
> org.apache.spark.sql.types.StructType), _)
> [error] [warn] def attributesFor[T: TypeTag]: Seq[Attribute] = schemaFor[T]
> match {
> [error] [warn]
> [error] [warn]
> /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala:90:
> org.apache.spark.sql.catalyst.expressions.codegen.ExprValue and String are
> unrelated: they will most likely never compare equal
> [error] [warn] if (inputs.map(_.isNull).forall(_ == "false")) {
> [error] [warn]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional
