[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19707994
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala
---
@@ -70,16 +70,29 @@ case class GeneratedAggregate(
val computeFunctions = aggregatesToCompute.map {
case c @ Count(expr) =
+// If we're evaluating UnscaledValue(x), we can do Count on x
directly, since its
+// UnscaledValue will be null if and only if x is null; helps with
Average on decimals
+val toCount = expr match {
+ case UnscaledValue(e) = e
+ case _ = expr
+}
--- End diff --
Unfortunately I'm not sure where to put this into a rule. This is currently
in the execute() method of GenratedAggregate, which only gets created in the
Spark query planning stage (SparkStrategies), so rules from the optimizer won't
apply here. Is there a place to add further rules after planning?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19708002
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala
---
@@ -70,16 +70,29 @@ case class GeneratedAggregate(
val computeFunctions = aggregatesToCompute.map {
case c @ Count(expr) =
+// If we're evaluating UnscaledValue(x), we can do Count on x
directly, since its
+// UnscaledValue will be null if and only if x is null; helps with
Average on decimals
+val toCount = expr match {
+ case UnscaledValue(e) = e
+ case _ = expr
+}
--- End diff --
Oh that a really good point. Maybe we should have rules that also run
after planning? That can wait till another PR though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61386344 [Test build #22705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22705/consoleFull) for PR 2983 at commit [`35e6b02`](https://github.com/apache/spark/commit/35e6b02dafcd389df86f03b7e783e5179a4de3c8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61388496 [Test build #22705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22705/consoleFull) for PR 2983 at commit [`35e6b02`](https://github.com/apache/spark/commit/35e6b02dafcd389df86f03b7e783e5179a4de3c8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61388498 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22705/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61388595 [Test build #504 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/504/consoleFull) for PR 2983 at commit [`35e6b02`](https://github.com/apache/spark/commit/35e6b02dafcd389df86f03b7e783e5179a4de3c8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61390927 [Test build #504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/504/consoleFull) for PR 2983 at commit [`35e6b02`](https://github.com/apache/spark/commit/35e6b02dafcd389df86f03b7e783e5179a4de3c8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2983 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61226735 [Test build #22605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22605/consoleFull) for PR 2983 at commit [`23d3c16`](https://github.com/apache/spark/commit/23d3c160fd9bb88c2ded6e6a699cfede2312b23e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61232397 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22605/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61232391 [Test build #22605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22605/consoleFull) for PR 2983 at commit [`23d3c16`](https://github.com/apache/spark/commit/23d3c160fd9bb88c2ded6e6a699cfede2312b23e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class RegressionMetrics(predictionAndObservations: RDD[(Double, Double)]) extends Logging ` * `class DecimalType(DataType):` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` * `class WriteAheadLogBackedBlockRDDPartition(` * `class WriteAheadLogBackedBlockRDD[T: ClassTag](` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61296754 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/2983#discussion_r19680604 --- Diff: python/pyspark/sql.py --- @@ -148,13 +149,35 @@ class TimestampType(PrimitiveType): -class DecimalType(PrimitiveType): +class DecimalType(DataType): Spark SQL DecimalType The data type representing decimal.Decimal values. +def __init__(self, precision=None, scale=None): +if precision is None: +self.hasPrecisionInfo = False +self.precision = None +self.scale = None +else: +self.hasPrecisionInfo = True +self.precision = precision +self.scale = scale --- End diff -- nit: these lines could be simplified as: ``` self.precision = precision self.scale = scale self.hasPrecisionInfo = precision is not None ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/2983#discussion_r19680835 --- Diff: python/pyspark/sql.py --- @@ -439,9 +462,17 @@ def _parse_datatype_json_string(json_string): return _parse_datatype_json_value(json.loads(json_string)) +_FIXED_DECIMAL = re.compile(decimal\\((\\d+),(\\d+)\\)) + + def _parse_datatype_json_value(json_value): if type(json_value) is unicode and json_value in _all_primitive_types.keys(): return _all_primitive_types[json_value]() +elif type(json_value) is unicode and json_value == u'decimal': +return DecimalType() +elif type(json_value) is unicode and _FIXED_DECIMAL.match(json_value): +m = _FIXED_DECIMAL.match(json_value) +return DecimalType(int(m.group(1)), int(m.group(2))) --- End diff -- Could you change these into two levels and raise exception if json_value is unicode but can not recognize the value? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61297780 [Test build #22615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22615/consoleFull) for PR 2983 at commit [`23d3c16`](https://github.com/apache/spark/commit/23d3c160fd9bb88c2ded6e6a699cfede2312b23e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61305070 Thanks @daveis for the comments; I've now fixed those. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61306142 [Test build #22617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22617/consoleFull) for PR 2983 at commit [`3d84532`](https://github.com/apache/spark/commit/3d84532f46f22745a27d10c3392741803b77bd7b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61306356 [Test build #22617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22617/consoleFull) for PR 2983 at commit [`3d84532`](https://github.com/apache/spark/commit/3d84532f46f22745a27d10c3392741803b77bd7b). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class DecimalType(DataType):` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61306363 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22617/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61307912 [Test build #22620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22620/consoleFull) for PR 2983 at commit [`75ce192`](https://github.com/apache/spark/commit/75ce192054697e0dfd8ce3e3ca0df354c4d707c0). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61311521 [Test build #22615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22615/consoleFull) for PR 2983 at commit [`23d3c16`](https://github.com/apache/spark/commit/23d3c160fd9bb88c2ded6e6a699cfede2312b23e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class DecimalType(DataType):` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61311537 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22615/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61323385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22620/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19697168
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
---
@@ -90,11 +91,17 @@ object DataType {
| LongType ^^^ LongType
| BinaryType ^^^ BinaryType
| BooleanType ^^^ BooleanType
- | DecimalType ^^^ DecimalType
| DateType ^^^ DateType
+ | DecimalType() ^^^ DecimalType.Unlimited
+ | fixedDecimalType
| TimestampType ^^^ TimestampType
)
+protected lazy val fixedDecimalType: Parser[DataType] =
--- End diff --
I think this is technically not required. This parser is only for reading
data from old parquet files that were encoded with old versions of spark sql.
Hopefully we can drop it completely some day.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19697295
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala ---
@@ -869,4 +871,35 @@ class ParquetQuerySuite extends QueryTest with
FunSuiteLike with BeforeAndAfterA
assert(a.dataType === b.dataType)
}
}
+
+ test(read/write fixed-length decimals) {
+for ((precision, scale) - Seq((5, 2), (1, 0), (1, 1), (18, 10), (18,
17))) {
+ val tempDir = getTempFilePath(parquetTest).getCanonicalPath
+ val data = sparkContext.parallelize(0 to 1000)
+.map(i = NumericData(i, i / 100.0))
+.select('i, 'd cast DecimalType(precision, scale))
+ data.saveAsParquetFile(tempDir)
+ checkAnswer(parquetFile(tempDir), data.toSchemaRDD.collect().toSeq)
+}
+
+// Decimals with precision above 18 are not yet supported
--- End diff --
If we don't do this in this PR then we might want to open a JIRA so someone
else does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19697433
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala
---
@@ -286,18 +286,38 @@ case class ApproxCountDistinct(child: Expression,
relativeSD: Double = 0.05)
case class Average(child: Expression) extends PartialAggregate with
trees.UnaryNode[Expression] {
override def nullable = false
- override def dataType = DoubleType
+
+ override def dataType = child.dataType match {
--- End diff --
Not really for this PR, but I wonder if we should be doing this with rules
too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19697736
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
---
@@ -90,11 +91,17 @@ object DataType {
| LongType ^^^ LongType
| BinaryType ^^^ BinaryType
| BooleanType ^^^ BooleanType
- | DecimalType ^^^ DecimalType
| DateType ^^^ DateType
+ | DecimalType() ^^^ DecimalType.Unlimited
+ | fixedDecimalType
| TimestampType ^^^ TimestampType
)
+protected lazy val fixedDecimalType: Parser[DataType] =
--- End diff --
I see, wouldn't it be less confusing to leave it in for now though? I can
also remove it if you prefer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19697753
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala ---
@@ -869,4 +871,35 @@ class ParquetQuerySuite extends QueryTest with
FunSuiteLike with BeforeAndAfterA
assert(a.dataType === b.dataType)
}
}
+
+ test(read/write fixed-length decimals) {
+for ((precision, scale) - Seq((5, 2), (1, 0), (1, 1), (18, 10), (18,
17))) {
+ val tempDir = getTempFilePath(parquetTest).getCanonicalPath
+ val data = sparkContext.parallelize(0 to 1000)
+.map(i = NumericData(i, i / 100.0))
+.select('i, 'd cast DecimalType(precision, scale))
+ data.saveAsParquetFile(tempDir)
+ checkAnswer(parquetFile(tempDir), data.toSchemaRDD.collect().toSeq)
+}
+
+// Decimals with precision above 18 are not yet supported
--- End diff --
Alright, will open one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19698052
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala
---
@@ -70,16 +70,29 @@ case class GeneratedAggregate(
val computeFunctions = aggregatesToCompute.map {
case c @ Count(expr) =
+// If we're evaluating UnscaledValue(x), we can do Count on x
directly, since its
+// UnscaledValue will be null if and only if x is null; helps with
Average on decimals
+val toCount = expr match {
+ case UnscaledValue(e) = e
+ case _ = expr
+}
--- End diff --
This could be a rule right? I only mention this because I thought you
weren't doing this optimization when I was playing around with some query
plans. Of course you actually thought of everything :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19698090
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala
---
@@ -70,16 +70,29 @@ case class GeneratedAggregate(
val computeFunctions = aggregatesToCompute.map {
case c @ Count(expr) =
+// If we're evaluating UnscaledValue(x), we can do Count on x
directly, since its
+// UnscaledValue will be null if and only if x is null; helps with
Average on decimals
+val toCount = expr match {
+ case UnscaledValue(e) = e
+ case _ = expr
+}
--- End diff --
If we change it to a rule though we'll get it in both places in this file
and also in the non-generated case too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19699927
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -151,6 +164,7 @@ trait HiveTypeCoercion {
import HiveTypeCoercion._
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ // TODO: unions with fixed-precision decimals
--- End diff --
Maybe a JIRA for this too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19699984
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
---
@@ -250,27 +252,46 @@ case class Cast(child: Expression, dataType:
DataType) extends UnaryExpression w
buildCast[Date](_, d = dateToLong(d))
case TimestampType =
buildCast[Timestamp](_, t = timestampToLong(t).toByte)
-case DecimalType =
- buildCast[BigDecimal](_, _.toByte)
+case DecimalType() =
+ buildCast[Decimal](_, _.toByte)
case x: NumericType =
b = x.numeric.asInstanceOf[Numeric[Any]].toInt(b).toByte
}
- // DecimalConverter
- private[this] def castToDecimal: Any = Any = child.dataType match {
+ /**
+ * Change the precision / scale in a given decimal to those set in
`decimalType` (if any),
+ * returning null if it overflows or modifying `value` in-place and
returning it if successful.
+ *
+ * NOTE: this modifies `value` in-place, so don't call it on external
data.
+ */
+ private[this] def changePrecision(value: Decimal, decimalType:
DecimalType): Decimal = {
+decimalType match {
+ case DecimalType.Unlimited =
+value
+ case DecimalType.Fixed(precision, scale) =
+if (value.changePrecision(precision, scale)) value else null
+}
+ }
+
+ private[this] def castToDecimal(target: DecimalType): Any = Any =
child.dataType match {
+// TODO: handle casts of null to decimal?
--- End diff --
I believe null is generally handled elsewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2983#discussion_r19700036
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -204,6 +205,8 @@ private[parquet] class RowWriteSupport extends
WriteSupport[Row] with Logging {
case DoubleType = writer.addDouble(value.asInstanceOf[Double])
case FloatType = writer.addFloat(value.asInstanceOf[Float])
case BooleanType = writer.addBoolean(value.asInstanceOf[Boolean])
+case DecimalType.Fixed(precision, scale) if precision = 18 =
--- End diff --
I wonder if its cheaper to avoid pattern matching here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61348621 Hey Matei, this is pretty awesome! A few minor comments and this needs to be merged. Otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61198286 [Test build #22583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22583/consoleFull) for PR 2983 at commit [`ed04858`](https://github.com/apache/spark/commit/ed048587315996551ee223588cb789fa3c7d2ac7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61207042 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22583/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61207038 [Test build #22583 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22583/consoleFull) for PR 2983 at commit [`ed04858`](https://github.com/apache/spark/commit/ed048587315996551ee223588cb789fa3c7d2ac7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60884047 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60884309 [Test build #22443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22443/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60887244 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22443/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60887240 [Test build #22443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22443/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60979400 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60979905 [Test build #22459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22459/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60986161 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22459/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60986155 [Test build #22459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22459/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61008300 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61016830 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61017699 [Test build #22484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22484/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61018102 [Test build #22484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22484/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61018105 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22484/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61020192 Updated this to also read / write decimal precision data from the Hive 0.13 metastore, and fixed a compile error introduced by other recent patches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61020575 [Test build #22486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22486/consoleFull) for PR 2983 at commit [`1a52359`](https://github.com/apache/spark/commit/1a52359c0c1808bd729386bf7b38f37a9fa97f6d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61020961 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22486/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61020958 [Test build #22486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22486/consoleFull) for PR 2983 at commit [`1a52359`](https://github.com/apache/spark/commit/1a52359c0c1808bd729386bf7b38f37a9fa97f6d). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61025985 [Test build #22493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22493/consoleFull) for PR 2983 at commit [`3ec0d53`](https://github.com/apache/spark/commit/3ec0d536f5b8733502fc1655424cc99081232a51). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61026683 [Test build #22493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22493/consoleFull) for PR 2983 at commit [`3ec0d53`](https://github.com/apache/spark/commit/3ec0d536f5b8733502fc1655424cc99081232a51). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class AddWebUIFilter(filterName:String, filterParams: Map[String, String], proxyBase: String)` * ` case class RequestExecutors(requestedTotal: Int) extends CoarseGrainedClusterMessage` * ` case class KillExecutors(executorIds: Seq[String]) extends CoarseGrainedClusterMessage` * `class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val actorSystem: ActorSystem)` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61026689 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22493/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61029320 [Test build #22500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22500/consoleFull) for PR 2983 at commit [`f08fdb3`](https://github.com/apache/spark/commit/f08fdb3a376e1904d7c84b503bbfbe8ec39b27c1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61032345 [Test build #22500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22500/consoleFull) for PR 2983 at commit [`f08fdb3`](https://github.com/apache/spark/commit/f08fdb3a376e1904d7c84b503bbfbe8ec39b27c1). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61032354 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22500/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61039162 [Test build #22511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22511/consoleFull) for PR 2983 at commit [`3360a0e`](https://github.com/apache/spark/commit/3360a0ecb8e383a6d1ae9f023fe343af8418db90). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61039237 [Test build #22511 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22511/consoleFull) for PR 2983 at commit [`3360a0e`](https://github.com/apache/spark/commit/3360a0ecb8e383a6d1ae9f023fe343af8418db90). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class DecimalType(DataType):` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61039238 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22511/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61042108 [Test build #22514 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22514/consoleFull) for PR 2983 at commit [`69dba42`](https://github.com/apache/spark/commit/69dba425dd28877212e359887d8c6c86f527e4b8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61046798 [Test build #22514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22514/consoleFull) for PR 2983 at commit [`69dba42`](https://github.com/apache/spark/commit/69dba425dd28877212e359887d8c6c86f527e4b8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class DecimalType(DataType):` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-61046800 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22514/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60859445 I've marked this as not WIP anymore, because the main TODOs left are in the Hive support. I intend to send that as a separate patch, though I can also add it here. Right now this makes each Hive type be the unlimited-precision decimal, whereas in fact we should respect the precision and scale set in the Hive metastore in Hive 13; but the previous Spark SQL code doesn't respect that either so these are not a regression. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60859753 [Test build #22409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22409/consoleFull) for PR 2983 at commit [`44301f6`](https://github.com/apache/spark/commit/44301f61e2a144c5a7ba3507f6526fd771d10f2f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60861486 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22406/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60861481 [Test build #22406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22406/consoleFull) for PR 2983 at commit [`f136942`](https://github.com/apache/spark/commit/f136942e7c26d4bc6b11604fbac10b7b9a39845c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkContext(config: SparkConf) extends SparkStatusAPI with Logging ` * `abstract class Broadcast[T: ClassTag](val id: Long) extends Serializable with Logging ` * ` class JobUIData(` * `public final class JavaStatusAPIDemo ` * ` public static final class IdentityWithDelayT implements FunctionT, T ` * `class VectorTransformer(object):` * `class Normalizer(VectorTransformer):` * `class JavaModelWrapper(VectorTransformer):` * `class StandardScalerModel(JavaModelWrapper):` * `class StandardScaler(object):` * `class HashingTF(object):` * `class IDFModel(JavaModelWrapper):` * `class IDF(object):` * `class Word2VecModel(JavaModelWrapper):` * `class DateType(PrimitiveType):` * `case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic ` * `case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic ` * `case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic ` * `case class BitwiseNot(child: Expression) extends UnaryExpression ` * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class CreateTableAsSelect[T](` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` * ` logDebug(Found class for $serdeName)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60862371 [Test build #22409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22409/consoleFull) for PR 2983 at commit [`44301f6`](https://github.com/apache/spark/commit/44301f61e2a144c5a7ba3507f6526fd771d10f2f). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60862374 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22409/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60862637 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60862892 [Test build #22414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22414/consoleFull) for PR 2983 at commit [`44301f6`](https://github.com/apache/spark/commit/44301f61e2a144c5a7ba3507f6526fd771d10f2f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60865367 [Test build #22414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22414/consoleFull) for PR 2983 at commit [`44301f6`](https://github.com/apache/spark/commit/44301f61e2a144c5a7ba3507f6526fd771d10f2f). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60865371 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22414/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60866175 [Test build #22418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22418/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60868804 [Test build #22418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22418/consoleFull) for PR 2983 at commit [`4ca62cd`](https://github.com/apache/spark/commit/4ca62cd306d96890a7a56da04710ae5548715c4a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnscaledValue(child: Expression) extends UnaryExpression ` * `case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression ` * `case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)` * `case class PrecisionInfo(precision: Int, scale: Int)` * `case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType ` * `final class Decimal extends Ordered[Decimal] with Serializable ` * ` trait DecimalIsConflicted extends Numeric[Decimal] ` * ` class DeferredObjectAdapter(oi: ObjectInspector) extends DeferredObject ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60868809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22418/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
