[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571890#comment-15571890 ] Pete Robbins commented on SPARK-17827: -- I have a PR ready which I will submit as soon as I have run the tests on both Big and Little Endian > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571901#comment-15571901 ] Apache Spark commented on SPARK-17827: -- User 'robbinspg' has created a pull request for this issue: https://github.com/apache/spark/pull/15464 > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571881#comment-15571881 ] Zhenhua Wang commented on SPARK-17827: -- [~srowen] Thanks for notification. [~robbinspg] It's strange I didn't receive your email, sorry for this late response. I think we hit a bug here, the maxColLen should be Int type because the return type for the `Length` aggregate function is Int. But I don't have a big endian platform to verify this, is it ok for me to create a pr and let you to rerun the test suite using my branch? > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571546#comment-15571546 ] Pete Robbins commented on SPARK-17827: -- right so in these two cases maxLength in AnalyzeColumnCommand is returning an Int type and I guess in other cases it could be Long?? > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571448#comment-15571448 ] Sean Owen commented on SPARK-17827: --- [~ZenWzh] I think you added this? what do you think? I suspect that these can be integers if they're the max length of a string or byte array, which is an int anyway. > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348
]
Pete Robbins commented on SPARK-17827:
--
In Statistics,scala
case class StringColumnStat(statRow: InternalRow) {
println("StringColumnStat: " + statRow)
// The indices here must be consistent with
`ColumnStatStruct.stringColumnStat`.
val numNulls: Long = statRow.getLong(0)
val avgColLen: Double = statRow.getDouble(1)
val maxColLen: Long = statRow.getLong(2) << Actual type in
statRow is Int
val ndv: Long = statRow.getLong(3)
}
case class BinaryColumnStat(statRow: InternalRow) {
// The indices here must be consistent with
`ColumnStatStruct.binaryColumnStat`.
val numNulls: Long = statRow.getLong(0)
val avgColLen: Double = statRow.getDouble(1)
val maxColLen: Long = statRow.getLong(2)<< Actual type in
statRow is Int
}
So either the code above should be using getInt for the maxColLen or the code
generating the row should be creating a Long
> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>Reporter: Pete Robbins
> Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
> org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for
> string column
> org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for
> binary column
> org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for
> columns with different types
> org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics
> and load them from hive metastore
> all fail in checkColStat eg:
> java.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:156)
> at
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
> at
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
> at
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
> at
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
> at
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
> at
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
> at
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
> at
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
> at
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
> at
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569048#comment-15569048 ] Pete Robbins commented on SPARK-17827: -- So this looks like the max field is being written as an Int into the UnsafeRow but is later read as a Long. Code stack to the write: java.lang.Thread.dumpStack(Thread.java:462) at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:149) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:232) at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:221) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:392) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:79) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at org.apache.spark.sql.execution.aggregate.AggregationIterator.foreach(AggregationIterator.scala:35) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at org.apache.spark.sql.execution.aggregate.AggregationIterator.to(AggregationIterator.scala:35) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at org.apache.spark.sql.execution.aggregate.AggregationIterator.toBuffer(AggregationIterator.scala:35) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at org.apache.spark.sql.execution.aggregate.AggregationIterator.toArray(AggregationIterator.scala:35) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785) > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at >
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561849#comment-15561849 ] Pete Robbins commented on SPARK-17827: -- [~ZenWzh] Any ideas what code introduced that could cause endian issues? This is usually something like writing a field as one type but reading it as another eg putLong but then readInt. > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms
[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554825#comment-15554825 ] Pete Robbins commented on SPARK-17827: -- I'm investigating this > StatisticsColumnSuite failures on big endian platforms > -- > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 > Environment: big endian >Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
