srowen commented on a change in pull request #26787: [SPARK-30158][SQL][CORE]
Seq -> Array for sc.parallelize for 2.13 compatibility; remove WrappedArray
URL: https://github.com/apache/spark/pull/26787#discussion_r355652017
##########
File path:
sql/core/src/test/java/test/org/apache/spark/sql/JavaHigherOrderFunctionsSuite.java
##########
@@ -17,33 +17,68 @@
package test.org.apache.spark.sql;
+import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
+import static java.util.stream.Collectors.toList;
import static scala.collection.JavaConverters.mapAsScalaMap;
import org.junit.After;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
-import org.apache.spark.sql.types.*;
-import static org.apache.spark.sql.types.DataTypes.*;
+import org.apache.spark.sql.RowFactory;
import static org.apache.spark.sql.functions.*;
import org.apache.spark.sql.test.TestSparkSession;
-import static test.org.apache.spark.sql.JavaTestUtils.*;
+import org.apache.spark.sql.types.*;
+import static org.apache.spark.sql.types.DataTypes.*;
public class JavaHigherOrderFunctionsSuite {
private transient TestSparkSession spark;
private Dataset<Row> arrDf;
private Dataset<Row> mapDf;
+ private void checkAnswer(Dataset<Row> actualDS, List<Row> expected) throws
Exception {
+ List<Row> actual = actualDS.collectAsList();
+ Assert.assertEquals(expected.size(), actual.size());
+ for (int i = 0; i < expected.size(); i++) {
+ Row expectedRow = expected.get(i);
+ Row actualRow = actual.get(i);
+ Assert.assertEquals(expectedRow.size(), actualRow.size());
+ for (int j = 0; j < expectedRow.size(); j++) {
+ Object expectedValue = expectedRow.get(j);
+ Object actualValue = actualRow.get(j);
+ if (expectedValue != null &&
expectedValue.getClass().isArray()) {
+ actualValue =
actualValue.getClass().getMethod("array").invoke(actualValue);
Review comment:
To explain the punchline here: both WrappedArray (2.12) and ArraySeq (2.13)
have an .array method that returns the underlying array, and we're trying to
compare to the underlying array. It will be an array of reference types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
