[
https://issues.apache.org/jira/browse/IGNITE-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554163#comment-16554163
]
ASF GitHub Bot commented on IGNITE-9063:
----------------------------------------
GitHub user stuartmacd opened a pull request:
https://github.com/apache/ignite/pull/4419
IGNITE-9063 Correct DataFrame Partition Ordering
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/stuartmacd/ignite fixSparkDfSequencing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/4419.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4419
----
commit c9e7294c71de9e7b2bddfae671605a71260b80b3
Author: Stuart Macdonald <29892836+stuartmacd@...>
Date: 2018-07-20T14:50:37Z
Fix out of order partition sequencing
commit 362a748be1151031222f115bf16f4ab9c66b1c99
Author: Stuart Macdonald <29892836+stuartmacd@...>
Date: 2018-07-20T14:50:37Z
Fix out of order partition sequencing
commit 3713bac704b842b22d6c300137f3dca7ccd6b67a
Author: Stuart Macdonald <29892836+stuartmacd@...>
Date: 2018-07-24T11:52:20Z
Merge branch 'fixSparkDfSequencing' of https://github.com/stuartmacd/ignite
into fixSparkDfSequencing
commit b189ce0b0b92b9a7ac0ad99be5565dc59b59f1e9
Author: Stuart Macdonald <29892836+stuartmacd@...>
Date: 2018-07-24T11:58:56Z
Correct server count
----
> Wrong Data Frame parition order
> -------------------------------
>
> Key: IGNITE-9063
> URL: https://issues.apache.org/jira/browse/IGNITE-9063
> Project: Ignite
> Issue Type: Bug
> Components: spark
> Affects Versions: 2.6
> Reporter: Nikolay Izhikov
> Priority: Blocker
> Fix For: 2.7
>
>
> If cluster size is greater that 2 Spark Data Frame tests failed with
> exception.
> {noformat}
> java.lang.IllegalArgumentException: requirement failed:
> partitions(0).partition == 3, but it should equal 0
> at scala.Predef$.require(Predef.scala:224)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:255)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:254)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:254)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1162)
> at
> org.apache.ignite.spark.IgniteSQLDataFrameSpec$$anonfun$1$$anonfun$apply$mcV$sp$11.apply$mcV$sp(IgniteSQLDataFrameSpec.scala:145)
> {noformat}
> https://github.com/stuartmacd/ignite/commit/c9e7294c71de9e7b2bddfae671605a71260b80b3
> - reproducer and fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
