Nikolay, Java 7 support will be dropped by Ignite soon, so let's do the upgrade now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on TC to use JDK 8. Can you try it out and let me know if it works?
-Val On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov....@gmail.com> wrote: > Valentin, > > > Oh, so this is because of upgrade to 2.2.0? > > Yes, we should upgrade spark module to jdk1.8 because switching to spark > 2.2.0. > > > least we should consider not dropping previous version yet. > > Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1. > > The only thing we can't have is IgniteCatalog. > > Currently, in my PR I include IgniteRDD and Ignite Data Frame in > spark_2.10 module > So we doesn't have to drop spark 2.1 completely. > > https://github.com/apache/ignite/pull/2742 > > 30.11.2017 02:09, Valentin Kulichenko пишет: > >> Nikolay, >> >> Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should >> upgrade in the first place, or at least we should consider not dropping >> previous version yet. I sent a message to the original thread about >> upgrade, let's decide there and then come back to this issue. >> >> -Val >> >> On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov....@gmail.com >> <mailto:nizhikov....@gmail.com>> wrote: >> >> Valentin, >> >> For now `Ignite RDD` build runs on jdk1.7. >> We need to update it to jdk1.8. >> >> I wrote the whole versions numbers to be clear: >> >> 1. Current master - Spark version is 2.1.0. >> So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7. >> >> 2. My branch - >> `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on >> jdk1.7. >> `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has >> to be changed to run on jdk1.8* >> >> >> 29.11.2017 03:27, Valentin Kulichenko пишет: >> >> Nikolay, >> >> If Spark requires Java 8, then I guess we have no choice. How TC >> is configured at the moment? My understanding is that Spark related suites >> are successfully executed there, so is there an issue? >> >> -Val >> >> On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков < >> nizhikov....@gmail.com <mailto:nizhikov....@gmail.com> <mailto: >> nizhikov....@gmail.com <mailto:nizhikov....@gmail.com>>> wrote: >> >> Hello, Valentin. >> >> Added '-Dscala-2.10' to the build config. Let me know if >> it helps. >> >> >> Yes, it helps. Thank you! >> Now, 'Ignite RDD spark 2_10' succeed for my branch. >> >> >> Do you mean that IgniteRDD does not compile on JDK7? If >> yes, do we know the reason? I don't think switching it to JDK8 is a >> solution as it should work with both. >> >> >> I mean that latest version of spark doesn't support jdk7. >> >> http://spark.apache.org/docs/latest/ < >> http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/ >> latest/ <http://spark.apache.org/docs/latest/>> >> >> "Spark runs on Java 8+..." >> "For the Scala API, Spark 2.2.0 uses Scala 2.11..." >> "Note that support for Java 7... were removed as of Spark >> 2.2.0" >> "Note that support for Scala 2.10 is deprecated..." >> >> Moreover, We can't have IgniteCatalog for spark 2.1. >> Please, see my explanation in jira ticket - >> >> https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom >> mentId=16268523&page=com.atlassian.jira.plugin.system. >> issuetabpanels:comment-tabpanel#comment-16268523 >> <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo >> mmentId=16268523&page=com.atlassian.jira.plugin.system. >> issuetabpanels:comment-tabpanel#comment-16268523> >> <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo >> mmentId=16268523&page=com.atlassian.jira.plugin.system. >> issuetabpanels:comment-tabpanel#comment-16268523 >> <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo >> mmentId=16268523&page=com.atlassian.jira.plugin.system. >> issuetabpanels:comment-tabpanel#comment-16268523>> >> >> Do you see any options to support jdk7 for spark module? >> >> > I think all tests should be executed on TC. Can you check >> if they work and add them to corresponding suites >> >> OK, I file a ticket and try to fix it shortly. >> >> https://issues.apache.org/jira/browse/IGNITE-7042 < >> https://issues.apache.org/jira/browse/IGNITE-7042> < >> https://issues.apache.org/jira/browse/IGNITE-7042 >> <https://issues.apache.org/jira/browse/IGNITE-7042>> >> >> 28.11.2017 03:33, Valentin Kulichenko пишет: >> >> Hi Nikolay, >> >> Please see my responses inline. >> >> -Val >> >> On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков < >> nizhikov....@gmail.com <mailto:nizhikov....@gmail.com> <mailto: >> nizhikov....@gmail.com <mailto:nizhikov....@gmail.com>> >> <mailto:nizhikov....@gmail.com <mailto:nizhikov....@gmail.com> >> <mailto:nizhikov....@gmail.com <mailto:nizhikov....@gmail.com>>>> wrote: >> >> Hello, guys. >> >> I have some issues on TC with my PR [1] for >> IGNITE-3084(Spark Data Frame API). >> Can you, please, help me: >> >> >> 1. `Ignite RDD spark 2_10` - >> >> Currently this build runs with following profiles: >> `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2] >> That means `scala` profile is activated too for >> `Ignite RDD spark 2_10` >> Because `scala` activation is done like [3]: >> >> ``` >> <activation> >> <property><name>!scala-2.10</n >> ame></property> >> </activation> >> ``` >> >> I think it a misconfiguration because scala(2.11) >> shouldn't be activated for 2.10 build. >> Am I miss something? >> >> Can someone edit build property? >> * Add `-scala` to profiles list >> * Or add `-Dscala-2.10` to jvm properties >> to turn off `scala` profile in this build. >> >> >> Added '-Dscala-2.10' to the build config. Let me know if >> it helps. >> >> >> 2. `Ignite RDD` - >> >> Currently this build run on jvm7 [4]. >> As I wrote in my previous mail [5] current version >> of spark(2.2) runs only on jvm8. >> >> Can someone edit build property to run it on jvm8? >> >> >> Do you mean that IgniteRDD does not compile on JDK7? If >> yes, do we know the reason? I don't think switching it to JDK8 is a >> solution as it should work with both. >> >> >> 3. For now `Ignite RDD` and `Ignite RDD spark 2_10` >> only runs java tests [6] existing in `spark` module. >> There are several existing tests written in >> scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example. >> Is it turned off by a purpose or I miss something? >> Should we run scala-test for spark and spark_2.10 >> modules? >> >> I think all tests should be executed on TC. Can you >> check if they work and add them to corresponding suites? >> >> >> [1] https://github.com/apache/ignite/pull/2742 < >> https://github.com/apache/ignite/pull/2742> < >> https://github.com/apache/ignite/pull/2742 >> <https://github.com/apache/ignite/pull/2742>> < >> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign >> ite/pull/2742> <https://github.com/apache/ignite/pull/2742 >> <https://github.com/apache/ignite/pull/2742>>> >> [2] https://ci.ignite.apache.org/v >> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR >> ddSpark210&tab=buildLog&_focus=379#_state=371 >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_ >> focus=379#_state=371> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=buildLog&_focus=379#_state=371 >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_ >> focus=379#_state=371>> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=buildLog&_focus=379#_state=371 >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_ >> focus=379#_state=371> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=buildLog&_focus=379#_state=371 >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_ >> focus=379#_state=371>>> >> [3] https://github.com/apache/igni >> te/blob/master/pom.xml#L533 <https://github.com/apache/ign >> ite/blob/master/pom.xml#L533> >> <https://github.com/apache/ignite/blob/master/pom.xml#L533 < >> https://github.com/apache/ignite/blob/master/pom.xml#L533>> < >> https://github.com/apache/ignite/blob/master/pom.xml#L533 >> >> <https://github.com/apache/ignite/blob/master/pom.xml#L533> >> <https://github.com/apache/ign >> ite/blob/master/pom.xml#L533 <https://github.com/apache/ign >> ite/blob/master/pom.xml#L533>>> >> [4] https://ci.ignite.apache.org/v >> iewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteR >> dd&tab=buildParameters >> <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu >> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite >> Rdd&tab=buildParameters >> <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu >> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite >> Rdd&tab=buildParameters >> <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu >> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite >> Rdd&tab=buildParameters >> <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu >> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>> >> [5] http://apache-ignite-developer >> s.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit >> e-Prototype-tp22649p23099.html >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html> >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>> >> <http://apache-ignite-develope >> rs.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit >> e-Prototype-tp22649p23099.html >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html> >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html >> <http://apache-ignite-developers.2346864.n4.nabble.com/ >> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>> >> [6] https://ci.ignite.apache.org/v >> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR >> ddSpark210&tab=testsInfo >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=testsInfo >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=testsInfo >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo> >> <https://ci.ignite.apache.org/ >> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite >> RddSpark210&tab=testsInfo >> <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu >> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>> >> [7] https://github.com/apache/igni >> te/blob/master/modules/spark/src/test/scala/org/apache/ >> ignite/spark/IgniteRDDSpec.scala >> <https://github.com/apache/ignite/blob/master/modules/spark/ >> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala> >> <https://github.com/apache/ign >> ite/blob/master/modules/spark/src/test/scala/org/apache/ >> ignite/spark/IgniteRDDSpec.scala >> <https://github.com/apache/ignite/blob/master/modules/spark/ >> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>> >> <https://github.com/apache/ign >> ite/blob/master/modules/spark/src/test/scala/org/apache/ >> ignite/spark/IgniteRDDSpec.scala >> <https://github.com/apache/ignite/blob/master/modules/spark/ >> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala> >> <https://github.com/apache/ign >> ite/blob/master/modules/spark/src/test/scala/org/apache/ >> ignite/spark/IgniteRDDSpec.scala >> <https://github.com/apache/ignite/blob/master/modules/spark/ >> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>> >> >> >> >> >>
