Hi Nikolay, I reviewed the code and left several comments in the ticket [1]. Please take a look.
[1] https://issues.apache.org/jira/browse/IGNITE-3084 -Val On Mon, Dec 4, 2017 at 3:03 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Denis, > > Nikolay was doing final changes and TC stabilization. I'm planning to do > final review this week, so hopefully we will merge the code soon. > > -Val > > On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda <dma...@apache.org> wrote: > >> Nikolay, Val, >> >> Since we agreed to release the feature without the strategy support, can >> the current integration meet the world in 2.4 release? Please chime in this >> conversation: >> http://apache-ignite-developers.2346864.n4.nabble.com/Time- >> and-scope-for-Apache-Ignite-2-4-td24987.html >> >> — >> Denis >> >> > On Nov 28, 2017, at 5:42 PM, Valentin Kulichenko < >> valentin.kuliche...@gmail.com> wrote: >> > >> > Denis, >> > >> > Agree. I will do the final review in next few days and merge the code. >> > >> > -Val >> > >> > On Tue, Nov 28, 2017 at 5:28 PM, Denis Magda <dma...@apache.org> wrote: >> > >> >> Guys, >> >> >> >> Looking into the parallel discussion about the strategy support I would >> >> change my initial stance and support the idea of releasing the >> integration >> >> in its current state. Is the code ready to be merged into the master? >> Let’s >> >> concentrate on this first and handle the strategy support as a separate >> >> JIRA task. Agree? >> >> >> >> — >> >> Denis >> >> >> >>> On Nov 27, 2017, at 3:47 PM, Valentin Kulichenko < >> >> valentin.kuliche...@gmail.com> wrote: >> >>> >> >>> Nikolay, >> >>> >> >>> Let's estimate the strategy implementation work, and then decide >> weather >> >> to >> >>> merge the code in current state or not. If anything is unclear, please >> >>> start a separate discussion. >> >>> >> >>> -Val >> >>> >> >>> On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков < >> nizhikov....@gmail.com> >> >>> wrote: >> >>> >> >>>> Hello, Val, Denis. >> >>>> >> >>>>> Personally, I think that we should release the integration only >> after >> >>>> the strategy is fully supported. >> >>>> >> >>>> I see two major reason to propose merge of DataFrame API >> implementation >> >>>> without custom strategy: >> >>>> >> >>>> 1. My PR is relatively huge, already. From my experience of >> interaction >> >>>> with Ignite community - the bigger PR becomes, the more time of >> >> commiters >> >>>> required to review PR. >> >>>> So, I propose to move smaller, but complete steps here. >> >>>> >> >>>> 2. It is not clear for me what exactly includes "custom strategy and >> >>>> optimization". >> >>>> Seems, that additional discussion required. >> >>>> I think, I can put my thoughts on the paper and start discussion >> right >> >>>> after basic implementation is done. >> >>>> >> >>>>> Custom strategy implementation is actually very important for this >> >>>> integration. >> >>>> >> >>>> Understand and fully agreed. >> >>>> I'm ready to continue work in that area. >> >>>> >> >>>> 23.11.2017 02:15, Denis Magda пишет: >> >>>> >> >>>> Val, Nikolay, >> >>>>> >> >>>>> Personally, I think that we should release the integration only >> after >> >> the >> >>>>> strategy is fully supported. Without the strategy we don’t really >> >> leverage >> >>>>> from Ignite’s SQL engine and introduce redundant data movement >> between >> >>>>> Ignite and Spark nodes. >> >>>>> >> >>>>> How big is the effort to support the strategy in terms of the >> amount of >> >>>>> work left? 40%, 60%, 80%? >> >>>>> >> >>>>> — >> >>>>> Denis >> >>>>> >> >>>>> On Nov 22, 2017, at 2:57 PM, Valentin Kulichenko < >> >>>>>> valentin.kuliche...@gmail.com> wrote: >> >>>>>> >> >>>>>> Nikolay, >> >>>>>> >> >>>>>> Custom strategy implementation is actually very important for this >> >>>>>> integration. Basically, it will allow to create a SQL query for >> Ignite >> >>>>>> and >> >>>>>> execute it directly on the cluster. Your current implementation >> only >> >>>>>> adds a >> >>>>>> new DataSource which means that Spark will fetch data in its own >> >> memory >> >>>>>> first, and then do most of the work (like joins for example). Does >> it >> >>>>>> make >> >>>>>> sense to you? Can you please take a look at this and provide your >> >>>>>> thoughts >> >>>>>> on how much development is implied there? >> >>>>>> >> >>>>>> Current code looks good to me though and I'm OK if the strategy is >> >>>>>> implemented as a next step in a scope of separate ticket. I will do >> >> final >> >>>>>> review early next week and will merge it if everything is OK. >> >>>>>> >> >>>>>> -Val >> >>>>>> >> >>>>>> On Thu, Oct 19, 2017 at 7:29 AM, Николай Ижиков < >> >> nizhikov....@gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>> Hello. >> >>>>>>> >> >>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two >> >> Catalog >> >>>>>>>> >> >>>>>>> implementations and what is the difference? >> >>>>>>> >> >>>>>>> IgniteCatalog removed. >> >>>>>>> >> >>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have >> to be >> >>>>>>>> >> >>>>>>> set manually on SQLContext each time it's created....Is there any >> >> way to >> >>>>>>> automate this and improve usability? >> >>>>>>> >> >>>>>>> IgniteStrategy and IgniteOptimization are removed as it empty now. >> >>>>>>> >> >>>>>>> Actually, I think it makes sense to create a builder similar to >> >>>>>>>> >> >>>>>>> SparkSession.builder()... >> >>>>>>> >> >>>>>>> IgniteBuilder added. >> >>>>>>> Syntax looks like: >> >>>>>>> >> >>>>>>> ``` >> >>>>>>> val igniteSession = IgniteSparkSession.builder() >> >>>>>>> .appName("Spark Ignite catalog example") >> >>>>>>> .master("local") >> >>>>>>> .config("spark.executor.instances", "2") >> >>>>>>> .igniteConfig(CONFIG) >> >>>>>>> .getOrCreate() >> >>>>>>> >> >>>>>>> igniteSession.catalog.listTables().show() >> >>>>>>> ``` >> >>>>>>> >> >>>>>>> Please, see updated PR - https://github.com/apache/igni >> te/pull/2742 >> >>>>>>> >> >>>>>>> 2017-10-18 20:02 GMT+03:00 Николай Ижиков <nizhikov....@gmail.com >> >: >> >>>>>>> >> >>>>>>> Hello, Valentin. >> >>>>>>>> >> >>>>>>>> My answers is below. >> >>>>>>>> Dmitry, do we need to move discussion to Jira? >> >>>>>>>> >> >>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our >> >> codebase? >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> As I mentioned earlier, to implement and override Spark Catalog >> one >> >>>>>>>> have >> >>>>>>>> to use internal(private) Spark API. >> >>>>>>>> So I have to use package `org.spark.sql.***` to have access to >> >> private >> >>>>>>>> class and variables. >> >>>>>>>> >> >>>>>>>> For example, SharedState class that stores link to >> ExternalCatalog >> >>>>>>>> declared as `private[sql] class SharedState` - i.e. package >> private. >> >>>>>>>> >> >>>>>>>> Can these classes reside under org.apache.ignite.spark instead? >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> No, as long as we want to have our own implementation of >> >>>>>>>> ExternalCatalog. >> >>>>>>>> >> >>>>>>>> 2. IgniteRelationProvider contains multiple constants which I >> guess >> >> are >> >>>>>>>>> >> >>>>>>>> some king of config options. Can you describe the purpose of >> each of >> >>>>>>>> them? >> >>>>>>>> >> >>>>>>>> I extend comments for this options. >> >>>>>>>> Please, see my commit [1] or PR HEAD: >> >>>>>>>> >> >>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two >> >> Catalog >> >>>>>>>>> >> >>>>>>>> implementations and what is the difference? >> >>>>>>>> >> >>>>>>>> Good catch, thank you! >> >>>>>>>> After additional research I founded that only >> IgniteExternalCatalog >> >>>>>>>> required. >> >>>>>>>> I will update PR with IgniteCatalog remove in a few days. >> >>>>>>>> >> >>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op. >> What >> >> are >> >>>>>>>>> >> >>>>>>>> our plans on implementing them? Also, what exactly is planned in >> >>>>>>>> IgniteOptimization and what is its purpose? >> >>>>>>>> >> >>>>>>>> Actually, this is very good question :) >> >>>>>>>> And I need advice from experienced community members here: >> >>>>>>>> >> >>>>>>>> `IgniteOptimization` purpose is to modify query plan created by >> >> Spark. >> >>>>>>>> Currently, we have one optimization described in IGNITE-3084 [2] >> by >> >>>>>>>> you, >> >>>>>>>> Valentin :) : >> >>>>>>>> >> >>>>>>>> “If there are non-Ignite relations in the plan, we should fall >> back >> >> to >> >>>>>>>> native Spark strategies“ >> >>>>>>>> >> >>>>>>>> I think we can go little further and reduce join of two Ignite >> >> backed >> >>>>>>>> Data Frames into single Ignite SQL query. Currently, this >> feature is >> >>>>>>>> unimplemented. >> >>>>>>>> >> >>>>>>>> *Do we need it now? Or we can postpone it and concentrates on >> basic >> >>>>>>>> Data >> >>>>>>>> Frame and Catalog implementation?* >> >>>>>>>> >> >>>>>>>> `Strategy` purpose, as you correctly mentioned in [2], is >> transform >> >>>>>>>> LogicalPlan into physical operators. >> >>>>>>>> I don’t have ideas how to use this opportunity. So I think we >> don’t >> >>>>>>>> need >> >>>>>>>> IgniteStrategy. >> >>>>>>>> >> >>>>>>>> Can you or anyone else suggest some optimization strategy to >> speed >> >> up >> >>>>>>>> SQL >> >>>>>>>> query execution? >> >>>>>>>> >> >>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have >> to >> >> be >> >>>>>>>>> >> >>>>>>>> set manually on SQLContext each time it's created....Is there any >> >> way >> >>>>>>>> to >> >>>>>>>> automate this and improve usability? >> >>>>>>>> >> >>>>>>>> These classes added to `extraOptimizations` when one using >> >>>>>>>> IgniteSparkSession. >> >>>>>>>> As far as I know, there is no way to automatically add these >> >> classes to >> >>>>>>>> regular SparkSession. >> >>>>>>>> >> >>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used in >> >>>>>>>>> >> >>>>>>>> IgniteCatalogExample but not in IgniteDataFrameExample, which is >> >>>>>>>> Confusing. >> >>>>>>>> >> >>>>>>>> DataFrame API is *public* Spark API. So anyone can provide >> >>>>>>>> implementation >> >>>>>>>> and plug it into Spark. That’s why IgniteDataFrameExample doesn’t >> >> need >> >>>>>>>> any >> >>>>>>>> Ignite specific session. >> >>>>>>>> >> >>>>>>>> Catalog API is *internal* Spark API. There is no way to plug >> custom >> >>>>>>>> catalog implementation into Spark [3]. So we have to use >> >>>>>>>> `IgniteSparkSession` that extends regular SparkSession and >> overrides >> >>>>>>>> links >> >>>>>>>> to `ExternalCatalog`. >> >>>>>>>> >> >>>>>>>> 7. To create IgniteSparkSession we first create IgniteContext. >> Is it >> >>>>>>>>> >> >>>>>>>> really needed? It looks like we can directly provide the >> >> configuration >> >>>>>>>> file; if IgniteSparkSession really requires IgniteContext, it can >> >>>>>>>> create it >> >>>>>>>> by itself under the hood. >> >>>>>>>> >> >>>>>>>> Actually, IgniteContext is base class for Ignite <-> Spark >> >> integration >> >>>>>>>> for now. So I tried to reuse it here. I like the idea to remove >> >>>>>>>> explicit >> >>>>>>>> usage of IgniteContext. >> >>>>>>>> Will implement it in a few days. >> >>>>>>>> >> >>>>>>>> Actually, I think it makes sense to create a builder similar to >> >>>>>>>>> >> >>>>>>>> SparkSession.builder()... >> >>>>>>>> >> >>>>>>>> Great idea! I will implement such builder in a few days. >> >>>>>>>> >> >>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for the >> >> case >> >>>>>>>>> >> >>>>>>>> when we don't have SQL configured on Ignite side? >> >>>>>>>> >> >>>>>>>> Yes, IgniteCacheRelation is Data Frame implementation for a >> >> key-value >> >>>>>>>> cache. >> >>>>>>>> >> >>>>>>>> I thought we decided not to support this, no? Or this is >> something >> >>>>>>>>> else? >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> My understanding is following: >> >>>>>>>> >> >>>>>>>> 1. We can’t support automatic resolving key-value caches in >> >>>>>>>> *ExternalCatalog*. Because there is no way to reliably detect key >> >> and >> >>>>>>>> value >> >>>>>>>> classes. >> >>>>>>>> >> >>>>>>>> 2. We can support key-value caches in regular Data Frame >> >>>>>>>> implementation. >> >>>>>>>> Because we can require user to provide key and value classes >> >>>>>>>> explicitly. >> >>>>>>>> >> >>>>>>>> 8. Can you clarify the query syntax in >> >> IgniteDataFrameExample#nativeS >> >>>>>>>>> >> >>>>>>>> parkSqlFromCacheExample2? >> >>>>>>>> >> >>>>>>>> Key-value cache: >> >>>>>>>> >> >>>>>>>> key - java.lang.Long, >> >>>>>>>> value - case class Person(name: String, birthDate: >> java.util.Date) >> >>>>>>>> >> >>>>>>>> Schema of data frame for cache is: >> >>>>>>>> >> >>>>>>>> key - long >> >>>>>>>> value.name - string >> >>>>>>>> value.birthDate - date >> >>>>>>>> >> >>>>>>>> So we can select data from data from cache: >> >>>>>>>> >> >>>>>>>> SELECT >> >>>>>>>> key, `value.name`, `value.birthDate` >> >>>>>>>> FROM >> >>>>>>>> testCache >> >>>>>>>> WHERE key >= 2 AND `value.name` like '%0' >> >>>>>>>> >> >>>>>>>> [1] https://github.com/apache/ignite/pull/2742/commits/faf3ed6fe >> >>>>>>>> bf417bc59b0519156fd4d09114c8da7 >> >>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom >> >>>>>>>> mentId=15794210&page=com.atlassian.jira.plugin.system.issuet >> >>>>>>>> abpanels:comment-tabpanel#comment-15794210 >> >>>>>>>> [3] https://issues.apache.org/jira/browse/SPARK-17767?focusedCom >> >>>>>>>> mentId=15543733&page=com.atlassian.jira.plugin.system.issuet >> >>>>>>>> abpanels:comment-tabpanel#comment-15543733 >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> 18.10.2017 04:39, Dmitriy Setrakyan пишет: >> >>>>>>>> >> >>>>>>>> Val, thanks for the review. Can I ask you to add the same >> comments >> >> to >> >>>>>>>> the >> >>>>>>>> >> >>>>>>>>> ticket? >> >>>>>>>>> >> >>>>>>>>> On Tue, Oct 17, 2017 at 3:20 PM, Valentin Kulichenko < >> >>>>>>>>> valentin.kuliche...@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>> Nikolay, Anton, >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> I did a high level review of the code. First of all, impressive >> >>>>>>>>>> results! >> >>>>>>>>>> However, I have some questions/comments. >> >>>>>>>>>> >> >>>>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our >> >>>>>>>>>> codebase? >> >>>>>>>>>> Can >> >>>>>>>>>> these classes reside under org.apache.ignite.spark instead? >> >>>>>>>>>> 2. IgniteRelationProvider contains multiple constants which I >> >> guess >> >>>>>>>>>> are >> >>>>>>>>>> some king of config options. Can you describe the purpose of >> each >> >> of >> >>>>>>>>>> them? >> >>>>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two >> >>>>>>>>>> Catalog >> >>>>>>>>>> implementations and what is the difference? >> >>>>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op. >> What >> >>>>>>>>>> are >> >>>>>>>>>> our >> >>>>>>>>>> plans on implementing them? Also, what exactly is planned in >> >>>>>>>>>> IgniteOptimization and what is its purpose? >> >>>>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have >> >> to be >> >>>>>>>>>> set >> >>>>>>>>>> manually on SQLContext each time it's created. This seems to be >> >> very >> >>>>>>>>>> error >> >>>>>>>>>> prone. Is there any way to automate this and improve usability? >> >>>>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used >> >>>>>>>>>> in IgniteCatalogExample but not in IgniteDataFrameExample, >> which >> >> is >> >>>>>>>>>> confusing. >> >>>>>>>>>> 7. To create IgniteSparkSession we first create IgniteContext. >> Is >> >> it >> >>>>>>>>>> really >> >>>>>>>>>> needed? It looks like we can directly provide the configuration >> >>>>>>>>>> file; if >> >>>>>>>>>> IgniteSparkSession really requires IgniteContext, it can create >> >> it by >> >>>>>>>>>> itself under the hood. Actually, I think it makes sense to >> create >> >> a >> >>>>>>>>>> builder >> >>>>>>>>>> similar to SparkSession.builder(), it would be good if our APIs >> >> here >> >>>>>>>>>> are >> >>>>>>>>>> consistent with Spark APIs. >> >>>>>>>>>> 8. Can you clarify the query syntax >> >>>>>>>>>> inIgniteDataFrameExample#nativeSparkSqlFromCacheExample2? >> >>>>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for >> the >> >> case >> >>>>>>>>>> when >> >>>>>>>>>> we don't have SQL configured on Ignite side? I thought we >> decided >> >>>>>>>>>> not to >> >>>>>>>>>> support this, no? Or this is something else? >> >>>>>>>>>> >> >>>>>>>>>> Thanks! >> >>>>>>>>>> >> >>>>>>>>>> -Val >> >>>>>>>>>> >> >>>>>>>>>> On Tue, Oct 17, 2017 at 4:40 AM, Anton Vinogradov < >> >>>>>>>>>> avinogra...@gridgain.com> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Sounds awesome. >> >>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> I'll try to review API & tests this week. >> >>>>>>>>>>> >> >>>>>>>>>>> Val, >> >>>>>>>>>>> Your review still required :) >> >>>>>>>>>>> >> >>>>>>>>>>> On Tue, Oct 17, 2017 at 2:36 PM, Николай Ижиков < >> >>>>>>>>>>> nizhikov....@gmail.com> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Yes >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" < >> >>>>>>>>>>>> avinogra...@gridgain.com> написал: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Nikolay, >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> So, it will be able to start regular spark and ignite >> clusters >> >>>>>>>>>>>>> and, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> using >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> peer classloading via spark-context, perform any DataFrame >> >> request, >> >>>>>>>>>>>> >> >>>>>>>>>>>>> correct? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Tue, Oct 17, 2017 at 2:25 PM, Николай Ижиков < >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> nizhikov....@gmail.com> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hello, Anton. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> An example you provide is a path to a master *local* file. >> >>>>>>>>>>>>>> These libraries are added to the classpath for each remote >> >> node >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> running >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> submitted job. >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Please, see documentation: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/ >> >>>>>>>>>>>>>> spark/SparkContext.html#addJar(java.lang.String) >> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/ >> >>>>>>>>>>>>>> spark/SparkContext.html#addFile(java.lang.String) >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> 2017-10-17 13:10 GMT+03:00 Anton Vinogradov < >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> avinogra...@gridgain.com >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> : >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Nikolay, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> With Data Frame API implementation there are no >> requirements >> >> to >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> have >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>> any >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Ignite files on spark worker nodes. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> What do you mean? I see code like: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> spark.sparkContext.addJar(MAVEN_HOME + >> >>>>>>>>>>>>>>> "/org/apache/ignite/ignite-core/2.3.0-SNAPSHOT/ignite- >> >>>>>>>>>>>>>>> core-2.3.0-SNAPSHOT.jar") >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 5:22 PM, Николай Ижиков < >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> nizhikov....@gmail.com> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hello, guys. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> I have created example application to run Ignite Data >> Frame >> >> on >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> standalone >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Spark cluster. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> With Data Frame API implementation there are no >> >> requirements to >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> have >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>> any >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Ignite files on spark worker nodes. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> I ran this application on the free dataset: ATP tennis >> match >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> statistics. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> data - https://github.com/nizhikov/atp_matches >> >>>>>>>>>>>>>>>> app - https://github.com/nizhikov/ig >> nite-spark-df-example >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Valentin, do you have a chance to look at my changes? >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> 2017-10-12 6:03 GMT+03:00 Valentin Kulichenko < >> >>>>>>>>>>>>>>>> valentin.kuliche...@gmail.com >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> : >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Hi Nikolay, >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Sorry for delay on this, got a little swamped lately. I >> >> will >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>> my >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> best >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> review the code this week. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> -Val >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков < >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> nizhikov....@gmail.com> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Hello, Valentin. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Did you have a chance to look at my changes? >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Now I think I have done almost all required features. >> >>>>>>>>>>>>>>>>>> I want to make some performance test to ensure my >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> implementation >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>> work >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> properly with a significant amount of data. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> And I definitely need some feedback for my changes. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> 2017-10-09 18:45 GMT+03:00 Николай Ижиков < >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>> : >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hello, guys. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Which version of Spark do we want to use? >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> 1. Currently, Ignite depends on Spark 2.1.0. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> * Can be run on JDK 7. >> >>>>>>>>>>>>>>>>>>> * Still supported: 2.1.2 will be released soon. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> 2. Latest Spark version is 2.2.0. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> * Can be run only on JDK 8+ >> >>>>>>>>>>>>>>>>>>> * Released Jul 11, 2017. >> >>>>>>>>>>>>>>>>>>> * Already supported by huge vendors(Amazon for >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> example). >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>> Note that in IGNITE-3084 I implement some internal Spark >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> API. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>> So It will take some effort to switch between Spark 2.1 and >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> 2.2 >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko < >> >>>>>>>>>>>>>>>>>>> valentin.kuliche...@gmail.com>: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I will review in the next few days. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> -Val >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda < >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> dma...@apache.org >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Hello Nikolay, >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> This is good news. Finally this capability is >> coming to >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Ignite. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Val, Vladimir, could you do a preliminary review? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Answering on your questions. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> 1. Yardstick should be enough for performance >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> measurements. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> As a >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Spark >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> user, I will be curious to know what’s the point of this >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> integration. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Probably we need to compare Spark + Ignite and Spark + >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Hive >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> or >> >>>>>>>>>>>> >> >>>>>>>>>>>> Spark + >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> RDBMS cases. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> 2. If Spark community is reluctant let’s include the >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> module >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> in >> >>>>>>>>>>>> >> >>>>>>>>>>>> ignite-spark integration. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> — >> >>>>>>>>>>>>>>>>>>>>> Denis >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> On Sep 25, 2017, at 11:14 AM, Николай Ижиков < >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Hello, guys. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Currently, I’m working on integration between Spark >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>> Ignite >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>>> [1]. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> For now, I implement following: >> >>>>>>>>>>>>>>>>>>>>>> * Ignite DataSource implementation( >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> IgniteRelationProvider) >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>> * DataFrame support for Ignite SQL table. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> * IgniteCatalog implementation for a transparent >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> resolving >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>> of >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> ignites >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> SQL tables. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Implementation of it can be found in PR [2] >> >>>>>>>>>>>>>>>>>>>>>> It would be great if someone provides feedback for >> a >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> prototype. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I made some examples in PR so you can see how API >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> suppose >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> to >> >>>>>>>>>>>> >> >>>>>>>>>>>> be >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> used [3]. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> [4]. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> I need some advice. Can you help me? >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 1. How should this PR be tested? >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Of course, I need to provide some unit tests. But >> what >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> about >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> scalability >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> tests, etc. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Maybe we need some Yardstick benchmark or similar? >> >>>>>>>>>>>>>>>>>>>>>> What are your thoughts? >> >>>>>>>>>>>>>>>>>>>>>> Which scenarios should I consider in the first >> place? >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 2. Should we provide Spark Catalog implementation >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> inside >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>> Ignite >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>>>> codebase? >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> A current implementation of Spark Catalog based on >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> *internal >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> Spark >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> API*. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Spark community seems not interested in making >> Catalog >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> API >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> public >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>>> or >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> including Ignite Catalog in Spark code base [5], [6]. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> *Should we include Spark internal API >> implementation >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> inside >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> Ignite >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> code >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> base?* >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Or should we consider to include Catalog >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> implementation >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>> in >> >>>>>>>>>>> >> >>>>>>>>>>> some >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> external >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> module? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> That will be created and released outside >> Ignite?(we >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> still >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> can >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> support >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> develop it inside Ignite community). >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira >> /browse/IGNITE-3084 >> >>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/2742 >> >>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/apache/ >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff- >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> f4ff509cef3018e221394474775e0905 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> [4] https://github.com/apache/ >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff- >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>> f2b670497d81e780dfd5098c5dd8a89c >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> [5] http://apache-spark-developers-list.1001551.n3. >> >>>>>>>>>>>>>>>>>>>>>> nabble.com/Spark-Core-Custom- >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Catalog-Integration-between- >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>> Apache-Ignite-and-Apache-Spark-td22452.html >> >>>>>>>>>>>> >> >>>>>>>>>>>>> [6] https://issues.apache.org/jira/browse/SPARK-17767 >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>>>>>> Nikolay Izhikov >> >>>>>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>>> Nikolay Izhikov >> >>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>> Nikolay Izhikov >> >>>>>>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>> Nikolay Izhikov >> >>>>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>> Nikolay Izhikov >> >>>>>>>>>>>>>> nizhikov....@gmail.com >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Nikolay Izhikov >> >>>>>>> nizhikov....@gmail.com >> >>>>>>> >> >>>>>>> >> >>>>> >> >> >> >> >> >> >
