Great drill down analysis. Thanks, Val! I think, that this kind of integration between Spark and Ignite will be truly useful if we support a logical to physical plan conversion executing a query directly over the data cached in Ignite. Not sure that it worth our efforts to support Ignite just as one more Spark data source if, eventually, Spark still will prepare RDDs internally by copying data at query execution time.
Do you have an understanding on the efforts to support the logical to physical plan conversion? Do we need to ask Sergi or someone else from the community to investigate this deeper? — Denis > On Jan 2, 2017, at 9:39 PM, Valentin Kulichenko > <[email protected]> wrote: > > Folks, > > I made some investigation in this area and put my thoughts in the ticket: > https://issues.apache.org/jira/browse/IGNITE-3084 > > Can someone take a look and provide feedback? Feel free to ask questions if > anything is unclear. > > -Val > > On Tue, Apr 26, 2016 at 5:29 PM, Dmitriy Setrakyan <[email protected]> > wrote: > >> Igniters, >> >> Can someone provide some insight on what would it take to support the >> DataFrame API directly in Spark applications? I am not an expert in this >> part of the product, so it is very difficult for me to judge. >> >> Thanks, >> D. >>
