Kwon, Isn't that JIRA is part of integration with Arrow. As far as arrow as in-memory store goes it probably conflicts with spark's own tungsten memory representation, right?
Thanks Nir On Thu, May 19, 2016 at 8:03 PM, Hyukjin Kwon <gurwls...@gmail.com> wrote: > FYI, there is a JIRA for this, https://issues.apache. > org/jira/browse/SPARK-13534 > > I hope this link is helpful. > > Thanks! > > > 2016-05-20 11:18 GMT+09:00 Sun Rui <sunrise_...@163.com>: > >> 1. I don’t think so >> 2. Arrow is for in-memory columnar execution. While cache is for >> in-memory columnar storage >> >> On May 20, 2016, at 10:16, Todd <bit1...@163.com> wrote: >> >> From the official site http://arrow.apache.org/, Apache Arrow is used >> for Columnar In-Memory storage. I have two quick questions: >> 1. Does spark support Apache Arrow? >> 2. When dataframe is cached in memory, the data are saved in columnar >> in-memory style. What is the relationship between this feature and Apache >> Arrow,that is, >> when the data is in Apache Arrow format，does spark still need the effort >> to cache the dataframe in columnar in-memory? >> >> >> > -- [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] <https://twitter.com/Xactly> [image: Facebook] <https://www.facebook.com/XactlyCorp> [image: YouTube] <http://www.youtube.com/xactlycorporation>
