Github user colorant commented on the pull request:
https://github.com/apache/spark/pull/907#issuecomment-44495966
Hi @witgo This is aim to aligned with the spark pluggable storage roadmap.
say support different storage media across different existing / emerging
hardware. ( for now, say you could put data on SSD for performance, and would
fall back to HDD when SSD is full etc. )
And talked with @pwendell a little bit on this before. He suggested that
they (Databricks) will come up with a API redesign for pluggable storage. At
the same time we can try to add a disk store layer firstly to verify some ideas.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
