Hi Dawn - we had some results published at the last year's Spark Summit conference : https://www.slideshare.net/databricks/running-apache-spark-on-a-highperformance-cluster-using-rdma-and-nvme-flash-with-patrick-stuedi
see slide 27. We are currently working on testing new setups and workloads. Cheers, -- Animesh On Fri, Jan 26, 2018 at 6:01 AM, Dawn Shepherd <[email protected]> wrote: > Hi Patrick, > > Thank you for you answer. That’s really helpful. > I have another question. Do you have any testing report about the > capability/performance of Crail on a cluster with disaggregated storage? I > didn’t find it on the Crail official site. > > > On 25 Jan 2018, at 23:40, Patrick Stuedi <[email protected]> wrote: > > > > Hi Dawn, > > > > Not a stupid question at all, a very relevant question. Storage > > disaggregation is one of THE key use cases we have built Crail for. > > The goal is to enable existing data processing platforms frameworks to > > perform efficiently on remote storage (remote as seen from the compute > > nodes). This is not straight forward because most frameworks like > > hadoop, spark, etc.., have been designed explicitly to take advantage > > of local storage. In Crail, due to the low-latency high-throughput > > data access, storage disaggregation becomes feasible even for such > > frameworks. > > > > In terms of configuration there is nothing in particular that needs to > > be done, except that Crail permits you to put the storage nodes remote > > to the compute nodes. Both the NVMf and DRAM/RDMA tier are suitable > > for this. With regard to NVMf, there are storage enclosure that pack > > plenty of flash into a single box and export an NVMf interface, these > > boxes can also be used out-of-the box with the Crail NVMf tier. > > > > -Patrick > > > > On Thu, Jan 25, 2018 at 4:05 PM, Dawn Shepherd > > <[email protected]> wrote: > >> How about Crail's ability to support disaggregated storage? I will be > grateful if someone can tell me the detail. Thank you. > >
