Re: [VOTE] Retire Crail from Incubator

+1 -Patrick On Thu, Jun 2, 2022 at 8:58 AM Jonas Pfefferle wrote: > +1 > > On Wed, 1 Jun 2022 12:28:07 -0700 > Julian Hyde wrote: > > Crail has been in incubation since late 2017 [1]. For the last > >couple > > of years, activity has been low [2], and the chances of attracting > >new > >

Re: [DISCUSS] Retire Crail from the Incubator

Hi all, I share the opinions of Jonas and Bernard, happy to move on to an official voting process. Best regards, Patrick On Tue, May 31, 2022 at 1:30 PM bernard metzler wrote: > On 31/05/2022 09:54, Jonas Pfefferle wrote: > > Julian, > > > > Thanks for starting the discussion. As previously

Re: Board reports

I think offboarding the project from Apache Incubator is a reasonable step to consider given the activity in the last year. I would be fine with such a step if others are ok with it. I see Crail being used in benchmarks in research papers every once in a while but I"m not aware of a bigger use

Re: HDFS adapter

Hi Laurent Yes crail://localhost:9060/mypath would be the path you want to use in your Spark program to access Crail. Did you check if your Crail deployment is up and running and can be accessed using the crail hdfs client (./bin/crail fs)? If the crail hdfs client works access via Spark should

Re: Mechanism that allow datanode to leave

Adrian, thanks for providing this info. Looking forward to the implementation. Best regards, Patrick On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach < adrian.schuepb...@gribex.net> wrote: > Dear all > > Crail supports dynamically adding new datanodes, while the Crail cluster > is running. >

Re: [VOTE] Release of Apache Crail-1.2-incubating rc2

+1 * Release builds from source without errors * Basic setup and tools work fine Minor comment, we could give a better error message for the case where JAVA_HOME is not set. Cheers, Patrick On Tue, Dec 3, 2019 at 3:13 PM Adrian Schuepbach < adrian.schuepb...@gribex.net> wrote: > Hi all, > >

Re: Deploying Crail with NVMf storage backend

that more clear. -Patrick On Wed, Sep 25, 2019 at 7:46 PM Patrick Stuedi wrote: > Hi Shashank, > > Short answers to your questions: > > 1) The NVMf tier is a data tier, it stores data just like the RDMA tier or > the TCP tier. This is a bug in the documentation as far as I can se

Re: Local RDMA data path missing error in crail-client

It looks to me like you have localmap enabled (it's actually true by default) which is an optimization where for access to local blocks (served by a local datanode) mmap is used. Somehow it seems the local endpoint can't find the directory where the data is (which shouldn't be). In anycase you can

Re: [zrlio-users] Getting Crail to work over TCP

There is a bug currently in NaRPC which increases the likelyhood of hangs in Crail/TCP as the data sizes increase. We have identified the actual problem in NaRPC but didn't get to fixing it so far. I can look into this. -Patrick On Wed, Aug 21, 2019 at 12:35 AM 'Ben Sidhom' via zrlio-users <

Re: RDMA and Crail Implementation

HI William, You have to differentiate the server side registration from the client side registration. The links above are server side. There we allocate memory in larger segments (defined by the config variable crail.allocationSize). AllocationSize must be a multiples of crail.bufferSize which is

Re: Crail on support GPU memory status?

Hi, [https://issues.apache.org/jira/browse/CRAIL-86] GPU integration with Crail is an important item on our TODO list and has been on the roadmap since a while. There are two (maybe more) ways GPUs could be integrated in Crail: a) as a GPU tier: here, the memories of individual GPUs in the

Re: [VOTE] Release of Apache Crail-1.1-incubating [rc6]

+1 + Compiles from source + Runs from binaries Thanks! -Patrick On Thu, Nov 15, 2018 at 5:16 PM Jonas Pfefferle wrote: > Hi all > > > For another round, we prepared a new release to address the issues found > in > rc5. This is a call to vote on releasing Apache Crail 1.1-incubating, > release

Re: [VOTE] Release of Apache Crail-1.1-incubating [rc3]

+1. Thanks everyone for putting the release together! -Patrick On Sat, Oct 27, 2018 at 12:10 PM Animesh Trivedi wrote: > Thanks Jonas for preparing it > - checked checksum (both binary and source tar.gz) > - checked signature (both binary and source tar.gz) > - compared the release hash

Re: [VOTE] Release of Apache Crail-1.1-incubating [rc2]

checksum and signatures checked. builds from source. +1 Thanks Jonas and Animesh for preparing the release. -Patrick On Tue, Oct 23, 2018 at 11:06 AM Adrian Schuepbach wrote: > Hi all > > My vote is +1. > > I looked at the source and binary tarballs. Looks fine to me. > > Thanks > Adrian >

Re: Crail and data locality question

Hey Sumit, Data locality in Crail simply means that there is a locality API (getLocations(file, offset, length) that can be called by applications to find out where a particular range of data is physically located, i.e. where the corresponding blocks are. Crail has such an API, like HDFS also has

Spark Summit'18: Serverless machine learning using Apache Crail

Hi all, I know this is short notice, but in case you are attending Spark Summit'18 in San Francisco please consider joining my session on serverless machine learning using Spark[1]. This is exciting new work based on Crail. -Patrick [1]

Re: Release Next Steps, was Re: [VOTE] Crail v1.0-rc2 source release

Luciano, Julian, Is the idea that we put the artifacts into a staging directory (for maven central?) prior to starting the IPMC vote? If not required, can we make the call the IPMC vote? Source release is prepared, PPMC vote has passed, so from our side everything is ready. Thanks, Patrick

Re: [VOTE] Crail v1.0-rc2 source release

+1 (non-binding) Thanks Jonas for putting together this release candidate! It looks fine to me. -Patrick On Wed, Apr 25, 2018 at 4:26 PM, Adrian Schuepbach wrote: > +1 > > Thanks for fixing everything commented on rc1. > > It looks like everything is fine to release rc2. >

Re: [VOTE] Crail v1.0-rc0 source release

+1 -Patrick On Thu, Apr 12, 2018 at 2:15 PM, Animesh Trivedi wrote: > Nice job Jonas. > > My vote : +1 > > Thanks, > -- > Animesh > > On Thu, Apr 12, 2018 at 1:33 PM, Jonas Pfefferle wrote: > >> Hi all, >> >> I packaged the source and updated the

Re: C (or language agnostic) API for Crail

Thanks for sharing the doc. This is very much in line with the work we are doing since 6-7 years (https://github.com/zrlio, https://researcher.watson.ibm.com/researcher/view.php?person=zurich-stu). Please let us know if there is a specific aspect you would like to get involved in the context of

Re: C (or language agnostic) API for Crail

Another interesting opportunity would be the development of a gpu storage tier based on gpu direct On Feb 17, 2018 4:00 PM, "Patrick Stuedi" <pstu...@gmail.com> wrote: > That's great, one of the main goals of crail being an apache incubator > project is to ge

Re: C (or language agnostic) API for Crail

che MXNet), and we could encourage a > couple of students contributing code to Crail at this very stage if we see > fit. It could also bring novel system/networking research opportunities to > our lab. > > Let me know how we could better work together. > > Best, > Bairen > &g

Re: C (or language agnostic) API for Crail

Hi Bairen, Your comment is just on spot. The development of a c++ Api for crail is one of the top items on the roadmap, in partical to facilitate the integration into tensorflow and serverless. In fact i started drafting a prototype two weeks ago that i wanted to share soon. If you are interested

Re: A stupid question

Hi Dawn, Not a stupid question at all, a very relevant question. Storage disaggregation is one of THE key use cases we have built Crail for. The goal is to enable existing data processing platforms frameworks to perform efficiently on remote storage (remote as seen from the compute nodes). This