Re: preparing for a first release

[Julian Hyde]

Reviewing the repo at a particular commit is a good first step.

The next step will be to create a release candidate (a tar ball and checksum 
files and signatures), add them to dist.apache.org for review, and send an 
email to start a vote.

Here is an example: 
https://lists.apache.org/thread.html/03b49fbed8617e860f71bc4f80abe411451d5f112beb5837cb9e5367@%3Cdev.calcite.apache.org%3E
 

 

I’d be careful with tags. It’s not good to change/delete tags applied to the 
master branch. So I wouldn’t apply the ‘v1.0’ tag until the release vote has 
passed. You could use ‘v1.0-rc1’ etc. for release candidates.

I also strongly suggest that you compile a “howto” - a list of instructions 
that you followed to make the release, that can be followed by the next release 
manager next release - and commit it after the release. It is very important 
that the release process is consistent and reproducible.

Julian



> On Apr 10, 2018, at 7:18 AM, Animesh Trivedi  
> wrote:
> 
> Dear all,
> 
> We are ready for our first source code release. We have tagged the current
> candidate with v1.0 (https://github.com/apache/incubator-crail/tree/v1.0).
> 
> What is the next step? Start a vote for "source-only" release where people
> can review the currently tagged branch?
> 
> Thanks,
> --
> Animesh
> 
> 
> On Tue, Mar 13, 2018 at 9:57 PM, Julian Hyde  wrote:
> 
>> Bernard,
>> 
>> You have it about right.
>> 
>> One thing to remember is that a release is source-code only. (You may
>> do binary releases later, but they are much more complicated.)
>> 
>> Another is that releases don't need to be perfect in terms of
>> functionality. It's fine if there are bugs, as long as the code
>> compiles. The thing that trips up most podlings is getting the legal
>> aspects right (e.g. no GPL-licensed dependencies, including necessary
>> text in LICENSE and NOTICE files).
>> 
>> I suggest you make a first release candidate, start a vote, and get
>> feedback. The first couple of votes will likely fail. It will take a
>> couple of iterations, but we don't expect perfection first time.
>> 
>> When the PPMC vote passes, there is a second vote on the IPMC. (Due
>> Apache policy that only a "real" PMC can make a release. The IPMC is
>> real PMC, but the PPMC is only a PMC-in-training.) That second step is
>> arduous I'm afraid (and adds a few extra days) but it's not too bad by
>> the 2nd or 3rd release.
>> 
>> Julian
>> 
>> 
>> 
>> On Tue, Mar 13, 2018 at 11:35 AM, Luciano Resende 
>> wrote:
>>> I would say the most important thing would be to get the legal parts in a
>>> good state.
>>> 
>>> For guide, I would start with :
>>> - http://www.apache.org/dev/#releases
>>> 
>>> 
>>> For actual steps, here is what I have used in other projects (which
>> include
>>> building, staging, signing, etc):
>>> 
>>> https://github.com/apache/bahir/blob/master/dev/release-build.sh
>>> 
>>> 
>>> 
>>> On Tue, Mar 13, 2018 at 11:21 AM, bernard metzler 
>> wrote:
>>> 
 Dear Mentors,
 
 Looks like we are getting close to a point where we think the code
 base is good for a first release. I just searched around a little
 to see what is needed from an administrative point of view. What
 I found is an overwhelming amount of information, starting
 at https://incubator.apache.org/guides/releasemanagement.html
 
 So, what I understand is, we have to follow some formal policies
 applicable to all ASF project releases
 (http://www.apache.org/legal/release-policy.html) plus setting up
 the code base in a staged area (which would be
 https://dist.apache.org/repos/dist/dev/incubator/crail/ - which
 does not yet exist?), and finally go through a PMC voting procedure,
 as described at
 https://incubator.apache.org/guides/releasemanagement.html
 
 
 Is there anything else I am missing?
 
 Thank you!
 Bernard.
 
>>> 
>>> 
>>> 
>>> --
>>> Luciano Resende
>>> http://twitter.com/lresende1975
>>> http://lresende.blogspot.com/
>> 

Re: preparing for a first release

[Animesh Trivedi]

Dear all,

We are ready for our first source code release. We have tagged the current
candidate with v1.0 (https://github.com/apache/incubator-crail/tree/v1.0).

What is the next step? Start a vote for "source-only" release where people
can review the currently tagged branch?

Thanks,
--
Animesh


On Tue, Mar 13, 2018 at 9:57 PM, Julian Hyde  wrote:

> Bernard,
>
> You have it about right.
>
> One thing to remember is that a release is source-code only. (You may
> do binary releases later, but they are much more complicated.)
>
> Another is that releases don't need to be perfect in terms of
> functionality. It's fine if there are bugs, as long as the code
> compiles. The thing that trips up most podlings is getting the legal
> aspects right (e.g. no GPL-licensed dependencies, including necessary
> text in LICENSE and NOTICE files).
>
> I suggest you make a first release candidate, start a vote, and get
> feedback. The first couple of votes will likely fail. It will take a
> couple of iterations, but we don't expect perfection first time.
>
> When the PPMC vote passes, there is a second vote on the IPMC. (Due
> Apache policy that only a "real" PMC can make a release. The IPMC is
> real PMC, but the PPMC is only a PMC-in-training.) That second step is
> arduous I'm afraid (and adds a few extra days) but it's not too bad by
> the 2nd or 3rd release.
>
> Julian
>
>
>
> On Tue, Mar 13, 2018 at 11:35 AM, Luciano Resende 
> wrote:
> > I would say the most important thing would be to get the legal parts in a
> > good state.
> >
> > For guide, I would start with :
> > - http://www.apache.org/dev/#releases
> >
> >
> > For actual steps, here is what I have used in other projects (which
> include
> > building, staging, signing, etc):
> >
> > https://github.com/apache/bahir/blob/master/dev/release-build.sh
> >
> >
> >
> > On Tue, Mar 13, 2018 at 11:21 AM, bernard metzler 
> wrote:
> >
> >> Dear Mentors,
> >>
> >> Looks like we are getting close to a point where we think the code
> >> base is good for a first release. I just searched around a little
> >> to see what is needed from an administrative point of view. What
> >> I found is an overwhelming amount of information, starting
> >> at https://incubator.apache.org/guides/releasemanagement.html
> >>
> >> So, what I understand is, we have to follow some formal policies
> >> applicable to all ASF project releases
> >> (http://www.apache.org/legal/release-policy.html) plus setting up
> >> the code base in a staged area (which would be
> >> https://dist.apache.org/repos/dist/dev/incubator/crail/ - which
> >> does not yet exist?), and finally go through a PMC voting procedure,
> >> as described at
> >> https://incubator.apache.org/guides/releasemanagement.html
> >>
> >>
> >> Is there anything else I am missing?
> >>
> >> Thank you!
> >> Bernard.
> >>
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>

[GitHub] incubator-crail pull request #18: README: update NVMf tier constants

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-crail/pull/18


---

[GitHub] incubator-crail pull request #17: New DiSNI and DaRPC version

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-crail/pull/17


---

[GitHub] incubator-crail pull request #18: README: update NVMf tier constants

[PepperJo]

GitHub user PepperJo opened a pull request:

https://github.com/apache/incubator-crail/pull/18

README: update NVMf tier constants

Update NVMf tier constants in README.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PepperJo/incubator-crail readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-crail/pull/18.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18


commit ad64c52fea3a1e15f3c21ae522444749fc7ce1fd
Author: Jonas Pfefferle 
Date:   2018-04-10T13:06:19Z

README: update NVMf tier constants

Update NVMf tier constants in README.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle 




---

[GitHub] incubator-crail issue #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on the issue:

https://github.com/apache/incubator-crail/pull/16
  
Fixed all the changes requested by Animesh.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180339324
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageClient.java 
---
@@ -31,41 +30,92 @@
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.NvmeTransportType;
+import java.io.IOException;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
 
 public class NvmfStorageClient implements StorageClient {
private static final Logger LOG = CrailUtils.getLogger();
-   private static NvmeEndpointGroup clientGroup;
-   private boolean initialized = false;
+   private static Nvme nvme;
+   private boolean initialized;
+   private volatile boolean closing;
+   private final Thread keepAliveThread;
+   private List endpoints;
+   private CrailStatistics statistics;
+   private CrailBufferCache bufferCache;
+
+   public NvmfStorageClient() {
+   this.initialized = false;
+   this.endpoints = new CopyOnWriteArrayList<>();
+   this.closing = false;
+   this.keepAliveThread = new Thread(() -> {
+   while (!closing) {
+   for (NvmfStorageEndpoint endpoint : endpoints) {
+   try {
+   endpoint.keepAlive();
+   } catch (IOException e) {
+   e.printStackTrace();
+   return;
+   }
+   }
+   /* We use the default keep alive timer of 120s 
in jNVMf */
+   try {
+   
Thread.sleep(TimeUnit.MILLISECONDS.convert(110, TimeUnit.SECONDS));
+   } catch (InterruptedException e) {
+   return;
+   }
+   }
+   });
+   }
+
+   boolean isValid() {
+   return keepAliveThread.isAlive();
+   }
 
public void init(CrailStatistics statistics, CrailBufferCache 
bufferCache, CrailConfiguration crailConfiguration,
 String[] args) throws IOException {
if (initialized) {
throw new IOException("NvmfStorageTier already 
initialized");
}
initialized = true;
-
+   this.statistics = statistics;
+   this.bufferCache = bufferCache;
+   LOG.info("Initialize Nvmf storage client");
NvmfStorageConstants.parseCmdLine(crailConfiguration, args);
+   keepAliveThread.start();
}
 
public void printConf(Logger logger) {
NvmfStorageConstants.printConf(logger);
}
 
-   public static NvmeEndpointGroup getEndpointGroup() {
-   if (clientGroup == null) {
-   clientGroup = new NvmeEndpointGroup(new 
NvmeTransportType[]{NvmeTransportType.RDMA},
-   NvmfStorageConstants.CLIENT_MEMPOOL);
+   public static Nvme getEndpointGroup() throws UnknownHostException {
--- End diff --

Should not be called from anywhere outside this class, so I will make it 
private. It is protected in createEndpoint (synchronized)


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180338259
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
+   }
+
+   private final int leftInSector(long address) {
+   return endpoint.getLBADataSize() - offsetInSector(address);
+   }
+
+   private final int offsetInSector(long address) {
+   return (int)(address % endpoint.getLBADataSize());
+   }
+
+   NvmfUnalignedWriteFuture(NvmfStorageEndpoint endpoint, CrailBuffer 
buffer, BlockInfo blockInfo, long remoteOffset) throws Exception {
+   this.endpoint = endpoint;
+   this.written = buffer.remaining();
+   /* assume blockInfo.getAddr() is sector aligned */
+   assert isSectorAligned(blockInfo.getAddr());
+
+   long nextRemoteOffset = remoteOffset;
+   /* beginning */
+   if (!isSectorAligned(remoteOffset)) {
+   int copySize = Math.min(leftInSector(remoteOffset), 
buffer.remaining());
+   nextRemoteOffset = remoteOffset + copySize;
+   int oldLimit = buffer.limit();
+   buffer.limit(buffer.position() + copySize);
+   long alignedRemoteOffset = 
floorToSectorSize(remoteOffset);
+   long alignedRemoteAddress = blockInfo.getAddr() + 
alignedRemoteOffset;
+   beginBuffer = 
endpoint.getStagingBufferCache().getExisting(alignedRemoteAddress);
+   if (beginBuffer == null) {
+   /* we had to delete the old buffer because we 
ran out of space. This should happen rarely. */
+   beginBuffer = 
endpoint.getStagingBufferCache().get(alignedRemoteAddress);
+   endpoint.read(beginBuffer.getBuffer(), 
blockInfo, alignedRemoteOffset).get();
+   } else {
+   /* Wait for previous end operation to finish */
+   beginBuffer.getFuture().get();
+   }
+   CrailBuffer stagingBuffer = beginBuffer.getBuffer();
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   
stagingBuffer.getByteBuffer().put(buffer.getByteBuffer());
+   buffer.limit(oldLimit);
+   stagingBuffer.position(0);
+   beginFuture = endpoint.write(stagingBuffer, blockInfo, 
alignedRemoteOffset);
+   beginBuffer.setFuture(beginFuture);
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   }
+
+   /* middle */
+   if 

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180337795
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfStorageEndpoint.java
 ---
@@ -19,208 +19,229 @@
 
 package org.apache.crail.storage.nvmf.client;
 
-import com.ibm.disni.nvmef.NvmeCommand;
-import com.ibm.disni.nvmef.NvmeEndpoint;
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.IOCompletion;
-
+import com.ibm.jnvmf.*;
 import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.CrailStatistics;
 import org.apache.crail.conf.CrailConstants;
-import org.apache.crail.memory.BufferCache;
 import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.metadata.DataNodeInfo;
 import org.apache.crail.storage.StorageEndpoint;
 import org.apache.crail.storage.StorageFuture;
-import org.apache.crail.storage.nvmf.NvmfBufferCache;
 import org.apache.crail.storage.nvmf.NvmfStorageConstants;
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
 import java.io.IOException;
+import java.net.InetAddress;
 import java.net.InetSocketAddress;
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.util.concurrent.*;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
 
 public class NvmfStorageEndpoint implements StorageEndpoint {
private static final Logger LOG = CrailUtils.getLogger();
 
-   private final InetSocketAddress inetSocketAddress;
-   private final NvmeEndpoint endpoint;
-   private final int sectorSize;
-   private final BufferCache cache;
-   private final BlockingQueue freeCommands;
-   private final NvmeCommand[] commands;
-   private final NvmfStorageFuture[] futures;
-   private final ThreadLocal completed;
-   private final int ioQeueueSize;
-
-   public NvmfStorageEndpoint(NvmeEndpointGroup group, InetSocketAddress 
inetSocketAddress) throws IOException {
-   this.inetSocketAddress = inetSocketAddress;
-   endpoint = group.createEndpoint();
+   private final Controller controller;
+   private final IoQueuePair queuePair;
+   private final int lbaDataSize;
+   private final long namespaceCapacity;
+   private final NvmfRegisteredBufferCache registeredBufferCache;
+   private final NvmfStagingBufferCache stagingBufferCache;
+   private final CrailStatistics statistics;
+
+   private final Queue writeCommands;
+   private final Queue readCommands;
+
+   private final AtomicInteger outstandingOperations;
+
+   public NvmfStorageEndpoint(Nvme nvme, DataNodeInfo info, 
CrailStatistics statistics,
+  CrailBufferCache 
bufferCache) throws IOException {
+   InetSocketAddress inetSocketAddress = new InetSocketAddress(
+   InetAddress.getByAddress(info.getIpAddress()), 
info.getPort());
+   // XXX FIXME: nsid from datanodeinfo
+   NvmfTransportId transportId = new 
NvmfTransportId(inetSocketAddress,
+   new 
NvmeQualifiedName(NvmfStorageConstants.NQN.toString() + info.getPort()));
+   LOG.info("Connecting to NVMf target at " + 
transportId.toString());
+   controller = nvme.connect(transportId);
+   controller.getControllerConfiguration().setEnable(true);
+   controller.syncConfiguration();
try {
-   URI url = new URI("nvmef://" + 
inetSocketAddress.getHostString() + ":" + inetSocketAddress.getPort() +
-   "/0/" + NvmfStorageConstants.NAMESPACE 
+ "?subsystem=nqn.2016-06.io.spdk:cnode1");
-   LOG.info("Connecting to " + url.toString());
-   endpoint.connect(url);
-   } catch (URISyntaxException e) {
-   //FIXME
-   e.printStackTrace();
+   controller.waitUntilReady();
+   } catch (TimeoutException e) {
+   throw new IOException(e);
}
-   sectorSize = endpoint.getSectorSize();
-   cache = new NvmfBufferCache();
-   ioQeueueSize = endpoint.getIOQueueSize();
-   freeCommands = new 
ArrayBlockingQueue(ioQeueueSize);
-   commands = new NvmeCommand[ioQeueueSize];
-   for (int i = 0; i < ioQeueueSize; i++) {
-   NvmeCommand command = endpoint.newCommand();
-   command.setId(i);

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180337495
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfStorageEndpoint.java
 ---
@@ -19,208 +19,229 @@
 
 package org.apache.crail.storage.nvmf.client;
 
-import com.ibm.disni.nvmef.NvmeCommand;
-import com.ibm.disni.nvmef.NvmeEndpoint;
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.IOCompletion;
-
+import com.ibm.jnvmf.*;
 import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.CrailStatistics;
 import org.apache.crail.conf.CrailConstants;
-import org.apache.crail.memory.BufferCache;
 import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.metadata.DataNodeInfo;
 import org.apache.crail.storage.StorageEndpoint;
 import org.apache.crail.storage.StorageFuture;
-import org.apache.crail.storage.nvmf.NvmfBufferCache;
 import org.apache.crail.storage.nvmf.NvmfStorageConstants;
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
 import java.io.IOException;
+import java.net.InetAddress;
 import java.net.InetSocketAddress;
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.util.concurrent.*;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
 
 public class NvmfStorageEndpoint implements StorageEndpoint {
private static final Logger LOG = CrailUtils.getLogger();
 
-   private final InetSocketAddress inetSocketAddress;
-   private final NvmeEndpoint endpoint;
-   private final int sectorSize;
-   private final BufferCache cache;
-   private final BlockingQueue freeCommands;
-   private final NvmeCommand[] commands;
-   private final NvmfStorageFuture[] futures;
-   private final ThreadLocal completed;
-   private final int ioQeueueSize;
-
-   public NvmfStorageEndpoint(NvmeEndpointGroup group, InetSocketAddress 
inetSocketAddress) throws IOException {
-   this.inetSocketAddress = inetSocketAddress;
-   endpoint = group.createEndpoint();
+   private final Controller controller;
+   private final IoQueuePair queuePair;
+   private final int lbaDataSize;
+   private final long namespaceCapacity;
+   private final NvmfRegisteredBufferCache registeredBufferCache;
+   private final NvmfStagingBufferCache stagingBufferCache;
+   private final CrailStatistics statistics;
+
+   private final Queue writeCommands;
+   private final Queue readCommands;
+
+   private final AtomicInteger outstandingOperations;
+
+   public NvmfStorageEndpoint(Nvme nvme, DataNodeInfo info, 
CrailStatistics statistics,
+  CrailBufferCache 
bufferCache) throws IOException {
+   InetSocketAddress inetSocketAddress = new InetSocketAddress(
+   InetAddress.getByAddress(info.getIpAddress()), 
info.getPort());
+   // XXX FIXME: nsid from datanodeinfo
+   NvmfTransportId transportId = new 
NvmfTransportId(inetSocketAddress,
+   new 
NvmeQualifiedName(NvmfStorageConstants.NQN.toString() + info.getPort()));
+   LOG.info("Connecting to NVMf target at " + 
transportId.toString());
+   controller = nvme.connect(transportId);
+   controller.getControllerConfiguration().setEnable(true);
+   controller.syncConfiguration();
try {
-   URI url = new URI("nvmef://" + 
inetSocketAddress.getHostString() + ":" + inetSocketAddress.getPort() +
-   "/0/" + NvmfStorageConstants.NAMESPACE 
+ "?subsystem=nqn.2016-06.io.spdk:cnode1");
-   LOG.info("Connecting to " + url.toString());
-   endpoint.connect(url);
-   } catch (URISyntaxException e) {
-   //FIXME
-   e.printStackTrace();
+   controller.waitUntilReady();
+   } catch (TimeoutException e) {
+   throw new IOException(e);
}
-   sectorSize = endpoint.getSectorSize();
-   cache = new NvmfBufferCache();
-   ioQeueueSize = endpoint.getIOQueueSize();
-   freeCommands = new 
ArrayBlockingQueue(ioQeueueSize);
-   commands = new NvmeCommand[ioQeueueSize];
-   for (int i = 0; i < ioQeueueSize; i++) {
-   NvmeCommand command = endpoint.newCommand();
-   command.setId(i);

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180337304
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageServer.java 
---
@@ -86,15 +100,15 @@ public void run() {
@Override
public StorageResource allocateResource() throws Exception {
StorageResource resource = null;
-   
+
if (alignedSize > 0){
LOG.info("new block, length " + 
NvmfStorageConstants.ALLOCATION_SIZE);
-   LOG.debug("block stag 0, offset " + offset + ", length 
" + NvmfStorageConstants.ALLOCATION_SIZE);
+   LOG.debug("block stag 0, address " + address + ", 
length " + NvmfStorageConstants.ALLOCATION_SIZE);
alignedSize -= NvmfStorageConstants.ALLOCATION_SIZE;
-   resource = StorageResource.createResource(offset, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
-   offset += NvmfStorageConstants.ALLOCATION_SIZE;
+   resource = StorageResource.createResource(address, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
--- End diff --

Good point. Not sure why I made them long in the first place.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180332615
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
+   }
+
+   private final int leftInSector(long address) {
+   return endpoint.getLBADataSize() - offsetInSector(address);
+   }
+
+   private final int offsetInSector(long address) {
+   return (int)(address % endpoint.getLBADataSize());
+   }
+
+   NvmfUnalignedWriteFuture(NvmfStorageEndpoint endpoint, CrailBuffer 
buffer, BlockInfo blockInfo, long remoteOffset) throws Exception {
+   this.endpoint = endpoint;
+   this.written = buffer.remaining();
+   /* assume blockInfo.getAddr() is sector aligned */
+   assert isSectorAligned(blockInfo.getAddr());
+
+   long nextRemoteOffset = remoteOffset;
+   /* beginning */
+   if (!isSectorAligned(remoteOffset)) {
+   int copySize = Math.min(leftInSector(remoteOffset), 
buffer.remaining());
+   nextRemoteOffset = remoteOffset + copySize;
+   int oldLimit = buffer.limit();
+   buffer.limit(buffer.position() + copySize);
+   long alignedRemoteOffset = 
floorToSectorSize(remoteOffset);
+   long alignedRemoteAddress = blockInfo.getAddr() + 
alignedRemoteOffset;
+   beginBuffer = 
endpoint.getStagingBufferCache().getExisting(alignedRemoteAddress);
+   if (beginBuffer == null) {
+   /* we had to delete the old buffer because we 
ran out of space. This should happen rarely. */
+   beginBuffer = 
endpoint.getStagingBufferCache().get(alignedRemoteAddress);
+   endpoint.read(beginBuffer.getBuffer(), 
blockInfo, alignedRemoteOffset).get();
+   } else {
+   /* Wait for previous end operation to finish */
+   beginBuffer.getFuture().get();
+   }
+   CrailBuffer stagingBuffer = beginBuffer.getBuffer();
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   
stagingBuffer.getByteBuffer().put(buffer.getByteBuffer());
+   buffer.limit(oldLimit);
+   stagingBuffer.position(0);
+   beginFuture = endpoint.write(stagingBuffer, blockInfo, 
alignedRemoteOffset);
+   beginBuffer.setFuture(beginFuture);
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   }
+
+   /* middle */
+   if 

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180332561
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
+   }
+
+   private final int leftInSector(long address) {
+   return endpoint.getLBADataSize() - offsetInSector(address);
+   }
+
+   private final int offsetInSector(long address) {
+   return (int)(address % endpoint.getLBADataSize());
+   }
+
+   NvmfUnalignedWriteFuture(NvmfStorageEndpoint endpoint, CrailBuffer 
buffer, BlockInfo blockInfo, long remoteOffset) throws Exception {
+   this.endpoint = endpoint;
+   this.written = buffer.remaining();
+   /* assume blockInfo.getAddr() is sector aligned */
+   assert isSectorAligned(blockInfo.getAddr());
+
+   long nextRemoteOffset = remoteOffset;
+   /* beginning */
+   if (!isSectorAligned(remoteOffset)) {
+   int copySize = Math.min(leftInSector(remoteOffset), 
buffer.remaining());
+   nextRemoteOffset = remoteOffset + copySize;
+   int oldLimit = buffer.limit();
+   buffer.limit(buffer.position() + copySize);
+   long alignedRemoteOffset = 
floorToSectorSize(remoteOffset);
+   long alignedRemoteAddress = blockInfo.getAddr() + 
alignedRemoteOffset;
+   beginBuffer = 
endpoint.getStagingBufferCache().getExisting(alignedRemoteAddress);
+   if (beginBuffer == null) {
+   /* we had to delete the old buffer because we 
ran out of space. This should happen rarely. */
+   beginBuffer = 
endpoint.getStagingBufferCache().get(alignedRemoteAddress);
+   endpoint.read(beginBuffer.getBuffer(), 
blockInfo, alignedRemoteOffset).get();
+   } else {
+   /* Wait for previous end operation to finish */
+   beginBuffer.getFuture().get();
+   }
+   CrailBuffer stagingBuffer = beginBuffer.getBuffer();
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   
stagingBuffer.getByteBuffer().put(buffer.getByteBuffer());
+   buffer.limit(oldLimit);
+   stagingBuffer.position(0);
+   beginFuture = endpoint.write(stagingBuffer, blockInfo, 
alignedRemoteOffset);
+   beginBuffer.setFuture(beginFuture);
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   }
+
+   /* middle */
+   if 

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180332038
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
--- End diff --

Will do.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180331658
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageServer.java 
---
@@ -31,51 +28,68 @@
 
 import java.io.IOException;
 import java.net.InetSocketAddress;
-import java.net.URI;
+import java.util.List;
 
 public class NvmfStorageServer implements StorageServer {
private static final Logger LOG = CrailUtils.getLogger();
 
private boolean isAlive;
private long alignedSize;
-   private long offset;
+   private long address;
private boolean initialized = false;
-   private NvmeEndpoint endpoint;
+   private Controller controller;
 
public NvmfStorageServer() {}
-   
+
public void init(CrailConfiguration crailConfiguration, String[] args) 
throws Exception {
if (initialized) {
throw new IOException("NvmfStorageTier already 
initialized");
}
initialized = true;
NvmfStorageConstants.parseCmdLine(crailConfiguration, args);
 
-   NvmeEndpointGroup group = new NvmeEndpointGroup(new 
NvmeTransportType[]{NvmeTransportType.RDMA}, 
NvmfStorageConstants.SERVER_MEMPOOL);
-   endpoint = group.createEndpoint();
-
-   URI uri = new URI("nvmef://" + 
NvmfStorageConstants.IP_ADDR.getHostAddress() + ":" + NvmfStorageConstants.PORT 
+
-   "/0/" + NvmfStorageConstants.NAMESPACE 
+ "?subsystem=" + NvmfStorageConstants.NQN);
-   endpoint.connect(uri);
-
-   long namespaceSize = endpoint.getNamespaceSize();
+   Nvme nvme = new Nvme();
+   NvmfTransportId transportId = new NvmfTransportId(
+   new 
InetSocketAddress(NvmfStorageConstants.IP_ADDR, NvmfStorageConstants.PORT),
+   NvmfStorageConstants.NQN);
+   controller = nvme.connect(transportId);
+   controller.getControllerConfiguration().setEnable(true);
+   controller.syncConfiguration();
+   controller.waitUntilReady();
+
+   List namespaces = controller.getActiveNamespaces();
+   Namespace namespace = null;
+   for (Namespace n : namespaces) {
+   if 
(n.getIdentifier().equals(NvmfStorageConstants.NAMESPACE)) {
+   namespace = n;
+   break;
+   }
+   }
+   if (namespace == null) {
+   throw new IllegalArgumentException("No namespace with 
id " + NvmfStorageConstants.NAMESPACE +
+   " at controller " + 
transportId.toString());
+   }
+   IdentifyNamespaceData namespaceData = 
namespace.getIdentifyNamespaceData();
+   LbaFormat lbaFormat = namespaceData.getFormattedLbaSize();
+   int dataSize = lbaFormat.getLbaDataSize().toInt();
+   long namespaceSize = dataSize * 
namespaceData.getNamespaceCapacity();
alignedSize = namespaceSize - (namespaceSize % 
NvmfStorageConstants.ALLOCATION_SIZE);
-   offset = 0;
+   address = 0;
 
isAlive = true;
-   }   
+   }
 
@Override
public void printConf(Logger log) {
-   NvmfStorageConstants.printConf(log);
+   NvmfStorageConstants.printConf(log);
}
 
public void run() {
LOG.info("NnvmfStorageServer started with NVMf target " + 
getAddress());
while (isAlive) {
try {
Thread.sleep(1000 /* ms */);
--- End diff --

Let me make the keep alive time a final static to be used in both storage 
server and client.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180331213
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageConstants.java
 ---
@@ -89,31 +76,31 @@ public static void updateConstants(CrailConfiguration 
conf) throws UnknownHostEx
 
arg = get(conf, NQN_KEY);
if (arg != null) {
-   NQN = arg;
+   NQN = new NvmeQualifiedName(arg);
}
 
arg = get(conf, ALLOCATION_SIZE_KEY);
if (arg != null) {
ALLOCATION_SIZE = Long.parseLong(arg);
}
 
-   arg = get(conf, SERVER_MEMPOOL_KEY);
+   arg = get(conf, QUEUE_SIZE_KEY);
if (arg != null) {
-   SERVER_MEMPOOL = Long.parseLong(arg);
+   QUEUE_SIZE = Integer.parseInt(arg);
}
 
-   arg = get(conf, CLIENT_MEMPOOL_KEY);
+   arg = get(conf, STAGING_CACHE_SIZE_KEY);
if (arg != null) {
-   CLIENT_MEMPOOL = Long.parseLong(arg);
+   STAGING_CACHE_SIZE = Integer.parseInt(arg);
}
}
 
-   public static void verify() throws IOException {
-   if (NAMESPACE <= 0){
-   throw new IOException("Namespace must be > 0");
-   }
+   public static void verify() {
if (ALLOCATION_SIZE % CrailConstants.BLOCK_SIZE != 0){
-   throw new IOException("allocationsize must be multiple 
of crail.blocksize");
+   throw new IllegalArgumentException("allocationsize must 
be multiple of crail.blocksize");
--- End diff --

Will do.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180331153
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageClient.java 
---
@@ -31,41 +30,92 @@
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.NvmeTransportType;
+import java.io.IOException;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
 
 public class NvmfStorageClient implements StorageClient {
private static final Logger LOG = CrailUtils.getLogger();
-   private static NvmeEndpointGroup clientGroup;
-   private boolean initialized = false;
+   private static Nvme nvme;
+   private boolean initialized;
+   private volatile boolean closing;
+   private final Thread keepAliveThread;
+   private List endpoints;
+   private CrailStatistics statistics;
+   private CrailBufferCache bufferCache;
+
+   public NvmfStorageClient() {
+   this.initialized = false;
+   this.endpoints = new CopyOnWriteArrayList<>();
+   this.closing = false;
+   this.keepAliveThread = new Thread(() -> {
+   while (!closing) {
+   for (NvmfStorageEndpoint endpoint : endpoints) {
+   try {
+   endpoint.keepAlive();
+   } catch (IOException e) {
+   e.printStackTrace();
+   return;
+   }
+   }
+   /* We use the default keep alive timer of 120s 
in jNVMf */
+   try {
+   
Thread.sleep(TimeUnit.MILLISECONDS.convert(110, TimeUnit.SECONDS));
+   } catch (InterruptedException e) {
+   return;
+   }
+   }
+   });
+   }
+
+   boolean isValid() {
--- End diff --

Makes sense. Will change.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[PepperJo]

Github user PepperJo commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180331005
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageClient.java 
---
@@ -31,41 +30,92 @@
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.NvmeTransportType;
+import java.io.IOException;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
 
 public class NvmfStorageClient implements StorageClient {
private static final Logger LOG = CrailUtils.getLogger();
-   private static NvmeEndpointGroup clientGroup;
-   private boolean initialized = false;
+   private static Nvme nvme;
+   private boolean initialized;
+   private volatile boolean closing;
+   private final Thread keepAliveThread;
+   private List endpoints;
+   private CrailStatistics statistics;
+   private CrailBufferCache bufferCache;
+
+   public NvmfStorageClient() {
+   this.initialized = false;
+   this.endpoints = new CopyOnWriteArrayList<>();
+   this.closing = false;
+   this.keepAliveThread = new Thread(() -> {
+   while (!closing) {
+   for (NvmfStorageEndpoint endpoint : endpoints) {
+   try {
+   endpoint.keepAlive();
+   } catch (IOException e) {
+   e.printStackTrace();
+   return;
+   }
+   }
+   /* We use the default keep alive timer of 120s 
in jNVMf */
+   try {
+   
Thread.sleep(TimeUnit.MILLISECONDS.convert(110, TimeUnit.SECONDS));
--- End diff --

For now the keep alive timer is not configurable. 110 because we don't want 
to be late, if we wait until 120s the timer might have passed especially since 
we have to send keep alive messages to all connected endpoints.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180317997
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfStorageEndpoint.java
 ---
@@ -19,208 +19,229 @@
 
 package org.apache.crail.storage.nvmf.client;
 
-import com.ibm.disni.nvmef.NvmeCommand;
-import com.ibm.disni.nvmef.NvmeEndpoint;
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.IOCompletion;
-
+import com.ibm.jnvmf.*;
 import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.CrailStatistics;
 import org.apache.crail.conf.CrailConstants;
-import org.apache.crail.memory.BufferCache;
 import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.metadata.DataNodeInfo;
 import org.apache.crail.storage.StorageEndpoint;
 import org.apache.crail.storage.StorageFuture;
-import org.apache.crail.storage.nvmf.NvmfBufferCache;
 import org.apache.crail.storage.nvmf.NvmfStorageConstants;
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
 import java.io.IOException;
+import java.net.InetAddress;
 import java.net.InetSocketAddress;
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.util.concurrent.*;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
 
 public class NvmfStorageEndpoint implements StorageEndpoint {
private static final Logger LOG = CrailUtils.getLogger();
 
-   private final InetSocketAddress inetSocketAddress;
-   private final NvmeEndpoint endpoint;
-   private final int sectorSize;
-   private final BufferCache cache;
-   private final BlockingQueue freeCommands;
-   private final NvmeCommand[] commands;
-   private final NvmfStorageFuture[] futures;
-   private final ThreadLocal completed;
-   private final int ioQeueueSize;
-
-   public NvmfStorageEndpoint(NvmeEndpointGroup group, InetSocketAddress 
inetSocketAddress) throws IOException {
-   this.inetSocketAddress = inetSocketAddress;
-   endpoint = group.createEndpoint();
+   private final Controller controller;
+   private final IoQueuePair queuePair;
+   private final int lbaDataSize;
+   private final long namespaceCapacity;
+   private final NvmfRegisteredBufferCache registeredBufferCache;
+   private final NvmfStagingBufferCache stagingBufferCache;
+   private final CrailStatistics statistics;
+
+   private final Queue writeCommands;
+   private final Queue readCommands;
+
+   private final AtomicInteger outstandingOperations;
+
+   public NvmfStorageEndpoint(Nvme nvme, DataNodeInfo info, 
CrailStatistics statistics,
+  CrailBufferCache 
bufferCache) throws IOException {
+   InetSocketAddress inetSocketAddress = new InetSocketAddress(
+   InetAddress.getByAddress(info.getIpAddress()), 
info.getPort());
+   // XXX FIXME: nsid from datanodeinfo
+   NvmfTransportId transportId = new 
NvmfTransportId(inetSocketAddress,
+   new 
NvmeQualifiedName(NvmfStorageConstants.NQN.toString() + info.getPort()));
+   LOG.info("Connecting to NVMf target at " + 
transportId.toString());
+   controller = nvme.connect(transportId);
+   controller.getControllerConfiguration().setEnable(true);
+   controller.syncConfiguration();
try {
-   URI url = new URI("nvmef://" + 
inetSocketAddress.getHostString() + ":" + inetSocketAddress.getPort() +
-   "/0/" + NvmfStorageConstants.NAMESPACE 
+ "?subsystem=nqn.2016-06.io.spdk:cnode1");
-   LOG.info("Connecting to " + url.toString());
-   endpoint.connect(url);
-   } catch (URISyntaxException e) {
-   //FIXME
-   e.printStackTrace();
+   controller.waitUntilReady();
+   } catch (TimeoutException e) {
+   throw new IOException(e);
}
-   sectorSize = endpoint.getSectorSize();
-   cache = new NvmfBufferCache();
-   ioQeueueSize = endpoint.getIOQueueSize();
-   freeCommands = new 
ArrayBlockingQueue(ioQeueueSize);
-   commands = new NvmeCommand[ioQeueueSize];
-   for (int i = 0; i < ioQeueueSize; i++) {
-   NvmeCommand command = endpoint.newCommand();
-   

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180320774
  
--- Diff: 
storage-nvmf/src/test/java/org/apache/crail/storage/nvmf/client/NvmfStagingBufferCacheTest.java
 ---
@@ -0,0 +1,73 @@
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.conf.CrailConfiguration;
+import org.apache.crail.conf.CrailConstants;
+import org.apache.crail.memory.MappedBufferCache;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.junit.Assert.*;
+
+public class NvmfStagingBufferCacheTest {
--- End diff --

:+1: 


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180315936
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfStagingBufferCache.java
 ---
@@ -0,0 +1,160 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.storage.StorageFuture;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class NvmfStagingBufferCache {
+   private final Map remoteAddressMap;
+   private final Queue freeBuffers;
+   private int buffersLeft;
+   private final int lbaDataSize;
+   private final CrailBufferCache bufferCache;
+
+   private final CrailBufferCache getBufferCache() {
+   return bufferCache;
+   }
+
+   NvmfStagingBufferCache(CrailBufferCache bufferCache, int maxEntries, 
int lbaDataSize) {
+   if (maxEntries <= 0) {
+   throw new IllegalArgumentException("maximum entries <= 
0");
+   }
+   if (lbaDataSize <= 0) {
+   throw new IllegalArgumentException("LBA data size <= 
0");
+   }
+   this.remoteAddressMap = new ConcurrentHashMap<>(maxEntries);
+   this.freeBuffers = new ArrayBlockingQueue<>(maxEntries);
+   this.buffersLeft = maxEntries;
+   this.lbaDataSize = lbaDataSize;
+   this.bufferCache = bufferCache;
+   }
+
+   synchronized void allocateFreeBuffers() throws Exception {
+   if (!freeBuffers.isEmpty()) {
+   return;
+   }
+   if (buffersLeft == 0) {
+   /* TODO: make sure this happens rarely */
+   Iterator iterator = 
remoteAddressMap.values().iterator();
+   while (iterator.hasNext()) {
+   BufferCacheEntry currentEntry = iterator.next();
+   if (currentEntry.tryFree()) {
+   iterator.remove();
+   
freeBuffers.add(currentEntry.getBuffer());
+   return;
+   }
+   }
+   throw new OutOfMemoryError();
+   }
+
+   CrailBuffer buffer = getBufferCache().allocateBuffer();
+   if (buffer == null) {
+   throw new OutOfMemoryError();
+   }
+   if (buffer.capacity() < lbaDataSize) {
--- End diff --

please print them so that user knows what went wrong. 


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180312653
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageClient.java 
---
@@ -31,41 +30,92 @@
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.NvmeTransportType;
+import java.io.IOException;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
 
 public class NvmfStorageClient implements StorageClient {
private static final Logger LOG = CrailUtils.getLogger();
-   private static NvmeEndpointGroup clientGroup;
-   private boolean initialized = false;
+   private static Nvme nvme;
+   private boolean initialized;
+   private volatile boolean closing;
+   private final Thread keepAliveThread;
+   private List endpoints;
+   private CrailStatistics statistics;
+   private CrailBufferCache bufferCache;
+
+   public NvmfStorageClient() {
+   this.initialized = false;
+   this.endpoints = new CopyOnWriteArrayList<>();
+   this.closing = false;
+   this.keepAliveThread = new Thread(() -> {
+   while (!closing) {
+   for (NvmfStorageEndpoint endpoint : endpoints) {
+   try {
+   endpoint.keepAlive();
+   } catch (IOException e) {
+   e.printStackTrace();
+   return;
+   }
+   }
+   /* We use the default keep alive timer of 120s 
in jNVMf */
+   try {
+   
Thread.sleep(TimeUnit.MILLISECONDS.convert(110, TimeUnit.SECONDS));
+   } catch (InterruptedException e) {
+   return;
+   }
+   }
+   });
+   }
+
+   boolean isValid() {
+   return keepAliveThread.isAlive();
+   }
 
public void init(CrailStatistics statistics, CrailBufferCache 
bufferCache, CrailConfiguration crailConfiguration,
 String[] args) throws IOException {
if (initialized) {
throw new IOException("NvmfStorageTier already 
initialized");
}
initialized = true;
-
+   this.statistics = statistics;
+   this.bufferCache = bufferCache;
+   LOG.info("Initialize Nvmf storage client");
NvmfStorageConstants.parseCmdLine(crailConfiguration, args);
+   keepAliveThread.start();
}
 
public void printConf(Logger logger) {
NvmfStorageConstants.printConf(logger);
}
 
-   public static NvmeEndpointGroup getEndpointGroup() {
-   if (clientGroup == null) {
-   clientGroup = new NvmeEndpointGroup(new 
NvmeTransportType[]{NvmeTransportType.RDMA},
-   NvmfStorageConstants.CLIENT_MEMPOOL);
+   public static Nvme getEndpointGroup() throws UnknownHostException {
--- End diff --

Does this public and static function need protection against the race 
initialization condition? 


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180314443
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageServer.java 
---
@@ -86,15 +100,15 @@ public void run() {
@Override
public StorageResource allocateResource() throws Exception {
StorageResource resource = null;
-   
+
if (alignedSize > 0){
LOG.info("new block, length " + 
NvmfStorageConstants.ALLOCATION_SIZE);
-   LOG.debug("block stag 0, offset " + offset + ", length 
" + NvmfStorageConstants.ALLOCATION_SIZE);
+   LOG.debug("block stag 0, address " + address + ", 
length " + NvmfStorageConstants.ALLOCATION_SIZE);
alignedSize -= NvmfStorageConstants.ALLOCATION_SIZE;
-   resource = StorageResource.createResource(offset, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
-   offset += NvmfStorageConstants.ALLOCATION_SIZE;
+   resource = StorageResource.createResource(address, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
--- End diff --

#sad. Why not make ALLOCATION_SIZE an integer to start with. It is a long 
which is bad because a user might be tempted to give it a > 2GB values 
considering it is a long. 


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180315522
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfStagingBufferCache.java
 ---
@@ -0,0 +1,160 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.CrailBufferCache;
+import org.apache.crail.storage.StorageFuture;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class NvmfStagingBufferCache {
+   private final Map remoteAddressMap;
+   private final Queue freeBuffers;
+   private int buffersLeft;
+   private final int lbaDataSize;
+   private final CrailBufferCache bufferCache;
+
+   private final CrailBufferCache getBufferCache() {
+   return bufferCache;
+   }
+
+   NvmfStagingBufferCache(CrailBufferCache bufferCache, int maxEntries, 
int lbaDataSize) {
+   if (maxEntries <= 0) {
--- End diff --

Here as well, print the value that caused the error.


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180312351
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageClient.java 
---
@@ -31,41 +30,92 @@
 import org.apache.crail.utils.CrailUtils;
 import org.slf4j.Logger;
 
-import com.ibm.disni.nvmef.NvmeEndpointGroup;
-import com.ibm.disni.nvmef.spdk.NvmeTransportType;
+import java.io.IOException;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
 
 public class NvmfStorageClient implements StorageClient {
private static final Logger LOG = CrailUtils.getLogger();
-   private static NvmeEndpointGroup clientGroup;
-   private boolean initialized = false;
+   private static Nvme nvme;
+   private boolean initialized;
+   private volatile boolean closing;
+   private final Thread keepAliveThread;
+   private List endpoints;
+   private CrailStatistics statistics;
+   private CrailBufferCache bufferCache;
+
+   public NvmfStorageClient() {
+   this.initialized = false;
+   this.endpoints = new CopyOnWriteArrayList<>();
+   this.closing = false;
+   this.keepAliveThread = new Thread(() -> {
+   while (!closing) {
+   for (NvmfStorageEndpoint endpoint : endpoints) {
+   try {
+   endpoint.keepAlive();
+   } catch (IOException e) {
+   e.printStackTrace();
+   return;
+   }
+   }
+   /* We use the default keep alive timer of 120s 
in jNVMf */
+   try {
+   
Thread.sleep(TimeUnit.MILLISECONDS.convert(110, TimeUnit.SECONDS));
--- End diff --

110 or 120 seconds (as the comment says). Is this configurable? 


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180319133
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
+   }
+
+   private final int leftInSector(long address) {
+   return endpoint.getLBADataSize() - offsetInSector(address);
+   }
+
+   private final int offsetInSector(long address) {
+   return (int)(address % endpoint.getLBADataSize());
+   }
+
+   NvmfUnalignedWriteFuture(NvmfStorageEndpoint endpoint, CrailBuffer 
buffer, BlockInfo blockInfo, long remoteOffset) throws Exception {
+   this.endpoint = endpoint;
+   this.written = buffer.remaining();
+   /* assume blockInfo.getAddr() is sector aligned */
+   assert isSectorAligned(blockInfo.getAddr());
+
+   long nextRemoteOffset = remoteOffset;
+   /* beginning */
+   if (!isSectorAligned(remoteOffset)) {
+   int copySize = Math.min(leftInSector(remoteOffset), 
buffer.remaining());
+   nextRemoteOffset = remoteOffset + copySize;
+   int oldLimit = buffer.limit();
+   buffer.limit(buffer.position() + copySize);
+   long alignedRemoteOffset = 
floorToSectorSize(remoteOffset);
+   long alignedRemoteAddress = blockInfo.getAddr() + 
alignedRemoteOffset;
+   beginBuffer = 
endpoint.getStagingBufferCache().getExisting(alignedRemoteAddress);
+   if (beginBuffer == null) {
+   /* we had to delete the old buffer because we 
ran out of space. This should happen rarely. */
+   beginBuffer = 
endpoint.getStagingBufferCache().get(alignedRemoteAddress);
+   endpoint.read(beginBuffer.getBuffer(), 
blockInfo, alignedRemoteOffset).get();
+   } else {
+   /* Wait for previous end operation to finish */
+   beginBuffer.getFuture().get();
+   }
+   CrailBuffer stagingBuffer = beginBuffer.getBuffer();
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   
stagingBuffer.getByteBuffer().put(buffer.getByteBuffer());
+   buffer.limit(oldLimit);
+   stagingBuffer.position(0);
+   beginFuture = endpoint.write(stagingBuffer, blockInfo, 
alignedRemoteOffset);
+   beginBuffer.setFuture(beginFuture);
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   }
+
+   /* middle */
+   if 

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180318375
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
--- End diff --

use your own function? (offsetInSector) "return address - offsetInSector; "


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180320719
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/client/NvmfUnalignedWriteFuture.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2018, IBM Corporation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.crail.storage.nvmf.client;
+
+import org.apache.crail.CrailBuffer;
+import org.apache.crail.metadata.BlockInfo;
+import org.apache.crail.storage.StorageFuture;
+import org.apache.crail.storage.StorageResult;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+
+public class NvmfUnalignedWriteFuture implements StorageFuture {
+   private final NvmfStorageEndpoint endpoint;
+   private StorageFuture beginFuture;
+   private StorageFuture middleFuture;
+   private StorageFuture endFuture;
+   private final int written;
+   private NvmfStagingBufferCache.BufferCacheEntry beginBuffer;
+   private NvmfStagingBufferCache.BufferCacheEntry endBuffer;
+
+   private final boolean isSectorAligned(long address) {
+   return address % endpoint.getLBADataSize() == 0;
+   }
+
+   private final long floorToSectorSize(long address) {
+   return address - (address % endpoint.getLBADataSize());
+   }
+
+   private final int leftInSector(long address) {
+   return endpoint.getLBADataSize() - offsetInSector(address);
+   }
+
+   private final int offsetInSector(long address) {
+   return (int)(address % endpoint.getLBADataSize());
+   }
+
+   NvmfUnalignedWriteFuture(NvmfStorageEndpoint endpoint, CrailBuffer 
buffer, BlockInfo blockInfo, long remoteOffset) throws Exception {
+   this.endpoint = endpoint;
+   this.written = buffer.remaining();
+   /* assume blockInfo.getAddr() is sector aligned */
+   assert isSectorAligned(blockInfo.getAddr());
+
+   long nextRemoteOffset = remoteOffset;
+   /* beginning */
+   if (!isSectorAligned(remoteOffset)) {
+   int copySize = Math.min(leftInSector(remoteOffset), 
buffer.remaining());
+   nextRemoteOffset = remoteOffset + copySize;
+   int oldLimit = buffer.limit();
+   buffer.limit(buffer.position() + copySize);
+   long alignedRemoteOffset = 
floorToSectorSize(remoteOffset);
+   long alignedRemoteAddress = blockInfo.getAddr() + 
alignedRemoteOffset;
+   beginBuffer = 
endpoint.getStagingBufferCache().getExisting(alignedRemoteAddress);
+   if (beginBuffer == null) {
+   /* we had to delete the old buffer because we 
ran out of space. This should happen rarely. */
+   beginBuffer = 
endpoint.getStagingBufferCache().get(alignedRemoteAddress);
+   endpoint.read(beginBuffer.getBuffer(), 
blockInfo, alignedRemoteOffset).get();
+   } else {
+   /* Wait for previous end operation to finish */
+   beginBuffer.getFuture().get();
+   }
+   CrailBuffer stagingBuffer = beginBuffer.getBuffer();
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   
stagingBuffer.getByteBuffer().put(buffer.getByteBuffer());
+   buffer.limit(oldLimit);
+   stagingBuffer.position(0);
+   beginFuture = endpoint.write(stagingBuffer, blockInfo, 
alignedRemoteOffset);
+   beginBuffer.setFuture(beginFuture);
+   stagingBuffer.position(offsetInSector(remoteOffset));
+   }
+
+   /* middle */
+   if 

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180313817
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageServer.java 
---
@@ -31,51 +28,68 @@
 
 import java.io.IOException;
 import java.net.InetSocketAddress;
-import java.net.URI;
+import java.util.List;
 
 public class NvmfStorageServer implements StorageServer {
private static final Logger LOG = CrailUtils.getLogger();
 
private boolean isAlive;
private long alignedSize;
-   private long offset;
+   private long address;
private boolean initialized = false;
-   private NvmeEndpoint endpoint;
+   private Controller controller;
 
public NvmfStorageServer() {}
-   
+
public void init(CrailConfiguration crailConfiguration, String[] args) 
throws Exception {
if (initialized) {
throw new IOException("NvmfStorageTier already 
initialized");
}
initialized = true;
NvmfStorageConstants.parseCmdLine(crailConfiguration, args);
 
-   NvmeEndpointGroup group = new NvmeEndpointGroup(new 
NvmeTransportType[]{NvmeTransportType.RDMA}, 
NvmfStorageConstants.SERVER_MEMPOOL);
-   endpoint = group.createEndpoint();
-
-   URI uri = new URI("nvmef://" + 
NvmfStorageConstants.IP_ADDR.getHostAddress() + ":" + NvmfStorageConstants.PORT 
+
-   "/0/" + NvmfStorageConstants.NAMESPACE 
+ "?subsystem=" + NvmfStorageConstants.NQN);
-   endpoint.connect(uri);
-
-   long namespaceSize = endpoint.getNamespaceSize();
+   Nvme nvme = new Nvme();
+   NvmfTransportId transportId = new NvmfTransportId(
+   new 
InetSocketAddress(NvmfStorageConstants.IP_ADDR, NvmfStorageConstants.PORT),
+   NvmfStorageConstants.NQN);
+   controller = nvme.connect(transportId);
+   controller.getControllerConfiguration().setEnable(true);
+   controller.syncConfiguration();
+   controller.waitUntilReady();
+
+   List namespaces = controller.getActiveNamespaces();
+   Namespace namespace = null;
+   for (Namespace n : namespaces) {
+   if 
(n.getIdentifier().equals(NvmfStorageConstants.NAMESPACE)) {
+   namespace = n;
+   break;
+   }
+   }
+   if (namespace == null) {
+   throw new IllegalArgumentException("No namespace with 
id " + NvmfStorageConstants.NAMESPACE +
+   " at controller " + 
transportId.toString());
+   }
+   IdentifyNamespaceData namespaceData = 
namespace.getIdentifyNamespaceData();
+   LbaFormat lbaFormat = namespaceData.getFormattedLbaSize();
+   int dataSize = lbaFormat.getLbaDataSize().toInt();
+   long namespaceSize = dataSize * 
namespaceData.getNamespaceCapacity();
alignedSize = namespaceSize - (namespaceSize % 
NvmfStorageConstants.ALLOCATION_SIZE);
-   offset = 0;
+   address = 0;
 
isAlive = true;
-   }   
+   }
 
@Override
public void printConf(Logger log) {
-   NvmfStorageConstants.printConf(log);
+   NvmfStorageConstants.printConf(log);
}
 
public void run() {
LOG.info("NnvmfStorageServer started with NVMf target " + 
getAddress());
while (isAlive) {
try {
Thread.sleep(1000 /* ms */);
--- End diff --

Magic 1000 milliseconds :)


---

[GitHub] incubator-crail pull request #16: New NVMf storage based on jNVMf library

[animeshtrivedi]

Github user animeshtrivedi commented on a diff in the pull request:

https://github.com/apache/incubator-crail/pull/16#discussion_r180314572
  
--- Diff: 
storage-nvmf/src/main/java/org/apache/crail/storage/nvmf/NvmfStorageServer.java 
---
@@ -86,15 +100,15 @@ public void run() {
@Override
public StorageResource allocateResource() throws Exception {
StorageResource resource = null;
-   
+
if (alignedSize > 0){
LOG.info("new block, length " + 
NvmfStorageConstants.ALLOCATION_SIZE);
-   LOG.debug("block stag 0, offset " + offset + ", length 
" + NvmfStorageConstants.ALLOCATION_SIZE);
+   LOG.debug("block stag 0, address " + address + ", 
length " + NvmfStorageConstants.ALLOCATION_SIZE);
alignedSize -= NvmfStorageConstants.ALLOCATION_SIZE;
-   resource = StorageResource.createResource(offset, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
-   offset += NvmfStorageConstants.ALLOCATION_SIZE;
+   resource = StorageResource.createResource(address, 
(int)NvmfStorageConstants.ALLOCATION_SIZE, 0);
--- End diff --

Same goes for other NvmfStorageConstants constants as well. 


---