Repository: incubator-crail Updated Branches: refs/heads/master 34e49d29d -> 3a2c8098c
Documentation: Add programming documentation Add documentation how to program against Crail. https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-55 Signed-off-by: Jonas Pfefferle <peppe...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/incubator-crail/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-crail/commit/3a2c8098 Tree: http://git-wip-us.apache.org/repos/asf/incubator-crail/tree/3a2c8098 Diff: http://git-wip-us.apache.org/repos/asf/incubator-crail/diff/3a2c8098 Branch: refs/heads/master Commit: 3a2c8098c716bad8893bfdbec1bc2ea97811ecae Parents: cdb5401 Author: Jonas Pfefferle <peppe...@apache.org> Authored: Thu Oct 18 16:30:42 2018 +0200 Committer: Jonas Pfefferle <peppe...@apache.org> Committed: Thu Oct 18 16:43:35 2018 +0200 ---------------------------------------------------------------------- doc/source/develop.rst | 58 +++++++++++++++++++++++++++++++++++++++++++++ doc/source/index.rst | 2 ++ 2 files changed, 60 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/3a2c8098/doc/source/develop.rst ---------------------------------------------------------------------- diff --git a/doc/source/develop.rst b/doc/source/develop.rst new file mode 100644 index 0000000..d0c3845 --- /dev/null +++ b/doc/source/develop.rst @@ -0,0 +1,58 @@ +Programming against Crail +========================= + +The best way to program against Crail is to use Maven. Make sure you have the Crail dependency specified in your application pom.xml file +with the latest Crail version (e.g. 1.1-incubating): + +.. code-block:: xml + + <dependency> + <groupId>org.apache.crail</groupId> + <artifactId>crail-client</artifactId> + <version>X.Y</version> + </dependency> + +Then, create a Crail client as follows: + +.. code-block:: Java + + CrailConfiguration conf = new CrailConfiguration(); + CrailStore store = CrailStore.newInstance(conf); + +Make sure the :code:`$CRAIL_HOME/conf` directory is part of the classpath. + +Crail supports different file types. The simplest way to create a file in Crail is as follows: + +.. code-block:: Java + + CrailFile file = store.create(filename, CrailNodeType.DATAFILE, CrailStorageClass.DEFAULT, CrailLocationClass.DEFAULT).get().syncDir(); + +Aside from the actual filename, the :code:`create()` call takes as input the storage and location classes which are preferences for the storage tier and physical location that this file should be created in. Crail tries to satisfy these preferences later when the file is written. In the example we do not request any particular storage or location affinity. + +This :code:`create()` command is non-blocking, calling :code:`get()` on the returning future object awaits the completion of the call. At that time, the file has been created, but its directory entry may not be visible. Therefore, the file may not yet show up in a file enumeration of the given parent directory. Calling :code:`syncDir()` waits to for the directory entry to be completed. Both the :code:`get()` and the :code:`syncDir()` operation can be deffered to a later time at which they may become non-blocking operations. + +Once the file is created, a file stream can be obtained for writing: + +.. code-block:: Java + + CrailBufferedOutputStream outstream = file.getBufferedOutputStream(1024); + +Here, we create a buffered stream so that we can pass heap byte arrays as well. We could also create a non-buffered stream using + +.. code-block:: Java + + CrailOutputStream outstream = file.getDirectOutputStream(1024); + +In both cases, we pass a write hint (1024 in the example) that indicates to Crail how much data we are intending to write. This allows Crail to optimize metadatanode lookups. Crail never prefetches data, but it may fetch the metadata of the very next operation concurrently with the current data operation if the write hint allows to do so. + +Once the stream has been obtained, there exist various ways to write a file. The code snippet below shows the use of the asynchronous interface: + +.. code-block:: Java + + CrailBuffer dataBuf = fs.allocateBuffer(); + Future<DataResult> future = outputStream.write(dataBuf); + ... + future.get(); + +Reading files works very similar to writing. There exist various examples in org.apache.crail.tools.CrailBenchmark. + http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/3a2c8098/doc/source/index.rst ---------------------------------------------------------------------- diff --git a/doc/source/index.rst b/doc/source/index.rst index eb8b573..3a2666e 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -31,6 +31,8 @@ Apache Crail incubating :maxdepth: 1 :caption: Development + develop + .. toctree:: :maxdepth: 1 :caption: Help
