Repository: flink Updated Branches: refs/heads/master e97f4e6df -> 17953d267
[docs] Add note about S3AFileSystem 'buffer.dir' property Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/17953d26 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/17953d26 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/17953d26 Branch: refs/heads/master Commit: 17953d267aac7a9490ae2e8e0301e860c9f53b30 Parents: e97f4e6 Author: Ufuk Celebi <[email protected]> Authored: Fri May 6 14:46:02 2016 +0200 Committer: Ufuk Celebi <[email protected]> Committed: Fri May 6 14:46:02 2016 +0200 ---------------------------------------------------------------------- docs/setup/aws.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/17953d26/docs/setup/aws.md ---------------------------------------------------------------------- diff --git a/docs/setup/aws.md b/docs/setup/aws.md index 156b4f0..ce1dcfa 100644 --- a/docs/setup/aws.md +++ b/docs/setup/aws.md @@ -95,13 +95,20 @@ There are two popular S3 file system implementations available: This is the recommended S3 FileSystem implementation to use. It uses Amazon's SDK internally and works with IAM roles (see [Configure Access Credential](#configure-access-credentials)). -You need to point Flink to a valid Hadoop configuration, which contains the following property in `core-site.xml`: +You need to point Flink to a valid Hadoop configuration, which contains the following properties in `core-site.xml`: ```xml <property> <name>fs.s3.impl</name> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> </property> + +<!-- Comma separated list of local directories used to buffer + large results prior to transmitting them to S3. --> +<property> + <name>fs.s3.buffer.dir</name> + <value>/tmp</value> +</property> ``` This registers `S3AFileSystem` as the default FileSystem for URIs with the `s3://` scheme. @@ -311,3 +318,24 @@ Caused by: org.jets3t.service.impl.rest.HttpException [...] ``` {% top %} + +### NullPointerException at org.apache.hadoop.fs.LocalDirAllocator + +This Exception is usually caused by skipping the local buffer directory configuration `fs.s3.buffer.dir` for the `S3AFileSystem`. Please refer to the [S3AFileSystem configuration](#s3afilesystem-recommended) section to see how to configure the `S3AFileSystem` properly. + +``` +[...] +Caused by: java.lang.NullPointerException at +o.a.h.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at +o.a.h.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) at +o.a.h.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:416) at +o.a.h.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:198) at +o.a.h.fs.s3a.S3AOutputStream.<init>(S3AOutputStream.java:87) at +o.a.h.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:410) at +o.a.h.fs.FileSystem.create(FileSystem.java:907) at +o.a.h.fs.FileSystem.create(FileSystem.java:888) at +o.a.h.fs.FileSystem.create(FileSystem.java:785) at +o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:404) at +o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:48) at +... 25 more +```
