Repository: incubator-gearpump Updated Branches: refs/heads/master 86e9709b4 -> b8556fa45
fix GEARPUMP-148, Document how to use HDFS HA for JarStore Author: manuzhang <[email protected]> Closes #22 from manuzhang/GEARPUMP-148. Project: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/commit/b8556fa4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/tree/b8556fa4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/diff/b8556fa4 Branch: refs/heads/master Commit: b8556fa45558d2a7cd8738253510f6a7e00c371e Parents: 86e9709 Author: manuzhang <[email protected]> Authored: Mon May 23 09:07:37 2016 +0800 Committer: huafengw <[email protected]> Committed: Mon May 23 09:07:37 2016 +0800 ---------------------------------------------------------------------- conf/gear.conf | 1 + core/src/main/resources/geardefault.conf | 1 + docs/deployment-configuration.md | 4 ++-- docs/deployment-ha.md | 5 +++++ .../scala/org/apache/gearpump/external/hbase/HBaseSink.scala | 8 ++++++++ 5 files changed, 17 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/b8556fa4/conf/gear.conf ---------------------------------------------------------------------- diff --git a/conf/gear.conf b/conf/gear.conf index 30eacd2..69ff213 100644 --- a/conf/gear.conf +++ b/conf/gear.conf @@ -179,6 +179,7 @@ gearpump { ### This path follows the hadoop path schema ### For HDFS, use hdfs://host:port/path/ + ### For HDFS HA, use hdfs://namespace/path/ ### For shared NFS folder, use file:///your_nfs_mapping_directory ### jarstore.rootpath = "jarstore/" will points to relative directory where master is started. ### jarstore.rootpath = "/jarstore/" will points to absolute directory on master server http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/b8556fa4/core/src/main/resources/geardefault.conf ---------------------------------------------------------------------- diff --git a/core/src/main/resources/geardefault.conf b/core/src/main/resources/geardefault.conf index 6fb4eae..bb96718 100644 --- a/core/src/main/resources/geardefault.conf +++ b/core/src/main/resources/geardefault.conf @@ -227,6 +227,7 @@ gearpump { ### This path follows the hadoop path schema ### For HDFS, use hdfs://host:port/path/ + ### For HDFS HA, use hdfs://namespace/path/ ### For shared NFS folder, use file:///your_nfs_mapping_directory ### jarstore.rootpath = "jarstore/" will points to relative directory where master is started. ### jarstore.rootpath = "/jarstore/" will points to absolute directory on master server http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/b8556fa4/docs/deployment-configuration.md ---------------------------------------------------------------------- diff --git a/docs/deployment-configuration.md b/docs/deployment-configuration.md index b651fed..b1b60db 100644 --- a/docs/deployment-configuration.md +++ b/docs/deployment-configuration.md @@ -75,7 +75,7 @@ This is the default configuration for `gear.conf`. | gearpump.appmaster.extraClasspath | "" | JVM default class path for AppMaster | | gearpump.executor.vmargs | "-server -Xss1M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 -Djava.rmi.server.hostname=localhost" | JVM arguments for executor | | gearpump.executor.extraClasspath | "" | JVM default class path for executor | -| gearpump.jarstore.rootpath | "jarstore/" | Define where the submitted jar file will be stored. This path follows the hadoop path schema. For HDFS, use `hdfs://host:port/path/`; if you want to store on master nodes, then use local directory. `jarstore.rootpath = "jarstore/"` will point to relative directory where master is started. `jarstore.rootpath = "/jarstore/"` will point to absolute directory on master server | +| gearpump.jarstore.rootpath | "jarstore/" | Define where the submitted jar file will be stored. This path follows the hadoop path schema. For HDFS, use `hdfs://host:port/path/`, and HDFS HA, `hdfs://namespace/path/`; if you want to store on master nodes, then use local directory. `jarstore.rootpath = "jarstore/"` will point to relative directory where master is started. `jarstore.rootpath = "/jarstore/"` will point to absolute directory on master server | | gearpump.scheduling.scheduler-class |"org.apache.gearpump.cluster.scheduler.PriorityScheduler" | Class to schedule the applications. | | gearpump.services.host | "127.0.0.1" | dashboard UI host address | | gearpump.services.port | 8090 | dashboard UI host port | @@ -88,4 +88,4 @@ This is the default configuration for `gear.conf`. | gearpump.netty.dispatcher | "gearpump.shared-thread-pool-dispatcher" | default dispatcher for netty client and server | | gearpump.shared-thread-pool-dispatcher | default Dispatcher with "fork-join-executor" | default shared thread pool dispatcher | | gearpump.single-thread-dispatcher | PinnedDispatcher | default single thread dispatcher | -| serialization-framework | "org.apache.gearpump.serializer.FastKryoSerializationFramework" | Gearpump has built-in serialization framework using Kryo. Users are allowed to use a different serialization framework, like Protobuf. See `org.apache.gearpump.serializer.FastKryoSerializationFramework` to find how a custom serialization framework can be defined | +| gearpump.serialization-framework | "org.apache.gearpump.serializer.FastKryoSerializationFramework" | Gearpump has built-in serialization framework using Kryo. Users are allowed to use a different serialization framework, like Protobuf. See `org.apache.gearpump.serializer.FastKryoSerializationFramework` to find how a custom serialization framework can be defined | http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/b8556fa4/docs/deployment-ha.md ---------------------------------------------------------------------- diff --git a/docs/deployment-ha.md b/docs/deployment-ha.md index 053a7bd..b3cde5e 100644 --- a/docs/deployment-ha.md +++ b/docs/deployment-ha.md @@ -38,6 +38,11 @@ In `conf/gear.conf`, For entry `gearpump.jarstore.rootpath`, please choose the s ```bash hdfs://host:port/path/ ``` + For HDFS HA, + +```bash + hdfs"//namespace/path/ +``` 2). Shared NFS folder First you need to map the NFS directory to local directory(same path) on all machines of master nodes. http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/b8556fa4/external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala ---------------------------------------------------------------------- diff --git a/external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala b/external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala index 57751a4..f3c5209 100644 --- a/external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala +++ b/external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala @@ -86,11 +86,19 @@ class HBaseSink( table.close() } + /** + * Overrides Java's default serialization + * Please do not remove this + */ private def writeObject(out: ObjectOutputStream): Unit = { out.defaultWriteObject() configuration.write(out) } + /** + * Overrides Java's default deserialization + * Please do not remove this + */ private def readObject(in: ObjectInputStream): Unit = { in.defaultReadObject() val clientConf = new Configuration(false)
