Github user revans2 commented on a diff in the pull request:

    https://github.com/apache/storm/pull/2347#discussion_r141182693
  
    --- Diff: external/storm-blobstore-migration/README.md ---
    @@ -0,0 +1,87 @@
    +# Blobstore Migrator
    +
    +## Basic Use
    +-----
    +
    +### Build The Thing
    +Use make to build a tarball with everything needed.
    +```
    +$ make
    +```
    +
    +### Use The Thing
    +Copy and extract the tarball
    +```
    +$ scp blobstore-migrator.tgz my-nimbus-host.example.com:~/
    +$ ssh my-nimbus-host.example.com
    +... On my-nimbus-host ...
    +$ tar -xvzf blobstore-migrator.tgz
    +```
    +
    +This will expand into a blobstore-migrator directory with all the scripts 
and the jar.
    +```
    +$ cd blobstore-migrator
    +$ ls
    +blobstore-migrator-2.0.jar listHDFS.sh  listLocal.sh  migrate.sh
    +```
    +
    +To run, first create a config for the cluster.
    +The config must be named 'config'
    +It must contain definitions for `HDFS_BLOBSTORE_DIR`, 
`LOCAL_BLOBSTORE_DIR`, and `HADOOP_CLASSPATH`.
    +Hadoop jars are packaged with neither storm nor this package, so they must 
be installed separately.
    +
    +Optional configs used to configure security are: `BLOBSTORE_PRINCIPAL`, 
`KEYTAB_FILE`, and `JAAS_CONF`
    +
    +Example:
    +```
    +$ cat config
    
+HDFS_BLOBSTORE_DIR='hdfs://some-hdfs-namenode:8080/srv/storm/my-storm-blobstore'
    +LOCAL_BLOBSTORE_DIR='/srv/storm'
    +HADOOP_CLASSPATH='/hadoop/share/hdfs/*:/hadoop/common/*'
    +
    +# My security configs: 
    
+BLOBSTORE_PRINCIPAL='stormUser/[email protected]'
    +KEYTAB_FILE='/srv/my-keytab/stormUser.kt'
    +JAAS_CONF='/storm/conf/storm_jaas.conf'
    +```
    +
    +Now you can run any of the scripts, all of which require config to exist:
    + - listHDFS.sh: lists all blobs currently in the HDFS Blobstore
    + - listLocal.sh: lists all blobs currently in the local Blobstore
    + - migrate.sh: Begins the migration process for Nimbus. (Read instructions 
below first)
    + 
    + 
    +#### Migrating
    +##### Nimbus
    +To migrate blobs from nimbus, the following steps are necessary:
    +
    +1. Shut down Nimbus
    --- End diff --
    
    Can we make this shut down all nimbus instances, because of HA we cannot 
leave any up.


---

Reply via email to