I don't know that it necessarily needs to be changed but the convention we have in other images where we make use of a wrapper init script gives a little more flexibility for investigating/manipulating instance internals. That convention would have made things a bit more straightforward for this scenario. In terms of the signals handling, I was a bit surprised they weren't being picked up by Java but would need to investigate a bit more to see if it was just my hastiness or if there is something else at play.
On Thu, Jun 28, 2018 at 2:33 PM Kevin Doran <kdo...@apache.org> wrote: > Thanks Aldrin! That sequence makes sense to me and should result in a > clean copy of the H2 Database. Mike, you're probably aware of this, but you > will also want to grab the files in the HOME/flow_storage directory > assuming those were left I nthe default settings and are not on your Docker > volume. > > Aldrin, is there anything you see regarding the Registry Docker image that > could be improved, for which we could open a JIRA? Just curious to get your > insight while it is fresh in your mind. > > Thanks! > Kevin > > On 6/28/18, 14:24, "Aldrin Piri" <aldrinp...@gmail.com> wrote: > > Hi Mike, > > Registry's docker image is a little off from some of the other NiFi > project > configurations but the following approach should be fine and is fairly > conservative in how it is carried out. > > 1. Exec into the container - *docker exec -it <registry container > name/id> /bin/bash* > 2. Inside container: stop Registry, and have your interactive > session > terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop* > 3. On your host, copy the h2 file to your host - *docker cp > <registry > container name/id> > **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db > <path on host>* > 4. Copy the database file in question to your volume for your new > instance > 5. Start the new container to verify import/migration > 6. Cleanup the old container and files as appropriate > > 1 & 2 are questionable in terms of their necessity but with the > potential > for the container to get sigkilled, lend to a safer approach. The Java > process doesn't seem to handle the sigterm from my quick testing of > docker > stop. > > You have the ability to restart where you left off on the original > container via *docker start **<registry container name/id>*. > > I did a quick run through of these steps with vanilla versions of > 0.1.0 and > 0.2.0 and it seemed to work as anticipated. > > Let us know how it goes. > > On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <kdo...@apache.org> wrote: > > > Oh right. My mistake. Not sure if there is a way to stop the nifi > registry > > process without stopping the container. Other's more knowledgeable of > > Docker might know... In that case I'd say it's worth a shot as that > DB file > > should be valid so long as nothing is being written to it. > > > > The adoption of the git persistence provider stuff is probably still > small > > given how new it is, but anecdotally, I've been using it for about a > week > > now, syncing to a github repo as a remote, and I've not run into any > issues > > with it nor have I heard of anyone else having a problem with it. It > seems > > pretty solid to me. One of the nice things about the git option is > that the > > format saved to git in the snapshots is compatible with a tool like > NiFi > > CLI, so (while some custom scripting still required), you should be > in a > > pretty good position to do disaster recovery to recreate a NiFi > Registry > > instance with your bucket/flow/flow-snapshot data just from the > contents of > > your git repository should you need to for whatever reason (again, > would > > need to script the right NiFi CLI commands to crawl the git > directory tree, > > but it's pretty straightforward as it's just a matter of parsing > json and > > yml files). One really nice feature is that deletes (a flow or > bucket for > > instance) happen as a new commit so you can even recover objects > that were > > deleted from NiFi Registry by accident, for instance. > > > > If your team/users are using NiFi Registry and have feedback, > positive or > > otherwise, let us know as it would be great to factor that in to the > > existing backlog for what to target in future releases. > > > > Best, > > Kevin > > > > If your team runs into any trouble with this, do let us know as I'm > not > > sure the deployment footprint > > > > On 6/28/18, 13:31, "Mike Thomsen" <mikerthom...@gmail.com> wrote: > > > > Kevin, > > > > The problem with our situation is that it's all inside Docker so > I > > don't > > think I can just stop the registry and proceed. That's why I was > > asking if > > it sounded feasible to just say "hands off for 5 minutes while > we copy > > the > > data." > > > > Also, what is the maturity level of the GitFlow option at this > point? > > That > > was particularly attractive as a storage solution because it > should > > make it > > much easier to export everything from our dev team to our end > users. > > > > Thanks, > > > > Mike > > > > On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <kdo...@apache.org> > wrote: > > > > > Mike, > > > > > > I don't know the extent of H2's capabilities, but for NiFi > Registry > > > changing the location of those files will require restarting > the > > server to > > > pick up the new database location from > nifi-registry.properties, > > correct? > > > So if I follow correctly what the objective is, my > recommendation > > would be > > > the following steps: > > > > > > 1. Stop NiFi Registry > > > 2. Move/copy the H2 database files to the Docker volume > location > > (should > > > be safe to do when Registry is stopped) > > > 3. Change nifi-registry.properties to point the database config > > props to > > > the new location > > > 4. Start NiFi Registry > > > > > > If the goal is actually just to copy a snapshot of the H2 > database > > to the > > > docker volume and not re-point NiFi Registry, which you want > to keep > > > running, I suppose you could do that. It should be fine to > copy it if > > > nothing is being written, but of course no guarantees and > YMMV, so > > at the > > > least I'd try to verify that it is a validate db file after > copying > > it, > > > using something like [1]. > > > > > > [1] > > > > > > https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption > > > > > > Hope this helps, > > > Kevin > > > > > > On 6/28/18, 12:34, "Mike Thomsen" <mikerthom...@gmail.com> > wrote: > > > > > > Kevin/Bryan, > > > > > > We have a bunch of users that are using Docker and might > not > > have set > > > things up to point to a docker volume for the H2 database > > location. Do > > > either of you know if H2 will allow you to copy the files > over > > to the > > > Docker volume while everything's running? > > > > > > Thanks, > > > > > > Mike > > > > > > On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende < > bbe...@gmail.com> > > wrote: > > > > > > > Ah good point Mark... yes the old db properties are only > need > > this > > > > first time so that it can auto-migrate the old DB to the > new > > one, > > > > after that you don't need the old properties anymore. > > > > > > > > Thanks Kevin! > > > > > > > > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran < > > kdo...@apache.org> > > > wrote: > > > > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 > -> 0.2 > > > migration > > > > guide to include these steps as part of updating the > site with > > news > > > of the > > > > new release. > > > > > > > > > > Thanks, > > > > > Kevin > > > > > > > > > > ________________________________ > > > > > From: Mark Bean <mark.o.b...@gmail.com> > > > > > Sent: Wednesday, June 20, 2018 8:53:39 AM > > > > > To: dev@nifi.apache.org > > > > > Subject: Re: Upgrading NiFi Registry > > > > > > > > > > Thanks Bryan. There is actually another step not > explicitly > > > mentioned. At > > > > > least for 0.1.0 -> 0.2.0, I needed to modify the > > > nifi-registry.properties > > > > > file as well. The 0.2.0 version has new > properties/values > > not in > > > the > > > > 0.1.0. > > > > > And, I had to set the following for the database (using > > values from > > > > 0.1.0). > > > > > This was only required on the first startup; these > > properties can > > > be > > > > empty > > > > > values on subsequent startups. > > > > > nifi.registry.db.directory= > > > > > nifi.registry.db.url.append= > > > > > > > > > > In the future, if using a database location that is > external > > to the > > > > > installation directory, is nifi.registry.db.url the > only > > property > > > that > > > > > needs to be modified? > > > > > > > > > > > > > > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende < > > bbe...@gmail.com> > > > wrote: > > > > > > > > > >> Mark, > > > > >> > > > > >> The database directory and flow storage directory are > where > > all > > > the > > > > >> data are. By default these are created in the root of > NiFi > > > Registry, > > > > >> so depending how you want to set it up you could move > those > > > > >> directories to the new install, or you could set them > up to > > be > > > > >> external locations so you don't have to move them > every > > time, or > > > you > > > > >> could upgrade the lib directory of your current > install and > > leave > > > > >> everything in place. > > > > >> > > > > >> The policies are the same as NiFi... stored in > users.xml and > > > > >> authorizations.xml the conf directory, depending how > you > > > configured > > > > >> everything. So just copying those two files over to > the new > > > install. > > > > >> > > > > >> -Bryan > > > > >> > > > > >> > > > > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean < > > > mark.o.b...@gmail.com> > > > > wrote: > > > > >> > How does one upgrade the NiFi Registry? > > > > >> > > > > > >> > After unpacking the .tar.gz file, how does one get > all > > the flows > > > > >> registered > > > > >> > in a previous version of NiFi Registry into the > newly > > installed > > > > version? > > > > >> > And, how does one ensure all the policies transfer > as > > well? > > > > >> > > > > > >> > Thanks, > > > > >> > Mark > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
