Thanks, I'll take a look at the wrapper init script.
On 6/28/18, 15:11, "Aldrin Piri" <[email protected]> wrote:
I don't know that it necessarily needs to be changed but the convention we
have in other images where we make use of a wrapper init script gives a
little more flexibility for investigating/manipulating instance internals.
That convention would have made things a bit more straightforward for this
scenario. In terms of the signals handling, I was a bit surprised they
weren't being picked up by Java but would need to investigate a bit more to
see if it was just my hastiness or if there is something else at play.
On Thu, Jun 28, 2018 at 2:33 PM Kevin Doran <[email protected]> wrote:
> Thanks Aldrin! That sequence makes sense to me and should result in a
> clean copy of the H2 Database. Mike, you're probably aware of this, but
you
> will also want to grab the files in the HOME/flow_storage directory
> assuming those were left I nthe default settings and are not on your
Docker
> volume.
>
> Aldrin, is there anything you see regarding the Registry Docker image that
> could be improved, for which we could open a JIRA? Just curious to get
your
> insight while it is fresh in your mind.
>
> Thanks!
> Kevin
>
> On 6/28/18, 14:24, "Aldrin Piri" <[email protected]> wrote:
>
> Hi Mike,
>
> Registry's docker image is a little off from some of the other NiFi
> project
> configurations but the following approach should be fine and is fairly
> conservative in how it is carried out.
>
> 1. Exec into the container - *docker exec -it <registry container
> name/id> /bin/bash*
> 2. Inside container: stop Registry, and have your interactive
> session
> terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
> 3. On your host, copy the h2 file to your host - *docker cp
> <registry
> container name/id>
> **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
> <path on host>*
> 4. Copy the database file in question to your volume for your new
> instance
> 5. Start the new container to verify import/migration
> 6. Cleanup the old container and files as appropriate
>
> 1 & 2 are questionable in terms of their necessity but with the
> potential
> for the container to get sigkilled, lend to a safer approach. The
Java
> process doesn't seem to handle the sigterm from my quick testing of
> docker
> stop.
>
> You have the ability to restart where you left off on the original
> container via *docker start **<registry container name/id>*.
>
> I did a quick run through of these steps with vanilla versions of
> 0.1.0 and
> 0.2.0 and it seemed to work as anticipated.
>
> Let us know how it goes.
>
> On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <[email protected]> wrote:
>
> > Oh right. My mistake. Not sure if there is a way to stop the nifi
> registry
> > process without stopping the container. Other's more knowledgeable
of
> > Docker might know... In that case I'd say it's worth a shot as that
> DB file
> > should be valid so long as nothing is being written to it.
> >
> > The adoption of the git persistence provider stuff is probably still
> small
> > given how new it is, but anecdotally, I've been using it for about a
> week
> > now, syncing to a github repo as a remote, and I've not run into any
> issues
> > with it nor have I heard of anyone else having a problem with it. It
> seems
> > pretty solid to me. One of the nice things about the git option is
> that the
> > format saved to git in the snapshots is compatible with a tool like
> NiFi
> > CLI, so (while some custom scripting still required), you should be
> in a
> > pretty good position to do disaster recovery to recreate a NiFi
> Registry
> > instance with your bucket/flow/flow-snapshot data just from the
> contents of
> > your git repository should you need to for whatever reason (again,
> would
> > need to script the right NiFi CLI commands to crawl the git
> directory tree,
> > but it's pretty straightforward as it's just a matter of parsing
> json and
> > yml files). One really nice feature is that deletes (a flow or
> bucket for
> > instance) happen as a new commit so you can even recover objects
> that were
> > deleted from NiFi Registry by accident, for instance.
> >
> > If your team/users are using NiFi Registry and have feedback,
> positive or
> > otherwise, let us know as it would be great to factor that in to the
> > existing backlog for what to target in future releases.
> >
> > Best,
> > Kevin
> >
> > If your team runs into any trouble with this, do let us know as I'm
> not
> > sure the deployment footprint
> >
> > On 6/28/18, 13:31, "Mike Thomsen" <[email protected]> wrote:
> >
> > Kevin,
> >
> > The problem with our situation is that it's all inside Docker so
> I
> > don't
> > think I can just stop the registry and proceed. That's why I was
> > asking if
> > it sounded feasible to just say "hands off for 5 minutes while
> we copy
> > the
> > data."
> >
> > Also, what is the maturity level of the GitFlow option at this
> point?
> > That
> > was particularly attractive as a storage solution because it
> should
> > make it
> > much easier to export everything from our dev team to our end
> users.
> >
> > Thanks,
> >
> > Mike
> >
> > On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[email protected]>
> wrote:
> >
> > > Mike,
> > >
> > > I don't know the extent of H2's capabilities, but for NiFi
> Registry
> > > changing the location of those files will require restarting
> the
> > server to
> > > pick up the new database location from
> nifi-registry.properties,
> > correct?
> > > So if I follow correctly what the objective is, my
> recommendation
> > would be
> > > the following steps:
> > >
> > > 1. Stop NiFi Registry
> > > 2. Move/copy the H2 database files to the Docker volume
> location
> > (should
> > > be safe to do when Registry is stopped)
> > > 3. Change nifi-registry.properties to point the database
config
> > props to
> > > the new location
> > > 4. Start NiFi Registry
> > >
> > > If the goal is actually just to copy a snapshot of the H2
> database
> > to the
> > > docker volume and not re-point NiFi Registry, which you want
> to keep
> > > running, I suppose you could do that. It should be fine to
> copy it if
> > > nothing is being written, but of course no guarantees and
> YMMV, so
> > at the
> > > least I'd try to verify that it is a validate db file after
> copying
> > it,
> > > using something like [1].
> > >
> > > [1]
> > >
> >
>
https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
> > >
> > > Hope this helps,
> > > Kevin
> > >
> > > On 6/28/18, 12:34, "Mike Thomsen" <[email protected]>
> wrote:
> > >
> > > Kevin/Bryan,
> > >
> > > We have a bunch of users that are using Docker and might
> not
> > have set
> > > things up to point to a docker volume for the H2 database
> > location. Do
> > > either of you know if H2 will allow you to copy the files
> over
> > to the
> > > Docker volume while everything's running?
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > > On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <
> [email protected]>
> > wrote:
> > >
> > > > Ah good point Mark... yes the old db properties are only
> need
> > this
> > > > first time so that it can auto-migrate the old DB to the
> new
> > one,
> > > > after that you don't need the old properties anymore.
> > > >
> > > > Thanks Kevin!
> > > >
> > > > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
> > [email protected]>
> > > wrote:
> > > > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1
> -> 0.2
> > > migration
> > > > guide to include these steps as part of updating the
> site with
> > news
> > > of the
> > > > new release.
> > > > >
> > > > > Thanks,
> > > > > Kevin
> > > > >
> > > > > ________________________________
> > > > > From: Mark Bean <[email protected]>
> > > > > Sent: Wednesday, June 20, 2018 8:53:39 AM
> > > > > To: [email protected]
> > > > > Subject: Re: Upgrading NiFi Registry
> > > > >
> > > > > Thanks Bryan. There is actually another step not
> explicitly
> > > mentioned. At
> > > > > least for 0.1.0 -> 0.2.0, I needed to modify the
> > > nifi-registry.properties
> > > > > file as well. The 0.2.0 version has new
> properties/values
> > not in
> > > the
> > > > 0.1.0.
> > > > > And, I had to set the following for the database
(using
> > values from
> > > > 0.1.0).
> > > > > This was only required on the first startup; these
> > properties can
> > > be
> > > > empty
> > > > > values on subsequent startups.
> > > > > nifi.registry.db.directory=
> > > > > nifi.registry.db.url.append=
> > > > >
> > > > > In the future, if using a database location that is
> external
> > to the
> > > > > installation directory, is nifi.registry.db.url the
> only
> > property
> > > that
> > > > > needs to be modified?
> > > > >
> > > > >
> > > > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
> > [email protected]>
> > > wrote:
> > > > >
> > > > >> Mark,
> > > > >>
> > > > >> The database directory and flow storage directory are
> where
> > all
> > > the
> > > > >> data are. By default these are created in the root of
> NiFi
> > > Registry,
> > > > >> so depending how you want to set it up you could move
> those
> > > > >> directories to the new install, or you could set them
> up to
> > be
> > > > >> external locations so you don't have to move them
> every
> > time, or
> > > you
> > > > >> could upgrade the lib directory of your current
> install and
> > leave
> > > > >> everything in place.
> > > > >>
> > > > >> The policies are the same as NiFi... stored in
> users.xml and
> > > > >> authorizations.xml the conf directory, depending how
> you
> > > configured
> > > > >> everything. So just copying those two files over to
> the new
> > > install.
> > > > >>
> > > > >> -Bryan
> > > > >>
> > > > >>
> > > > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
> > > [email protected]>
> > > > wrote:
> > > > >> > How does one upgrade the NiFi Registry?
> > > > >> >
> > > > >> > After unpacking the .tar.gz file, how does one get
> all
> > the flows
> > > > >> registered
> > > > >> > in a previous version of NiFi Registry into the
> newly
> > installed
> > > > version?
> > > > >> > And, how does one ensure all the policies transfer
> as
> > well?
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Mark
> > > > >>
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
