Kevin,
Sounds sensible to me, thanks!
---
*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com
On Fri, 5 Nov 2021 at 14:10, Kevin Doran <kdo...@apache.org> wrote:
> Hi Chris,
>
> Thanks for the summary of your findings.
>
> I agree that we should clear up our process for generating Docker images
> and that the process should be consistent as possible across NiFi, MiNiFi
> Java, Registry, and Toolkit given they are all part of the same repository
> and maven project. This will clear up confusion and improve
> maintainability.
>
> I think both of these methods are important and useful:
>
> 1. building from downloaded convenience binaries for previously released
> versions
> 2. building from a local assembly output
>
> That said, I think we could probably consolidate to a single Dockerfile
> that takes the binary location as an argument that supports either a URL or
> local file path (or a version which could default to the current behavior
> of inferring a URL location). This would let us continue to support the
> flexibility we have today while maintaining a single Docker file rather
> than dockermaven and dockerhub variants. If you and others on the list
> agree, I can open up a Jira summarizing what needs to be done.
>
> Thanks,
> Kevin
>
> > On Nov 5, 2021, at 05:00, Chris Sampson <chris.samp...@naimuri.com.INVALID>
> wrote:
> >
> > So the good news is that I realised what I was doing wrong yesterday -
> I'd
> > started a local installation after building the RC, then not shut that
> down
> > before booting up the Docker instance, so the username/password I was
> > trying to use from the Docker logs was wrong against the local
> > installation, d'oh!
> >
> > Having corrected that this morning, I've successfully booted up and
> logged
> > into a Docker instance built using the RC (with my NIFI-8779 changes so I
> > could build from the Dev server instead of the main Apache Archive).
> >
> > The reason the dockermaven build works is that it uses the locally built
> > archive files (i.e. you have to do a full Maven build locally first to
> > generate the ZIP files and then create teh Docker image). The
> > dockerhub build actually downloads published artifacts from the Apache
> > servers - to do this the Dockerfile needs to know the exact path to the
> > artifacts it's trying to download, of course.
> >
> > There was a question recently in one of the Slack channels about whether
> > both of these builds (dockerhub and dockermaven) were still valid/needed
> -
> > I think Joe replied that he wasn't sure (Docker not being an area he
> > ventures into much, which is fair enough). It may be worth considering
> > whether these two modules are both still needed or can be rationalised
> and
> > if the latter, which approach should be used - download from the archive
> > server or build from local (both have dis/advantages, but the more
> > "official" way is arguably to download from the server).
> >
> > Also maybe worth noting:
> > * NiFi Registry only has the "build from local" Dockerfile setup
> > * MiniFi (Java) has both local and archive download options, but the
> latter
> > cannot be used to build an image from the Dev server (the BASE_URL cannot
> > be changed via a build arg/env var... this is the issue addressed by
> > NIFI-8779 for NiFi)
> > * NiFi Toolkit has both local and archive download options, but are
> located
> > in a single assembly module (instead of split into two like NiFi and
> MiniFi)
> >
> > Assuming the "nifi/<version>" path will be used for the actual release
> > artifacts, this shouldn't be a blocker, but it's one to note/remember
> when
> > the time comes (assuming the "dockerhub" module is what's used to publish
> > the "apache/nifi" image to Docker Hub that is).
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > On Fri, 5 Nov 2021 at 03:34, David Handermann <
> exceptionfact...@apache.org>
> > wrote:
> >
> >> Chris,
> >>
> >> I am running through several verification steps, but I built 1.15.0 RC 3
> >> from sources and then ran "mvn install -pl :dockermaven -P docker" to
> build
> >> the Docker image. When starting with the following command, I was able
> to
> >> use the generated username and password to access the running NiFi
> >> container:
> >>
> >> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
> >>
> >> If you are still having issues, it would be helpful to provide any logs
> or
> >> error messages you are seeing.
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kdoran.apa...@gmail.com>
> >> wrote:
> >>
> >>> Oh meant to send a link to this too:
> >>>
> >>>> ARG
> >>>
> >>
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> >>>
> >>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kdoran.apa...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi Joe and Chris,
> >>>>
> >>>> Our Dockerfile that we use to build the Dockerhub image defaults to
> >>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
> >> can
> >>> override, so this is easy to workaround incase the release folder does
> >>> change. Agree its nice to keep the tree structure consistent when we
> can.
> >>>>
> >>>> I’m about to do my verification and will also check the single user
> >> with
> >>> the docker image as part of that.
> >>>>
> >>>> Cheers,
> >>>> Kevin
> >>>>
> >>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >>>>>
> >>>>> Chris,
> >>>>>
> >>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >>>>> Should generally not really matter so the docker angle there is
> >>>>> interesting. Not sure why our docker images would have any
> >>>>> relationship to our dist/dev storage. But when I move them into the
> >>>>> release folder I can try to ensure I place them only in 1.15.0
> instead
> >>>>> of nifi/1.15.0.
> >>>>>
> >>>>> That directory the prov stuff makes does linger and can be annoying
> so
> >>>>> thanks for tackling that. Saw the PR. Will take a look soon if
> >>>>> nobody else has.
> >>>>>
> >>>>> Will keep an eye on your findings related to docker builds not
> working
> >>>>> with username/password things. Hopefully others can chime in there.
> >>>>>
> >>>>> Thanks
> >>>>> Send
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >>>>> <chris.samp...@naimuri.com.invalid> wrote:
> >>>>>>
> >>>>>> Worryingly, when I do get the Docker Image to build (further changes
> >>> to the
> >>>>>> Dockerfile), the auto-generated username and password from the
> >> startup
> >>> logs
> >>>>>> aren't being accepted for login via my browser.
> >>>>>>
> >>>>>> I'll try to spend a little more time looking at this (but await
> input
> >>> on my
> >>>>>> earlier question/concern also).
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> *Chris Sampson*
> >>>>>> IT Consultant
> >>>>>> chris.samp...@naimuri.com
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
> >> chris.samp...@naimuri.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I've got most of the way through the release check process in order
> >> to
> >>>>>>> vote for 1.15.0, but I wanted to check on a change in the
> >> distribution
> >>>>>>> release artifacts.
> >>>>>>>
> >>>>>>> For 1.14.0, the Dev artifacts were located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>>>>> For 1.15.0, the Dev artifacts are located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>>>>
> >>>>>>> i.e. there's been a change of directory/path from <version> to
> >>>>>>> nifi-<version>.
> >>>>>>>
> >>>>>>> The reason I raise this is that I can no longer build a Docker
> Image
> >>> using
> >>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
> >>> artifacts to
> >>>>>>> download. This may not be a problem if the path change is only for
> >>> the Dev
> >>>>>>> artifacts, but if the same change is going to happen for the
> >> released
> >>>>>>> artifacts, then the apache/nifi image (and presumably the
> >>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
> >> convenience
> >>>>>>> images will no longer be possible as part of the Release, which
> will
> >>> likely
> >>>>>>> be an issue for a number of users that have deployments using these
> >>> images.
> >>>>>>>
> >>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
> >> [2]
> >>>>>>>
> >>>>>>>
> >>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> >>>>>>> created by the unit tests executed during building and testing
> >>> 1.15.0-RC3.
> >>>>>>> I raised a PR [4] - this is a minor issue and not a reason to block
> >>> any
> >>>>>>> release.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/nifi/pull/5213
> >>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> >>>>>>>
> >>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> >>>>>>> [4] https://github.com/apache/nifi/pull/5510
> >>>>>>>
> >>>>>>> ---
> >>>>>>> *Chris Sampson*
> >>>>>>> IT Consultant
> >>>>>>> chris.samp...@naimuri.com
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <joew...@apache.org> wrote:
> >>>>>>>
> >>>>>>>> Otto
> >>>>>>>>
> >>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
> >> with
> >>> every
> >>>>>>>> release what we are actually voting upon is the source release.
> >> Now
> >>> the
> >>>>>>>> source release includes nifi, minifi, nifi registry, stateless
> nifi
> >>> and
> >>>>>>>> toolkits among all the other having always been there goodies.
> >>>>>>>>
> >>>>>>>> Some of these things we make available in the form of convenience
> >>> binaries
> >>>>>>>> to make it easier on folks to consume.
> >>>>>>>>
> >>>>>>>> I think you dont need to do any verification you would not have
> >> done
> >>>>>>>> before.
> >>>>>>>>
> >>>>>>>> But I do hope folks help maintain various ways of easing more
> folks
> >>>>>>>> knowing
> >>>>>>>> what to vet with a given release
> >>>>>>>>
> >>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
> >> ottobackwa...@gmail.com
> >>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
> >> Java.
> >>> At
> >>>>>>>>> least that is my understanding.
> >>>>>>>>>
> >>>>>>>>> This would be my first time having *anything* to do with minifi,
> >>> i’ve
> >>>>>>>> not
> >>>>>>>>> even run it before.
> >>>>>>>>>
> >>>>>>>>> As such, I think the RC validation guide needs to be update to
> >>> include
> >>>>>>>> the
> >>>>>>>>> information about now validating nifi and minifi together.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> From: Joe Witt <joew...@apache.org> <joew...@apache.org>
> >>>>>>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> <
> >>> dev@nifi.apache.org>
> >>>>>>>>> Date: October 25, 2021 at 11:59:39
> >>>>>>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> <
> >> dev@nifi.apache.org>
> >>>>>>>>> Subject: [discuss] NiFi 1.15
> >>>>>>>>>
> >>>>>>>>> Team,
> >>>>>>>>>
> >>>>>>>>> I thought I had already started such a thread but I dont see it
> so
> >>> here
> >>>>>>>>> goes...
> >>>>>>>>>
> >>>>>>>>> We have made a ton of progress again and there are some super
> >>>>>>>>> important fixes as well. It is definitely time to kick out a
> 1.15.
> >>>>>>>>> My intent will be to attempt to pull together an RC this week.
> >>>>>>>>> Haven't done the analysis yet of what is hanging out there but
> >> will
> >>> do
> >>>>>>>>> so. A quick look at all the features and fixes already landed
> >> though
> >>>>>>>>> make it clear we have more than enough to work with.
> >>>>>>>>>
> >>>>>>>>> Lets please use this thread for coordination on the RC rather
> than
> >>> it
> >>>>>>>>> becoming the wish list. We have new features/fixes arriving all
> >> the
> >>>>>>>>> time - those can be addressed in normal channels.
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>>
> >>
>
>
