Solved:
The guilty party requiring the node-{version}-headers.tar.gz was node-gyp.
In order to provide an alternate location (other than nodejs.org) to
download, the following environment variable was set:
NODEJS_ORG_MIRROR="{local location of node distributions}"
Thanks,
Mark
On Mon, Jul 19, 2021 at 4:04 PM Mark Bean <[email protected]> wrote:
> Reviving this thread for a similar question. During the build,
> nifi-registry/nifi-registry-core/nifi-registry-web-ui attempts another wget
> command.
>
> [ERROR] gyp http GET
> https://nodejs.org/download/release/v10.16.3/node-v10.16.3-headers.tar.gz
>
> For building nifi (previous to 1.14.0), we were able to resolve Node
> issues by pre-staging the required node version locally, and providing the
> following maven option on the command line:
>
> -DnodeDownloadRoot=file://{node-repo-directory}
>
> Within that repo directory, we have v10.16.3 subdirectory which contains
> node-v10.16.3-linux-x64.tar.gz (for example.) No problems there. However,
> this same methodology does not seem to work for the required
> node-v10.16.3-headers.tar.gz file.
>
> Is there a similar maven switch to use to locate the node headers file?
>
> Thanks,
> Mark
>
> On Fri, Jul 16, 2021 at 12:29 PM Mark Bean <[email protected]> wrote:
>
>> Thanks for the feedback Matt. I created a JIRA ticket to track this.
>>
>> I also created a separate ticket to create root level profiles to
>> exclude (or selectively choose) modules now part of nifi such as
>> minifi, nifi-registry, etc.
>>
>> Any debate or opinions can be entered in the tickets
>>
>> https://issues.apache.org/jira/browse/NIFI-8916
>> https://issues.apache.org/jira/browse/NIFI-8917
>>
>> Thanks,
>> Mark
>>
>>
>> On Fri, Jul 16, 2021 at 11:57 AM Matt Burgess <[email protected]>
>> wrote:
>>
>>> Mark,
>>>
>>> MiNiFi Java has been generating Windows artifacts for a while now
>>> (before it became part of NiFi), and there are quite a few users who
>>> deploy MiNiFi Java on Windows. In my experience, users deploying to
>>> small devices tend to prefer MiNiFi C++ as long as the target
>>> architecture is supported.
>>>
>>> That being said, as long as the Windows artifacts are generated on
>>> release and can be generated on-demand, I'm not opposed to making the
>>> profile inactive by default, but would like to get others' thoughts on
>>> this as well. We would want to update any READMEs, Dev guides, etc. if
>>> we disable by default any profiles that used to be enabled.
>>>
>>> Regards,
>>> Matt
>>>
>>> On Fri, Jul 16, 2021 at 11:46 AM Mark Bean <[email protected]>
>>> wrote:
>>> >
>>> > Thanks for the explanation Kevin. And, I hope the transition to the JAR
>>> > goes well.
>>> >
>>> > Meanwhile, I found another point in the build that is reaching out to
>>> an
>>> > external location. The minifi assembly attempts to download some
>>> Windows
>>> > executable (profile windows-service-assembly). This is active by
>>> default,
>>> > which all by itself seems an odd thing since I would expect minifi to
>>> run
>>> > (typically) on small devices - not Windows-based systems.
>>> >
>>> > I'd like to remove all external dependencies from the build - or at
>>> least
>>> > have a profile to exclude it (if not required) similar to
>>> -Pno-swagger-ui.
>>> > And/or, it would be useful to have profiles at the project root to
>>> exclude
>>> > any one of the additional modules which are not explicitly part of
>>> "nifi",
>>> > but are now included in the project (nifi-registry, minifi, toolkit,
>>> etc.)
>>> >
>>> > Thanks,
>>> > Mark
>>> >
>>> > On Fri, Jul 16, 2021 at 11:12 AM Kevin Doran <[email protected]>
>>> > wrote:
>>> >
>>> > > I just compared the contents of the swagger UI tarball from GitHub
>>> > > artifacts and the jar from maven central, and they definitely look
>>> like
>>> > > they both contain the same files we are rebuilding in NiFi Registry,
>>> so I
>>> > > have opened a ticket to migrate from using the download-maven-plugin
>>> > > pulling from GitHub artifacts to use the maven artifact:
>>> > >
>>> > > https://issues.apache.org/jira/browse/NIFIREG-456 <
>>> > > https://issues.apache.org/jira/browse/NIFIREG-456>
>>> > >
>>> > > Thanks again, Mark!
>>> > >
>>> > > > On Jul 16, 2021, at 10:15 AM, Kevin Doran <[email protected]
>>> >
>>> > > wrote:
>>> > > >
>>> > > > Good find! At a glance, I think this maven jar could work as a
>>> > > replacement. I’ll look into it.
>>> > > >
>>> > > > Regarding how we use the Swagger UI. When included, the NiFi
>>> Registry
>>> > > web server hosts the Swagger UI in addition to the Registry web app
>>> and
>>> > > REST API.
>>> > > >
>>> > > > I believe it is hosted at
>>> > > http(s)://<host>:<port>/nifi-registry-api/swagger/ui.html
>>> > > >
>>> > > > The Swagger UI is an alternative to the REST API documentation
>>> that is
>>> > > also included in the Registry self-hosted docs (or here [1]). One of
>>> the
>>> > > main benefits of the Swagger UI, is that in addition to documenting
>>> usage
>>> > > of the REST API, it offers interactivity. If you’ve never seen
>>> Swagger UI
>>> > > before, can see a demo of it here [2]. It is completely data driven
>>> by a
>>> > > swagger.json spec, so in Registry it dynamically generates our REST
>>> API
>>> > > endpoints.
>>> > > >
>>> > > > [1] https://nifi.apache.org/docs/nifi-registry-docs/index.html
>>> > > > [2] https://petstore.swagger.io/
>>> > > >
>>> > > >> On Jul 16, 2021, at 8:49 AM, Mark Bean <[email protected]>
>>> wrote:
>>> > > >>
>>> > > >> Thanks for laying out some clear options Kevin.
>>> > > >>
>>> > > >> I haven't dug into the inner workings of this component, and
>>> don't yet
>>> > > >> understand why a tarball is necessary. I looked at your first
>>> option.
>>> > > While
>>> > > >> I didn't find a tarball, there does appear to be
>>> org.webjars.swagger-ui
>>> > > >> JAR component in common maven repositories. I'm not sure if this
>>> is
>>> > > >> applicable - or could be made to be applicable.
>>> > > >>
>>> > > >> https://mvnrepository.com/artifact/org.webjars/swagger-ui
>>> > > >> https://search.maven.org/artifact/org.webjars/swagger-ui
>>> > > >>
>>> > > >> As a more general question, can you explain what option uses the
>>> Swagger
>>> > > >> UI? What feature or capability is not available without this
>>> component
>>> > > >> being available at build time?
>>> > > >>
>>> > > >> Thanks,
>>> > > >> Mark
>>> > > >>
>>> > > >> On Thu, Jul 15, 2021 at 4:52 PM Kevin Doran <
>>> [email protected]>
>>> > > wrote:
>>> > > >>
>>> > > >>> It’s certainly worth discussing. The binary in question is ~6MB,
>>> which
>>> > > >>> while not huge, if changed over time (to pull in new versions)
>>> could
>>> > > lead
>>> > > >>> to undesirable repository sizes. It's also debatable that it is
>>> > > “necessary”
>>> > > >>> as the bundled Swagger UI is an optional piece of NiFi Registry,
>>> which
>>> > > >>> itself is an optional component. Still we could handle it more
>>> > > gracefully.
>>> > > >>>
>>> > > >>> Options I can think of:
>>> > > >>> - See if there is an alternative Nexus / maven repository
>>> location,
>>> > > that
>>> > > >>> hosts the Swagger UI in a Jar format or something we could use
>>> as a
>>> > > Maven
>>> > > >>> dependency.
>>> > > >>> - Include the tarball in the source repo as Mark suggested
>>> > > >>> - Add an option to provide the swagger tarball as a local file,
>>> eg
>>> > > build
>>> > > >>> property to location
>>> > > >>> - Do what we do now, but detect if the tarball is not reachable
>>> via the
>>> > > >>> network and gracefully continue the build without that piece
>>> > > >>>
>>> > > >>>> On Jul 15, 2021, at 3:23 PM, Mark Bean <[email protected]>
>>> wrote:
>>> > > >>>>
>>> > > >>>> Thanks. That profile is what I needed to get beyond that point.
>>> (I ran
>>> > > >>> into
>>> > > >>>> other dependency issues I still need to resolve, so I don't yet
>>> have a
>>> > > >>>> complete build to test.)
>>> > > >>>>
>>> > > >>>> A more general question: should the wget functionality really
>>> be part
>>> > > of
>>> > > >>> a
>>> > > >>>> build? It seems to me that if a necessary component or
>>> dependency
>>> > > can't
>>> > > >>> be
>>> > > >>>> obtained from the user-configured maven repository, then that
>>> resource
>>> > > >>>> should be included in the source for the project. Comments?
>>> > > >>>>
>>> > > >>>> Thanks,
>>> > > >>>> Mark
>>> > > >>>>
>>> > > >>>> On Thu, Jul 15, 2021 at 2:11 PM Kevin Doran <
>>> [email protected]>
>>> > > >>> wrote:
>>> > > >>>>
>>> > > >>>>> Hi Mark,
>>> > > >>>>>
>>> > > >>>>> I’m not sure the two errors are related, but for the first one
>>> you
>>> > > can
>>> > > >>>>> pass -Pno-swagger-ui to mvn when building:
>>> > > >>>>>
>>> > > >>>>> See:
>>> > > >>>>>
>>> > > >>>
>>> > >
>>> https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-core/nifi-registry-web-api/pom.xml#L136
>>> > > >>>>> <
>>> > > >>>>>
>>> > > >>>
>>> > >
>>> https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-core/nifi-registry-web-api/pom.xml#L136
>>> > > >>>>>>
>>> > > >>>>>
>>> > > >>>>> Kevin
>>> > > >>>>>
>>> > > >>>>>> On Jul 15, 2021, at 1:54 PM, Mark Bean <[email protected]
>>> >
>>> > > wrote:
>>> > > >>>>>>
>>> > > >>>>>> I'm trying to build rel/nifi-1.14.0 on a private, non-Internet
>>> > > >>> connected
>>> > > >>>>>> network. The nifi-registry portion is failing.
>>> > > >>>>>>
>>> > > >>>>>> [ERROR] Failed to execute goal
>>> > > >>>>>>
>>> > > com.googlecode.maven-download-plugin:download-maven-plugin:1.2.1:wget
>>> > > >>>>>> (download-swagger-ui) on project nifi-registry-web-api: IO
>>> Error:
>>> > > Error
>>> > > >>>>>> while
>>> > > >>>>>> expalnding
>>> > > >>>>>
>>> > > >>>
>>> > >
>>> ~/nifi/nifi-registry/nifi-registry-core/nifi-registry-web-api/target/v3.12.0.tar.gz:
>>> > > >>>>>> Not in GZIP format
>>> > > >>>>>> [ERROR] Failed to execute goal
>>> > > >>>>>> com.github.eirslett:frontend-maven-plugin:1.5:npm
>>> (npm-install) on
>>> > > >>>>> project
>>> > > >>>>>> nifi-registry-web-ui: Failed to run task: 'npm --silent ci'
>>> failed.
>>> > > >>>>>> org.apache.commons.exec.ExecuteException: Process exited with
>>> an
>>> > > >>> error: 1
>>> > > >>>>>> (Exit value: 1)
>>> > > >>>>>>
>>> > > >>>>>> The two errors may be related (?) Either way, it appears to
>>> me that
>>> > > the
>>> > > >>>>>> build is attempting to use wget to reach out to an Internet
>>> > > location to
>>> > > >>>>>> download the v3.12.0.tar.gz.
>>> > > >>>>>>
>>> > > >>>>>> Can someone confirm that is the problem? Is there a way
>>> around this?
>>> > > >>> Can
>>> > > >>>>>> this file be provided from a reachable/local source location?
>>> > > >>>>>>
>>> > > >>>>>> Thanks,
>>> > > >>>>>> Mark
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>
>>> > > >>>
>>> > > >
>>> > >
>>> > >
>>>
>>
