[
https://issues.apache.org/jira/browse/NIFI-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Salwerowicz updated NIFI-11927:
--------------------------------------
Description:
I have been successfully running NiFi Registry version 1.20.0 (and others
before it) for about a year now. Now I needed to switch server that Registry
runs on, since the old one was outdated, and decided to set up NiFi Registry
version 1.22.0.
The setup worked great and NiFi seems to have been running properly. However I
recently noticed that it wasn't pushing anything to GIT. The changes are made,
new versions are committed, they show up both in Registry GUI as well as the
NiFi instance, but when I try to fetch some version it will not allow me giving
me the following error:
!image-2023-08-10-08-53-15-944.png!
I started investigating the issue and noticed that the GIT repository that NiFi
Registry was using was several versions ahead of origin, no version got pushed
to the GIT repo.
I investigated the logs and noticed following messages show up when I make new
version in NiFi and commit it to registry:
{code:java}
2023-08-10 07:52:03,317 WARN [NiFi Registry Web Server-18]
javax.persistence.spi javax.persistence.spi::No valid providers found.
2023-08-10 07:52:05,120 INFO [GitFlowMetaData Push thread]
o.a.s.c.u.s.e.EdDSASecurityProviderRegistrar getOrCreateProvider(EdDSA) created
instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
2023-08-10 07:52:05,326 INFO [GitFlowMetaData Push thread]
o.a.s.c.i.DefaultIoServiceFactoryFactory No detected/configured
IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
2023-08-10 07:52:05,351 ERROR [GitFlowMetaData Push thread]
o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin due to
org.eclipse.jgit.api.errors.TransportException:
git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
org.eclipse.jgit.api.errors.TransportException:
git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:147)
at
org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.lambda$startPushThread$1(GitFlowMetaData.java:299)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.eclipse.jgit.errors.TransportException:
g...@git.nb.no:digital-bevaring/nifi/nifi-flows.git: remote hung up unexpectedly
at
org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:382)
at
org.eclipse.jgit.transport.TransportGitSsh.openPush(TransportGitSsh.java:159)
at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:127)
at org.eclipse.jgit.transport.Transport.push(Transport.java:1384)
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:137)
... 7 common frames omitted
Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
org.apache.sshd.client.future.ConnectFuture.verify()'
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:189)
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:142)
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:99)
at
org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:235)
at
org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
at
org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
at
org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:358)
... 11 common frames omitted {code}
I think the line {{Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
org.apache.sshd.client.future.ConnectFuture.verify()'}} is what's causing the
issue with pushing to GIT. This does not show up on the old Registry instance
when I push to that.
I checked the setup and compared it to the previous version running on old
server. They have exactly the same configuration, save for new one running on
Java 17 not 11 (although I tried running it on 11 too).
The SSH keys are valid and recognized by our Gitlab instance, I can push
manually as the nifi user, there are literally no differences between the
configuration on one server compared to another. I even checked the
configuration inside .git folder in the flow_storage directory, it was the same
as on the other server. The ~/.ssh folder for nifi user has similar data
between two servers, both have keys and known hosts set up, with right
permissions, etc.
I decided to try running version 1.20.0 of registry on new server, and it seems
to have worked immediately, without any issue, it manages to push to GIT on its
own, with no changes to config. I tested version 1.23 as well and it had the
same issue, I haven't tested 1.21.0 though.
For more information, here is the structure I have:
{code:java}
├── nifi-registry-1.20.0
│ ├── bin
│ ├── conf
│ ├── docs
│ ├── ext
│ ├── lib
│ ├── logs
│ ├── run
│ └── work
├── nifi-registry-1.23.0
│ ├── bin
│ ├── conf
│ ├── docs
│ ├── ext
│ ├── lib
│ ├── logs
│ ├── run
│ └── work
└── nifi-registry-files
├── authorization-files
│ ├── authorizations.xml
│ ├── authorizers.xml
│ ├── login-identity-providers.xml
│ └── users.xml
├── certificate-files
│ ├── keystore.jks
│ └── truststore.jks
├── configuration-files
│ ├── providers.xml
│ └── registry-aliases.xml
├── database-drivers
│ └── mariadb-java-client-2.7.4.jar
├── extension-bundles
└── flow-storage
└── Buckets... {code}
Contents of the providers.xml:
{code:java}
<providers>
<flowPersistenceProvider>
<class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
<property name="Flow Storage
Directory">/disk1/nifi-registry/nifi-registry-files/flow-storage</property>
<property name="Remote To Push">origin</property>
<property name="Remote Access User"></property>
<property name="Remote Access Password"></property>
<property name="Remote Clone Repository"></property>
</flowPersistenceProvider>
<extensionBundlePersistenceProvider>
<class>org.apache.nifi.registry.provider.extension.FileSystemBundlePersistenceProvider</class>
<property name="Extension Bundle Storage
Directory">/disk1/nifi-registry/nifi-registry-files/extension-bundles</property>
</extensionBundlePersistenceProvider>
</providers>{code}
Contents of the .git/config file in the flow_storage directory:
{code:java}
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[http]
sslVerify = false
proxy = http://<PROXY_URL>:3128
[https]
sslVerify = false
proxy = http://<PROXY_URL>:3128
[user]
name = nifi_user
email = OMITTED_FOR_PRIVACY
[remote "origin"]
url = git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master {code}
was:
I have been successfully running NiFi Registry version 1.20.0 (and others
before it) for about a year now. Now I needed to switch server that Registry
runs on, since the old one was outdated, and decided to set up NiFi Registry
version 1.22.0.
The setup worked great and NiFi seems to have been running properly. However I
recently noticed that it wasn't pushing anything to GIT. The changes are made,
new versions are committed, they show up both in Registry GUI as well as the
NiFi instance, but when I try to fetch some version it will not allow me giving
me the following error:
!image-2023-08-10-08-53-15-944.png!
I started investigating the issue and noticed that the GIT repository that NiFi
Registry was using was several versions ahead of origin, no version got pushed
to the GIT repo.
I investigated the logs and noticed following messages show up when I make new
version in NiFi and commit it to registry:
{code:java}
2023-08-10 07:52:03,317 WARN [NiFi Registry Web Server-18]
javax.persistence.spi javax.persistence.spi::No valid providers found.
2023-08-10 07:52:05,120 INFO [GitFlowMetaData Push thread]
o.a.s.c.u.s.e.EdDSASecurityProviderRegistrar getOrCreateProvider(EdDSA) created
instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
2023-08-10 07:52:05,326 INFO [GitFlowMetaData Push thread]
o.a.s.c.i.DefaultIoServiceFactoryFactory No detected/configured
IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
2023-08-10 07:52:05,351 ERROR [GitFlowMetaData Push thread]
o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin due to
org.eclipse.jgit.api.errors.TransportException:
git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
org.eclipse.jgit.api.errors.TransportException:
git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:147)
at
org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.lambda$startPushThread$1(GitFlowMetaData.java:299)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.eclipse.jgit.errors.TransportException:
g...@git.nb.no:digital-bevaring/nifi/nifi-flows.git: remote hung up unexpectedly
at
org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:382)
at
org.eclipse.jgit.transport.TransportGitSsh.openPush(TransportGitSsh.java:159)
at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:127)
at org.eclipse.jgit.transport.Transport.push(Transport.java:1384)
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:137)
... 7 common frames omitted
Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
org.apache.sshd.client.future.ConnectFuture.verify()'
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:189)
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:142)
at
org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:99)
at
org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:235)
at
org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
at
org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
at
org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:358)
... 11 common frames omitted {code}
I checked the setup and compared it to the previous version running on old
server. They have exactly the same configuration, save for new one running on
Java 17 not 11 (although I tried running it on 11 too).
The SSH keys are valid and recognized by our Gitlab instance, I can push
manually as the nifi user, there are literally no differences between the
configuration on one server compared to another. I even checked the
configuration inside .git folder in the flow_storage directory, it was the same
as on the other server. The ~/.ssh folder for nifi user has similar data
between two servers, both have keys and known hosts set up, with right
permissions, etc.
I decided to try running version 1.20.0 of registry on new server, and it seems
to have worked immediately, without any issue, it manages to push to GIT on its
own, with no changes to config. I tested version 1.23 as well and it had the
same issue, I haven't tested 1.21.0 though.
For more information, here is the structure I have:
{code:java}
├── nifi-registry-1.20.0
│ ├── bin
│ ├── conf
│ ├── docs
│ ├── ext
│ ├── lib
│ ├── logs
│ ├── run
│ └── work
├── nifi-registry-1.23.0
│ ├── bin
│ ├── conf
│ ├── docs
│ ├── ext
│ ├── lib
│ ├── logs
│ ├── run
│ └── work
└── nifi-registry-files
├── authorization-files
│ ├── authorizations.xml
│ ├── authorizers.xml
│ ├── login-identity-providers.xml
│ └── users.xml
├── certificate-files
│ ├── keystore.jks
│ └── truststore.jks
├── configuration-files
│ ├── providers.xml
│ └── registry-aliases.xml
├── database-drivers
│ └── mariadb-java-client-2.7.4.jar
├── extension-bundles
└── flow-storage
└── Buckets... {code}
Contents of the providers.xml
{code:java}
<providers>
<flowPersistenceProvider>
<class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
<property name="Flow Storage
Directory">/disk1/nifi-registry/nifi-registry-files/flow-storage</property>
<property name="Remote To Push">origin</property>
<property name="Remote Access User"></property>
<property name="Remote Access Password"></property>
<property name="Remote Clone Repository"></property>
</flowPersistenceProvider>
<extensionBundlePersistenceProvider>
<class>org.apache.nifi.registry.provider.extension.FileSystemBundlePersistenceProvider</class>
<property name="Extension Bundle Storage
Directory">/disk1/nifi-registry/nifi-registry-files/extension-bundles</property>
</extensionBundlePersistenceProvider>
</providers>{code}
Contents of the .git/config file in the flow_storage directory:
{code:java}
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[http]
sslVerify = false
proxy = http://<PROXY_URL>:3128
[https]
sslVerify = false
proxy = http://<PROXY_URL>:3128
[user]
name = nifi_user
email = OMITTED_FOR_PRIVACY
[remote "origin"]
url = git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master {code}
> NiFi Registry fails to push flow to GIT using SSH
> -------------------------------------------------
>
> Key: NIFI-11927
> URL: https://issues.apache.org/jira/browse/NIFI-11927
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi Registry
> Affects Versions: 1.22.0, 1.23.0
> Environment: Java 17 & 11, CentOS 9, kernel 5.14.0-44.el9.x86_64
> Reporter: Daniel Salwerowicz
> Priority: Major
> Attachments: image-2023-08-10-08-53-15-944.png
>
>
> I have been successfully running NiFi Registry version 1.20.0 (and others
> before it) for about a year now. Now I needed to switch server that Registry
> runs on, since the old one was outdated, and decided to set up NiFi Registry
> version 1.22.0.
> The setup worked great and NiFi seems to have been running properly. However
> I recently noticed that it wasn't pushing anything to GIT. The changes are
> made, new versions are committed, they show up both in Registry GUI as well
> as the NiFi instance, but when I try to fetch some version it will not allow
> me giving me the following error:
> !image-2023-08-10-08-53-15-944.png!
> I started investigating the issue and noticed that the GIT repository that
> NiFi Registry was using was several versions ahead of origin, no version got
> pushed to the GIT repo.
> I investigated the logs and noticed following messages show up when I make
> new version in NiFi and commit it to registry:
> {code:java}
> 2023-08-10 07:52:03,317 WARN [NiFi Registry Web Server-18]
> javax.persistence.spi javax.persistence.spi::No valid providers found.
> 2023-08-10 07:52:05,120 INFO [GitFlowMetaData Push thread]
> o.a.s.c.u.s.e.EdDSASecurityProviderRegistrar getOrCreateProvider(EdDSA)
> created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
> 2023-08-10 07:52:05,326 INFO [GitFlowMetaData Push thread]
> o.a.s.c.i.DefaultIoServiceFactoryFactory No detected/configured
> IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
> 2023-08-10 07:52:05,351 ERROR [GitFlowMetaData Push thread]
> o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin due to
> org.eclipse.jgit.api.errors.TransportException:
> git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
> org.eclipse.jgit.api.errors.TransportException:
> git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
> at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:147)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.lambda$startPushThread$1(GitFlowMetaData.java:299)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: org.eclipse.jgit.errors.TransportException:
> g...@git.nb.no:digital-bevaring/nifi/nifi-flows.git: remote hung up
> unexpectedly
> at
> org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:382)
> at
> org.eclipse.jgit.transport.TransportGitSsh.openPush(TransportGitSsh.java:159)
> at
> org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:127)
> at org.eclipse.jgit.transport.Transport.push(Transport.java:1384)
> at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:137)
> ... 7 common frames omitted
> Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
> org.apache.sshd.client.future.ConnectFuture.verify()'
> at
> org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:189)
> at
> org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:142)
> at
> org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:99)
> at
> org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:235)
> at
> org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
> at
> org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
> at
> org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:358)
> ... 11 common frames omitted {code}
> I think the line {{Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
> org.apache.sshd.client.future.ConnectFuture.verify()'}} is what's causing the
> issue with pushing to GIT. This does not show up on the old Registry instance
> when I push to that.
>
> I checked the setup and compared it to the previous version running on old
> server. They have exactly the same configuration, save for new one running on
> Java 17 not 11 (although I tried running it on 11 too).
>
> The SSH keys are valid and recognized by our Gitlab instance, I can push
> manually as the nifi user, there are literally no differences between the
> configuration on one server compared to another. I even checked the
> configuration inside .git folder in the flow_storage directory, it was the
> same as on the other server. The ~/.ssh folder for nifi user has similar data
> between two servers, both have keys and known hosts set up, with right
> permissions, etc.
> I decided to try running version 1.20.0 of registry on new server, and it
> seems to have worked immediately, without any issue, it manages to push to
> GIT on its own, with no changes to config. I tested version 1.23 as well and
> it had the same issue, I haven't tested 1.21.0 though.
>
> For more information, here is the structure I have:
> {code:java}
> ├── nifi-registry-1.20.0
> │ ├── bin
> │ ├── conf
> │ ├── docs
> │ ├── ext
> │ ├── lib
> │ ├── logs
> │ ├── run
> │ └── work
> ├── nifi-registry-1.23.0
> │ ├── bin
> │ ├── conf
> │ ├── docs
> │ ├── ext
> │ ├── lib
> │ ├── logs
> │ ├── run
> │ └── work
> └── nifi-registry-files
> ├── authorization-files
> │ ├── authorizations.xml
> │ ├── authorizers.xml
> │ ├── login-identity-providers.xml
> │ └── users.xml
> ├── certificate-files
> │ ├── keystore.jks
> │ └── truststore.jks
> ├── configuration-files
> │ ├── providers.xml
> │ └── registry-aliases.xml
> ├── database-drivers
> │ └── mariadb-java-client-2.7.4.jar
> ├── extension-bundles
> └── flow-storage
> └── Buckets... {code}
>
> Contents of the providers.xml:
> {code:java}
> <providers>
> <flowPersistenceProvider>
>
> <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
> <property name="Flow Storage
> Directory">/disk1/nifi-registry/nifi-registry-files/flow-storage</property>
> <property name="Remote To Push">origin</property>
> <property name="Remote Access User"></property>
> <property name="Remote Access Password"></property>
> <property name="Remote Clone Repository"></property>
> </flowPersistenceProvider>
> <extensionBundlePersistenceProvider>
>
> <class>org.apache.nifi.registry.provider.extension.FileSystemBundlePersistenceProvider</class>
> <property name="Extension Bundle Storage
> Directory">/disk1/nifi-registry/nifi-registry-files/extension-bundles</property>
> </extensionBundlePersistenceProvider>
> </providers>{code}
>
> Contents of the .git/config file in the flow_storage directory:
> {code:java}
> [core]
> repositoryformatversion = 0
> filemode = true
> bare = false
> logallrefupdates = true
> [http]
> sslVerify = false
> proxy = http://<PROXY_URL>:3128
> [https]
> sslVerify = false
> proxy = http://<PROXY_URL>:3128
> [user]
> name = nifi_user
> email = OMITTED_FOR_PRIVACY
> [remote "origin"]
> url = git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git
> fetch = +refs/heads/*:refs/remotes/origin/*
> [branch "master"]
> remote = origin
> merge = refs/heads/master {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
