So you can get into each node's UI and they each show 1/1 for cluster nodes?
It doesn't really make sense how the second node would form its own cluster. On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <alexander.s...@nih.gov> wrote: > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, > removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever > reason, the nodes still don’t talk to each other, even though both of them > are connected to ZooKeeper on host-1. I still see two separate clusters, one > on host-1 with all the dataflows, and the other, on host-2, without any of > them. On the latter, the logs have no mention of host-1 whatsoever, neither > server name, nor IP address. On host-1, nifi-app.log contains a few lines > like the following: > > > > 2018-10-23 13:44:43,628 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] > o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new > session at /<host-2 IP address>:50412 > > 2018-10-23 13:44:43,629 INFO [SyncThread:0] > o.a.zookeeper.server.ZooKeeperServer Established session 0x166a1d139590002 > with negotiated timeout 4000 for client /<host-2 IP address>:50412 > > > > I apologize for bugging you with all this, converting our standalone NiFi > instances into cluster nodes turned out to be much more challenging than we > had anticipated… > > > > -----Original Message----- > From: Bryan Bende <bbe...@gmail.com> > Sent: Tuesday, October 23, 2018 1:17 PM > To: users@nifi.apache.org > Subject: Re: NiFi fails on cluster nodes > > > > Probably easiest to copy the files over since you have other existing > users/policies and you know the first node is working. > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] > <alexander.s...@nih.gov> wrote: > > > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the > > users.xml file. Thank you so much for pointing me in the right direction! > > Now, in order to add another node, should I copy users.xml and > > authorizations.xml from the connected node to it, or remove them there > > instead? > > > > > > > > > > > > -----Original Message----- > > > From: Bryan Bende <bbe...@gmail.com> > > > Sent: Tuesday, October 23, 2018 12:36 PM > > > To: users@nifi.apache.org > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > That means the user representing host-1 does not have permissions to proxy. > > > > > > > > > > > > You can look in authorizations.xml on nifi-1 for a policy like: > > > > > > > > > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" > > > > > > resource="/proxy" action="W"> > > > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/> > > > > > > </policy> > > > > > > > > > > > > That user identifier should point to a user in users.xml like: > > > > > > > > > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234" > > > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. > > > > > > Government, C=US"/> > > > > > > > > > > > > All of the user identities are case sensitive and white space sensitive so > > make sure whatever is in users.xml is exactly what is shown in the logs. > > > > > > > > > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly > > > what we are trying to do. Here are the steps I went through in this round: > > > > > > > > > > > > > > · restored the original configuration files (nifi.properties, > > > users.xml, authorizers.xml and authorizations.xml) > > > > > > > > > > > > > > · restarted one instance in the standalone mode > > > > > > > > > > > > > > · added two new node users in the NiFi web UI (CN=<host-1, > > > redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and > > > CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, > > > C=US) > > > > > > > > > > > > > > · granted them the “proxy user requests” privileges > > > > > > > > > > > > > > · edited the nifi.properties file > > > (nifi.state.management.embedded.zookeeper.start=true, > > > nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, > > > redacted>:2181) > > > > > > > > > > > > > > · restarted the node on host-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as > > > expected, although I’m unable to do anything there due to errors like > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Insufficient Permissions > > > > > > > > > > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. > > > Government, C=US Contact the system administrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The nifi-user.log also contains > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224] > > > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: > > > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, > > > > > > > O=U.S. Government, C=US > > > > > > > > > > > > > > > > > > > > > > > > > > > > From your experience, what the most likely causes for this exception? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexander > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Bryan Bende <bbe...@gmail.com> > > > > > > > Sent: Monday, October 22, 2018 1:25 PM > > > > > > > To: users@nifi.apache.org > > > > > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, to further clarify what I meant... > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you are trying to change the Initial Admin or Node Identities in > > > authorizers.xml, these will only be used when there are no other > > > users/group/policies present. People frequently make a mistake during > > > initial config and then try to edit authorizers.xml and try again, but it > > > won't actually do anything unless you remove the users.xml and > > > authorizations.xml to start over. > > > > > > > > > > > > > > > > > > > > > > > > > > > > In your case it sounds like you are trying to convert and existing > > > standalone node to a cluster, given that I would do the following... > > > > > > > > > > > > > > > > > > > > > > > > > > > > - In standalone mode, use the UI to add users for the DN's of the > > > > > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, > > > > OU=NIFI) > > > > > > > > > > > > > > - In the UI, grant those users Write access to "Proxy" > > > > > > > > > > > > > > - Convert to a cluster and keep your same authorizers.xml, > > > > users.xml, > > > > > > > and authorizations.xml when you setup your cluster, this way all > > > > your > > > > > > > users and policies are already setup and the Initial Admin and Node > > > > > > > Identities are not needed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] > > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing > > > > users.xml and authorizations.xml mean that we will need to re-create > > > > all users and groups that we had in the original standalone NiFi > > > > instance? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > > > > From: Bryan Bende <bbe...@gmail.com> > > > > > > > > > > > > > > > Sent: Monday, October 22, 2018 12:48 PM > > > > > > > > > > > > > > > To: users@nifi.apache.org > > > > > > > > > > > > > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry I was confused when you said two 1 node clusters and I assumed > > > > they each had their own ZooKeeper. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster > > > > using the embedded ZK on the first node. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This blog post shows how to setup a secure 2 node cluster: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-au > > > > > th > > > > > > > > or > > > > > > > > > > > > > > > ization-and-multi-tenancy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The only difference is that the authorizers.xml has changed slightly, > > > > so instead of: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <authorizer> > > > > > > > > > > > > > > > <identifier>file-provider</identifier> > > > > > > > > > > > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class> > > > > > > > > > > > > > > > <property name="Authorizations > > > > > > > > File">./conf/authorizations.xml</property> > > > > > > > > > > > > > > > <property name="Users File">./conf/users.xml</property> > > > > > > > > > > > > > > > <property name="Initial Admin Identity">CN=bbende, > > > > > > > > OU=ApacheNiFi</property> > > > > > > > > > > > > > > > <property name="Legacy Authorized Users File"></property> > > > > > > > > > > > > > > > <property name="Node Identity 1">CN=localhost, > > > > > > > > OU=NIFI</property> > > > > > > > > > > > > > > > </authorizer> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You need to add the the users to the user-group-provider and then to > > > > the access-policy-provider... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <userGroupProvider> > > > > > > > > > > > > > > > <identifier>file-user-group-provider</identifier> > > > > > > > > > > > > > > > > > > > > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</class> > > > > > > > > > > > > > > > <property name="Users File">./conf/users.xml</property> > > > > > > > > > > > > > > > <property name="Legacy Authorized Users File"></property> > > > > > > > > > > > > > > > <property name="Initial User Identity 1">CN=bbende, > > > > > > > > OU=Apache NiFI</property> > > > > > > > > > > > > > > > <property name="Initial User Identity 2">CN=nifi-host-1, > > > > > > > > OU=NIFI</property> > > > > > > > > > > > > > > > <property name="Initial User Identity 2">CN=nifi-host-2, > > > > > > > > OU=NIFI</property> > > > > > > > > > > > > > > > </userGroupProvider> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <accessPolicyProvider> > > > > > > > > > > > > > > > <identifier>file-access-policy-provider</identifier> > > > > > > > > > > > > > > > > > > > > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</cla > > > > > ss > > > > > > > > > > > > > > > > > > > > > > > > <property name="User Group > > > > > > > > > > > > > > > Provider">composite-configurable-user-group-provider</property> > > > > > > > > > > > > > > > <property name="Authorizations > > > > > > > > > > > > > > > File">./conf/authorizations.xml</property> > > > > > > > > > > > > > > > <property name="Initial Admin Identity">CN=bbende, > > > > > OU=Apache > > > > > > > > NiFI</property> > > > > > > > > > > > > > > > <property name="Legacy Authorized Users File"></property> > > > > > > > > > > > > > > > <property name="Node Identity 1">CN=nifi-host-1, > > > > > > > > OU=NIFI</property> > > > > > > > > > > > > > > > <property name="Node Identity 1">CN=nifi-host-2, > > > > > > > > OU=NIFI</property> > > > > > > > > > > > > > > > </accessPolicyProvider> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also, whenever you change any config in the authorizers.xml related to > > > > the file-based providers, then you will need to remove users.xml and > > > > authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander > > > > (NIH/CC/BTRIS) [C] <alexander.s...@nih.gov> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far > > > > > as I understand, it prefers an odd number of members in the > > > > > ensemble). Actually, the ZooKeeper running on one of them, sees both > > > > > NiFi instances, but they don't talk to each other. When we try to > > > > > make them do so by using a different authorizers.xml file, which is > > > > > very much just a customized version of the “composite” example from > > > > > the NiFi Admin Guide, then none of the nodes is able to start at all, > > > > > throwing the error I mentioned in my previous post. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Are you saying that we have to run ZooKeeper on both nodes? BTW, > > > > > > > > > do > > > > > > > > > > > > > > > > we still need > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nifi.login.identity.provider.configuration.file=./conf/login-ide > > > > > > nt > > > > > > > > > it > > > > > > > > > > > > > > > > y- > > > > > > > > > > > > > > > > providers.xml > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m > > > > > asking since we have the same LDAP authentication/authorization > > > > > settings in the latter. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexander > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > > > > > From: Bryan Bende <bbe...@gmail.com> > > > > > > > > > > > > > > > > Sent: Monday, October 22, 2018 11:55 AM > > > > > > > > > > > > > > > > To: users@nifi.apache.org > > > > > > > > > > > > > > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you are getting separate clusters then each node is likely only > > > > > using it's own ZooKeeper and therefore doesn't know about the other > > > > > node. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In nifi.properties the ZK connect string would need to be something > > > > > like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in > > > > > zoo.properties you would need entries for both ZooKeepers: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > server.1=nifi-node1-hostname:2888:3888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > server.2=nifi-node2-hostname:2888:3888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder if anyone has run into the same problem when trying > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > configure composite authentication/authorization (LDAP and > > > > > > > > > > local > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > file)? When we use the “stand-alone” authorizers.xml file with > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > addition of two extra properties > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <property name="Node Identity 1">… > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <property name="Node Identity 2">… > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and let ZooKeeper start on one on the nodes, we end up with > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one-node clusters, since apparently, the NiFi instances don’t > > > > > > > > > > talk > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > each other, but at least, they come alive… > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C] > > > > > > > > > > <alexander.s...@nih.gov> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent: Friday, October 19, 2018 11:18 AM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To: users@nifi.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: RE: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We have managed to get past that error by installing the CA cert in > > > > > > the truststore. So, we can get a one-node cluster up and running. > > > > > > In order to add another node, I edited the authorizers.xml file, > > > > > > basically, using the “example composite implementation loading > > > > > > users and groups from LDAP and a local file” from the Admin guide > > > > > > as a template. When I re-started the node running ZooKeeper, > > > > > > though, it crashed with the following error written into the > > > > > > nifi-app.log file: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2018-10-19 08:09:26,992 ERROR [main] > > > > > > > > > > o.s.web.context.ContextLoader > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Context initialization failed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Error creating bean with name > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'org.springframework.security.config.annotation.web.configuration. > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > bS > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ecurityConfiguration': Unsatisfied dependency expressed > > > > > > > through > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested > > > > > > > > > > > > > > > > > exception > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > is org.springframework.beans.factory.BeanExpressionException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Expression parsing failed; nested exception is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Error creating bean with name > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': > > > > > > > > > > Unsatisfied > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dependency expressed through method 'setJwtAuthenticationProvider' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > parameter 0; nested exception is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > creating bean with name 'jwtAuthenticationProvider' defined in > > > > > > > > > > > > > > > > > class > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > reference to bean 'authorizer' while setting constructor > > > > > > > > > > argument; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nested exception is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > creating bean with name 'authorizer': FactoryBean threw > > > > > > > > > > exception > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > object creation; nested exception is java.lang.NullPointerException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Name is null > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati > > > > > > > on > > > > > > > > > > Be > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > Po > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationB > > > > > > > ea > > > > > > > > > > nP > > > > > > > > > > > > > > > > > os > > > > > > > > > > > > > > > > > tP > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > rocessor.java:667) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.annotation.InjectionMetadata > > > > > > > .i > > > > > > > > > > nj > > > > > > > > > > > > > > > > > ec > > > > > > > > > > > > > > > > > t( > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > InjectionMetadata.java:88) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati > > > > > > > on > > > > > > > > > > Be > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > Po > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanP > > > > > > > os > > > > > > > > > > tP > > > > > > > > > > > > > > > > > ro > > > > > > > > > > > > > > > > > ce > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ssor.java:366) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa > > > > > > > bl > > > > > > > > > > eB > > > > > > > > > > > > > > > > > ea > > > > > > > > > > > > > > > > > nF > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:12 > > > > > > > 64 > > > > > > > > > > ) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa > > > > > > > bl > > > > > > > > > > eB > > > > > > > > > > > > > > > > > ea > > > > > > > > > > > > > > > > > nF > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:55 > > > > > > > 3) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa > > > > > > > bl > > > > > > > > > > eB > > > > > > > > > > > > > > > > > ea > > > > > > > > > > > > > > > > > nF > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1. > > > > > > > > > > ge > > > > > > > > > > > > > > > > > tO > > > > > > > > > > > > > > > > > bj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ect(AbstractBeanFactory.java:306) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.beans.factory.support.DefaultSingletonBean > > > > > > > Re > > > > > > > > > > gi > > >
