Today, after spending some more time investigating the database for dependencies as well as looking through the live configuration of the VPC routers, i decided to change the values in the database directly. Everything seems back to normal. IP is again availeable, can be aquired and used for portforwarding and released.
Nevertheless i hope someone more qualified and skilled will find some time to take a look into this. Thanks for your help, regards Chris Am Mi., 5. Jan. 2022 um 15:31 Uhr schrieb [email protected] <[email protected]>: > @ Wei > > As suggested i wanted to trie to update the network / tier to an offering > with userdata. > Sadly i don't find a way to update the serviceoffering for an existing vpc > in the gui. Updateing the serviceoffering for the network wasn't > successfull as well. Got the error that the serviceprovider (the existing > VPC) doesn't support service user data. Therefore i couldn't update the > network as well. > So this seems to be a dead-lock. > > Regards, > Chris > > Am Di., 4. Jan. 2022 um 16:41 Uhr schrieb [email protected] <[email protected]>: > >> Hi Wei, >> >> Will have to think that through as I am not sure if I could revert back >> an service offering without Service User data. Also do you guess It could >> be sufficent to only upgrade the effected network (as the error is pointing >> at the network id)? >> Why not using userdata... Well short: not needed in the current usecase >> of the VPC as it is quiet static in regards of deploying vm's. >> Also I have to argue such decisions with our compliance team which >> basically comes to: If not needed then disable it. Especially in Our >> CUrrent usecase. >> As the GUI as well as the docs didn't suggested the actual need for this >> service we are not deploying it. >> >> >> Wei ZHOU <[email protected]> schrieb am Di., 4. Jan. 2022, 14:41: >> >>> Hi Chris, >>> >>> You can try to update the network/vpc tier to an offering with Userdata >>> and then disable static nat. >>> May I ask why you decided not to use Userdata ? >>> >>> -Wei >>> >>> >>> >>> >>> On Tue, 4 Jan 2022 at 14:35, [email protected] <[email protected]> wrote: >>> >>>> As i will need to fix this - i guess somewhat during the reset of the >>>> week >>>> are there any ideas on how to "remove" this from CS? (BackUP Jobs will >>>> be >>>> started for DB and i wouldn't have the chance to perform a binarylog >>>> rollback / recovery) >>>> >>>> The "clean" way i guess would be a DB Rollback to a point prior to >>>> enableing static NAT. >>>> Taking a look into the DB itself, i only found one entry in the table >>>> "user_ip_address": Entry with rulestate / state "Releasing" for the ip >>>> is >>>> present. one_to_one_nat = 1 and the dnat_vmip is present. >>>> Does anyone has some experience on cleaning this up directly in db? >>>> >>>> Regards, >>>> Chris >>>> >>>> Am Di., 4. Jan. 2022 um 13:16 Uhr schrieb [email protected] <[email protected] >>>> >: >>>> >>>> > Hi Wei, >>>> > >>>> > of course! :-) >>>> > >>>> > https://github.com/apache/cloudstack/issues/5824 >>>> > >>>> > Regards, >>>> > Chris >>>> > >>>> > Am Di., 4. Jan. 2022 um 11:32 Uhr schrieb Wei ZHOU < >>>> [email protected] >>>> > >: >>>> > >>>> >> Hi Chris, >>>> >> >>>> >> This seems to be a bug. Can you file an issue on >>>> >> https://github.com/apache/cloudstack/issues ? >>>> >> >>>> >> -Wei >>>> >> >>>> >> On Tue, 4 Jan 2022 at 11:16, [email protected] <[email protected]> wrote: >>>> >> >>>> >>> Hi Dan, >>>> >>> >>>> >>> Sorry for the misunderstanding. >>>> >>> The VPC was created using a "custome" service offering which is not >>>> >>> using / >>>> >>> offering "service user data" . The same goes for the NW-Offering >>>> which is >>>> >>> used for creation of the tiers. There is no service user data in >>>> this >>>> >>> VPC. >>>> >>> >>>> >>> The default / out-of-the-box VPC offering has the service "user >>>> data" >>>> >>> included. Same goes for the default nework offering for VPC Tiers. >>>> So >>>> >>> this >>>> >>> service should be supported at all from my understanding. >>>> >>> That's why this error seems a bit "bogus" to me. I was understanding >>>> >>> somethin like "static nat can't be disabled due to missing service >>>> "user >>>> >>> data" for the network". >>>> >>> >>>> >>> Deleting and building the whole thing from scratch will be the last >>>> >>> resort, >>>> >>> yes. Even taking the risk that it won't be deletable as the >>>> allocated >>>> >>> ressourcces can't be released properly at the moment, e.g. using the >>>> >>> cloudstack.api.command.user.nat.DisableStaticNatCmd. >>>> >>> Seems like i found a configuration which doesn't work well but can >>>> be >>>> >>> created via the GUI. Even if i don't understand the connection >>>> between >>>> >>> the >>>> >>> service user data and the service stativ nat. >>>> >>> >>>> >>> >>>> >>> >>>> >>> Daan Hoogland <[email protected]> schrieb am Di., 4. Jan. >>>> 2022, >>>> >>> 09:21: >>>> >>> >>>> >>> > Chris, it looks to me like you should delete the VPC and then >>>> check if >>>> >>> all >>>> >>> > IPs are available before creating a new one. >>>> >>> > >>>> >>> > As for the user data service, the message already says that user >>>> data >>>> >>> is >>>> >>> > not supported, so why create another offering? >>>> >>> > >>>> >>> > On Tue, Jan 4, 2022 at 2:50 AM [email protected] <[email protected]> >>>> wrote: >>>> >>> > >>>> >>> >> hi everyone, >>>> >>> >> >>>> >>> >> sadly i am encountering some problems in my CS 4.16 set-up. >>>> >>> >> >>>> >>> >> today i wanted to take a look into static-nat for the routers of >>>> my >>>> >>> vpc. >>>> >>> >> Sadly i missed, that the VM i wanted to assaign to the static >>>> public >>>> >>> ip >>>> >>> >> had >>>> >>> >> was already targeted by an port forwarding. >>>> >>> >> As you can imagine i got an error that static can't be applied >>>> for >>>> >>> this VM >>>> >>> >> / default interface. >>>> >>> >> >>>> >>> >> Thats when strange things happend. >>>> >>> >> The static NAT configuration was successfully applied. >>>> >>> >> I wanted to revert everything so i tried to delete the static >>>> NAT And >>>> >>> i >>>> >>> >> got >>>> >>> >> the following error: >>>> >>> >> >>>> >>> >> 2022-01-04 00:55:58,394 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) >>>> >>> (logid:90c06d44) >>>> >>> >> APPLYING VPC IP RULES >>>> >>> >> 2022-01-04 00:55:58,399 DEBUG [o.a.c.n.t.BasicNetworkTopology] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) >>>> >>> (logid:90c06d44) >>>> >>> >> Applying vpc ip association in network Ntwk[246|Guest|30] >>>> >>> >> 2022-01-04 00:55:58,442 DEBUG [c.c.a.t.Request] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) >>>> >>> (logid:90c06d44) >>>> >>> >> Seq 5-4862198747700030042: Sending { Cmd , MgmtId: >>>> 90520740759984, >>>> >>> via: >>>> >>> >> 5(srv-4-comp.dnd-digital.infra), Ver: v1, Flags: 100001, >>>> >>> >> >>>> >>> >> >>>> >>> >>>> [{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":"5","publicIp":"10.129.0.37","sourceNat":"true","add":"false","oneToOneNat":"true","firstIP":"true","broadcastUri":"vlan://2990","vlanGateway":"10.129.0.1","vlanNetmask":"255.255.255.128","vifMacAddress":"1e:00:59:00:00:b2","networkRate":"-1","trafficType":"Public","networkName":"br3_CS_Pub","newNic":"false","isPrivateGateway":"false"}],"accessDetails":{" >>>> >>> >> router.name >>>> >>> >> >>>> >>> >>>> ":"r-62-VM","router.guest.ip":"10.129.0.42","network.public.last.ip":"false","router.ip":"169.254.1.55","zone.network.type":"Advanced"},"wait":"0","bypassHostMaintenance":"false"}}] >>>> >>> >> } >>>> >>> >> 2022-01-04 00:55:59,476 DEBUG [c.c.a.t.Request] >>>> >>> >> (AgentManager-Handler-3:null) (logid:) Seq 5-4862198747700030042: >>>> >>> >> Processing: { Ans: , MgmtId: 90520740759984, via: 5, Ver: v1, >>>> Flags: >>>> >>> 0, >>>> >>> >> [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - >>>> >>> success: >>>> >>> >> Creating file in VR, with ip: 169.254.1.55, file: >>>> >>> >> ip_associations.json.167dc5ba-c89a-48c8-9b99-81b375e59691","null >>>> - >>>> >>> >> success: >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> Error: Nexthop has invalid gateway. >>>> >>> >> "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] >>>> } >>>> >>> >> 2022-01-04 00:55:59,476 DEBUG [c.c.a.t.Request] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) >>>> >>> (logid:90c06d44) >>>> >>> >> Seq 5-4862198747700030042: Received: { Ans: , MgmtId: >>>> 90520740759984, >>>> >>> >> via: >>>> >>> >> 5(srv-4-comp.dnd-digital.infra), Ver: v1, Flags: 0, { >>>> GroupAnswer } } >>>> >>> >> 2022-01-04 00:55:59,486 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> >>> Unexpected >>>> >>> >> exception while executing >>>> >>> >> org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd >>>> >>> >> com.cloud.exception.UnsupportedServiceException: Service >>>> UserData is >>>> >>> not >>>> >>> >> supported in the network id=246 >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.dao.NetworkServiceMapDaoImpl.getProviderForServiceInNetwork(NetworkServiceMapDaoImpl.java:126) >>>> >>> >> at >>>> jdk.internal.reflect.GeneratedMethodAccessor1754.invoke(Unknown >>>> >>> Source) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) >>>> >>> >> at com.sun.proxy.$Proxy61.getProviderForServiceInNetwork(Unknown >>>> >>> Source) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.NetworkModelImpl.getUserDataUpdateProvider(NetworkModelImpl.java:902) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.rules.RulesManagerImpl.applyUserData(RulesManagerImpl.java:633) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.rules.RulesManagerImpl.revokeAllPFAndStaticNatRulesForIp(RulesManagerImpl.java:1147) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1328) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1290) >>>> >>> >> at >>>> >>> >>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> >>> >> Method) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) >>>> >>> >> at com.sun.proxy.$Proxy122.disableStaticNat(Unknown Source) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd.execute(DisableStaticNatCmd.java:88) >>>> >>> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) >>>> >>> >> at >>>> >>> >> >>>> >>> >>>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) >>>> >>> >> at >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>>> >>> >> at >>>> >>> >> >>>> >>> >> >>>> >>> >>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>>> >>> >> at java.base/java.lang.Thread.run(Thread.java:829) >>>> >>> >> 2022-01-04 00:55:59,488 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Complete >>>> >>> >> async >>>> >>> >> job-2135, jobStatus: FAILED, resultCode: 530, result: >>>> >>> >> >>>> >>> >> >>>> >>> >>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Service >>>> >>> >> UserData is not supported in the network id=246"} >>>> >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Publish >>>> >>> async >>>> >>> >> job-2135 complete on message bus >>>> >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Wake up >>>> >>> jobs >>>> >>> >> related to job-2135 >>>> >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Update db >>>> >>> >> status for job-2135 >>>> >>> >> 2022-01-04 00:55:59,490 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Wake up >>>> >>> jobs >>>> >>> >> joined with job-2135 and disjoin all subjobs created from job- >>>> 2135 >>>> >>> >> 2022-01-04 00:55:59,494 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Done >>>> >>> >> executing >>>> >>> >> org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd >>>> for >>>> >>> >> job-2135 >>>> >>> >> 2022-01-04 00:55:59,494 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>> >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) >>>> Remove >>>> >>> >> job-2135 from job monitoring >>>> >>> >> >>>> >>> >> As far as i understand there is a problem with the "service user >>>> >>> data". >>>> >>> >> That's correct as i created my own offerings for VPC and the >>>> >>> Tier-Networks >>>> >>> >> which don't offer the service. >>>> >>> >> How ever now i have the situation that i can't dele the static >>>> nat >>>> >>> via the >>>> >>> >> GUI and the public ip is stuck in the state "released", is not >>>> >>> availeable >>>> >>> >> for further usage and the static NAT is still shown as >>>> configured. >>>> >>> >> >>>> >>> >> Maybe someone can point me on how to release the IP properly? >>>> May i >>>> >>> need >>>> >>> >> to >>>> >>> >> work directly in the DB to clean this "mess" ? >>>> >>> >> Any suggestion is appreciated. >>>> >>> >> >>>> >>> >> With regards, >>>> >>> >> Chris >>>> >>> >> >>>> >>> > >>>> >>> > >>>> >>> > -- >>>> >>> > Daan >>>> >>> > >>>> >>> >>>> >> >>>> >>>
