Hi Brad
Could you please check the cloudstack management server logs (/var/log/cloudstack/management/management-server.log) to see if there is any exception. Also can you check directly via the api https://cloudstack.apache.org/api/apidocs-4.21/apis/createPortForwardingRule.html Regards Kiran From: Brad House <[email protected]> Date: Wednesday, 1 October 2025 at 11:15 PM To: [email protected] <[email protected]> Subject: Re: [VOTE] Release Apache CloudStack Terraform Provider v0.6.0 RC-2 that's not set at all ... so it should be the default 300, no? But its having an issue at 30s. By the way, I have another cluster I just spun up on different hardware, but otherwise identical and it too has the same problem. Both are running 4.21.0 I'm not familiar enough with the internals here, what would I need to do to investigate this further? On 10/1/25 12:50 PM, Kiran Chavala wrote: > Hi Brad > > > > Can you try using the timeout value in provider configuration > > https://registry.terraform.io/providers/cloudstack/cloudstack/latest/docs > > > A value in seconds. This is the time allowed for Cloudstack to complete each > asynchronous job triggered. If unset, this can be sourced from the > CLOUDSTACK_TIMEOUT environment variable. Otherwise, this will default to 300 > seconds > > Regards > Kiran > > > > > > > From: Brad House <[email protected]> > Date: Tuesday, 30 September 2025 at 11:42 PM > To: [email protected] <[email protected]> > Subject: Re: [VOTE] Release Apache CloudStack Terraform Provider v0.6.0 RC-2 > The rule actually gets created, but it throws the error. I have to > manually delete the rule to re-add it and test again. I tried adding > your depends_on clause just to see, but it still fails. > > It does take about 30s (even though it prints 20s) for it to fail, maybe > there's a timeout and my system is just slow (it is a test lab on 7yr > old hw). > > Can I do something to see a network trace? I enabled TF_LOG=TRACE and > it doesn't really show much: > > cloudstack_port_forward.bastion: Still creating... [00m20s elapsed] > 2025-09-30T18:10:07.464Z [TRACE] dag/walk: vertex > "provider[\"registry.terraform.io/cloudstack/cloudstack\"] (close)" is > waiting for "cloudstack_port_forward.bastion" > 2025-09-30T18:10:07.465Z [TRACE] dag/walk: vertex "root" is waiting for > "provider[\"registry.terraform.io/cloudstack/cloudstack\"] (close)" > 2025-09-30T18:10:12.469Z [TRACE] dag/walk: vertex "root" is waiting for > "provider[\"registry.terraform.io/cloudstack/cloudstack\"] (close)" > 2025-09-30T18:10:12.469Z [TRACE] dag/walk: vertex > "provider[\"registry.terraform.io/cloudstack/cloudstack\"] (close)" is > waiting for "cloudstack_port_forward.bastion" > 2025-09-30T18:10:16.347Z [TRACE] provider.terraform-provider-cloudstack: > Called downstream: > @caller=github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:910 > @module=sdk.helper_schema > tf_provider_addr=registry.terraform.io/cloudstack/cloudstack > tf_resource_type=cloudstack_port_forward tf_rpc=ApplyResourceChange > tf_mux_provider=tf5to6server.v5tov6Server > tf_req_id=c8dcb1e6-d571-7d29-8136-8bf92a942554 > timestamp=2025-09-30T18:10:16.347Z > 2025-09-30T18:10:16.347Z [TRACE] provider.terraform-provider-cloudstack: > Received downstream response: > tf_req_id=c8dcb1e6-d571-7d29-8136-8bf92a942554 > tf_resource_type=cloudstack_port_forward > @caller=github.com/hashicorp/[email protected]/tfprotov6/internal/tf6serverlogging/downstream_request.go:42 > @module=sdk.proto diagnostic_warning_count=0 > tf_provider_addr=registry.terraform.io/cloudstack/cloudstack > tf_rpc=ApplyResourceChange diagnostic_error_count=0 tf_proto_version=6.6 > tf_req_duration_ms=28888 timestamp=2025-09-30T18:10:16.347Z > 2025-09-30T18:10:16.347Z [TRACE] provider.terraform-provider-cloudstack: > Served request: @module=sdk.proto tf_proto_version=6.6 > tf_provider_addr=registry.terraform.io/cloudstack/cloudstack > @caller=github.com/hashicorp/[email protected]/tfprotov6/tf6server/server.go:878 > tf_req_id=c8dcb1e6-d571-7d29-8136-8bf92a942554 > tf_resource_type=cloudstack_port_forward tf_rpc=ApplyResourceChange > timestamp=2025-09-30T18:10:16.347Z > 2025-09-30T18:10:16.348Z [TRACE] maybeTainted: > cloudstack_port_forward.bastion encountered an error during creation, so > it is now marked as tainted > 2025-09-30T18:10:16.348Z [TRACE] terraform.contextPlugins: Schema for > provider "registry.terraform.io/cloudstack/cloudstack" is in the global > cache > 2025-09-30T18:10:16.348Z [TRACE] > NodeAbstractResouceInstance.writeResourceInstanceState to workingState > for cloudstack_port_forward.bastion > 2025-09-30T18:10:16.348Z [TRACE] > NodeAbstractResouceInstance.writeResourceInstanceState: removing state > object for cloudstack_port_forward.bastion > 2025-09-30T18:10:16.348Z [TRACE] evalApplyProvisioners: > cloudstack_port_forward.bastion is tainted, so skipping provisioning > 2025-09-30T18:10:16.348Z [TRACE] maybeTainted: > cloudstack_port_forward.bastion was already tainted, so nothing to do > 2025-09-30T18:10:16.348Z [TRACE] terraform.contextPlugins: Schema for > provider "registry.terraform.io/cloudstack/cloudstack" is in the global > cache > 2025-09-30T18:10:16.348Z [TRACE] > NodeAbstractResouceInstance.writeResourceInstanceState to workingState > for cloudstack_port_forward.bastion > 2025-09-30T18:10:16.348Z [TRACE] > NodeAbstractResouceInstance.writeResourceInstanceState: removing state > object for cloudstack_port_forward.bastion > 2025-09-30T18:10:16.349Z [TRACE] statemgr.Filesystem: creating backup > snapshot at terraform.tfstate.backup > 2025-09-30T18:10:16.351Z [TRACE] statemgr.Filesystem: state has changed > since last snapshot, so incrementing serial to 600 > 2025-09-30T18:10:16.351Z [TRACE] statemgr.Filesystem: writing snapshot > at terraform.tfstate > 2025-09-30T18:10:16.356Z [DEBUG] State storage *statemgr.Filesystem > declined to persist a state snapshot > 2025-09-30T18:10:16.356Z [ERROR] vertex > "cloudstack_port_forward.bastion" error: Provider produced inconsistent > result after apply > 2025-09-30T18:10:16.356Z [TRACE] vertex > "cloudstack_port_forward.bastion": visit complete, with errors > 2025-09-30T18:10:16.356Z [TRACE] dag/walk: upstream of > "provider[\"registry.terraform.io/cloudstack/cloudstack\"] (close)" > errored, so skipping > 2025-09-30T18:10:16.356Z [TRACE] dag/walk: upstream of "root" errored, > so skipping > 2025-09-30T18:10:16.356Z [TRACE] terraform.contextPlugins: Schema for > provider "registry.terraform.io/cloudstack/cloudstack" is in the global > cache > 2025-09-30T18:10:16.356Z [TRACE] statemgr.Filesystem: have already > backed up original terraform.tfstate to terraform.tfstate.backup on a > previous write > 2025-09-30T18:10:16.357Z [TRACE] statemgr.Filesystem: state has changed > since last snapshot, so incrementing serial to 601 > 2025-09-30T18:10:16.357Z [TRACE] statemgr.Filesystem: writing snapshot > at terraform.tfstate > > On 9/30/25 1:51 PM, Kiran Chavala wrote: >> Hi Brad >> >> I am not facing the issue , here is the terraform config that I have used >> >> screenshot >> >> >> https://prnt.sc/8FOLXkgwmYWm >> >> --------- >> >> resource "cloudstack_ipaddress" "bastion" { >> vpc_id = "b12facb3-5787-4ce9-bee2-0ea8d676e26d" >> zone = "fe54a4ca-6ca8-4ac9-9769-dbc1244f138b" >> } >> >> resource "cloudstack_port_forward" "bastion" { >> ip_address_id = cloudstack_ipaddress.bastion.id >> >> forward { >> protocol = "tcp" >> private_port = 22 >> public_port = 5022 >> virtual_machine_id = "92e55318-2ae3-457c-811a-d6885efb381c" >> } >> depends_on = [ >> cloudstack_ipaddress.bastion >> ] >> } >> >> >> >> terraform apply >> >> Terraform used the selected providers to generate the following execution >> plan. Resource actions are indicated with the following symbols: >> + create >> >> Terraform will perform the following actions: >> >> # cloudstack_ipaddress.bastion will be created >> + resource "cloudstack_ipaddress" "bastion" { >> + id = (known after apply) >> + ip_address = (known after apply) >> + is_portable = false >> + is_source_nat = (known after apply) >> + project = (known after apply) >> + tags = (known after apply) >> + vpc_id = "b12facb3-5787-4ce9-bee2-0ea8d676e26d" >> + zone = "fe54a4ca-6ca8-4ac9-9769-dbc1244f138b" >> } >> >> # cloudstack_port_forward.bastion will be created >> + resource "cloudstack_port_forward" "bastion" { >> + id = (known after apply) >> + ip_address_id = (known after apply) >> + managed = false >> >> + forward { >> + private_end_port = (known after apply) >> + private_port = 22 >> + protocol = "tcp" >> + public_end_port = (known after apply) >> + public_port = 5022 >> + uuid = (known after apply) >> + virtual_machine_id = "92e55318-2ae3-457c-811a-d6885efb381c" >> # (1 unchanged attribute hidden) >> } >> } >> >> Plan: 2 to add, 0 to change, 0 to destroy. >> >> Do you want to perform these actions? >> Terraform will perform the actions described above. >> Only 'yes' will be accepted to approve. >> >> Enter a value: yes >> >> cloudstack_ipaddress.bastion: Creating... >> cloudstack_ipaddress.bastion: Creation complete after 1s >> [id=ae73e8e4-93ea-46de-95ff-eece5828cdb5] >> cloudstack_port_forward.bastion: Creating... >> cloudstack_port_forward.bastion: Creation complete after 5s >> [id=ae73e8e4-93ea-46de-95ff-eece5828cdb5] >> >> >> >> >> Regards >> kiran >> >> >> >> >> >> From: Brad House <[email protected]> >> Date: Tuesday, 30 September 2025 at 10:42 PM >> To: [email protected] <[email protected]> >> Subject: Re: [VOTE] Release Apache CloudStack Terraform Provider v0.6.0 RC-2 >> It appears port forwarding is still broken even though this discussion >> indicated it was fixed: >> >> https://github.com/apache/cloudstack-terraform-provider/issues/227 >> >> $ terraform init -upgrade >> ... >> Initializing provider plugins... >> - Finding cloudstack/cloudstack versions matching "0.6.0-rc2"... >> - Installing cloudstack/cloudstack v0.6.0-rc2... >> - Installed cloudstack/cloudstack v0.6.0-rc2 (self-signed, key ID >> 7B3F1C5E93F97FAB) >> >> >> >> Terraform will perform the following actions: >> >> # cloudstack_port_forward.bastion will be created >> + resource "cloudstack_port_forward" "bastion" { >> + id = (known after apply) >> + ip_address_id = "02aae2fe-ae43-4030-8123-8967931fd7e1" >> + managed = false >> >> + forward { >> + private_end_port = (known after apply) >> + private_port = 22 >> + protocol = "tcp" >> + public_end_port = (known after apply) >> + public_port = 5022 >> + uuid = (known after apply) >> + virtual_machine_id = "bebcac94-65f5-4c73-8586-e6e860bcc17e" >> # (1 unchanged attribute hidden) >> } >> } >> >> Plan: 1 to add, 0 to change, 0 to destroy. >> >> Do you want to perform these actions? >> Terraform will perform the actions described above. >> Only 'yes' will be accepted to approve. >> >> Enter a value: yes >> >> cloudstack_port_forward.bastion: Creating... >> cloudstack_port_forward.bastion: Still creating... [00m10s elapsed] >> cloudstack_port_forward.bastion: Still creating... [00m20s elapsed] >> ╷ >> │ Error: Provider produced inconsistent result after apply >> │ >> │ When applying changes to cloudstack_port_forward.bastion, provider >> "provider[\"registry.terraform.io/cloudstack/cloudstack\"]" produced an >> unexpected new value: Root object >> │ was present, but now absent. >> │ >> │ This is a bug in the provider, which should be reported in the >> provider's own issue tracker. >> ╵ >> >> On 9/30/25 5:30 AM, Kiran Chavala wrote: >>> Hi ALL >>> >>> I've created a CloudStack Terraform Provider release candidate for version >>> 0.6.0, with the following artefacts up for a vote: >>> >>> Git Branch and Commit SH: >>> >>> https://github.com/cloudstack/terraform-provider-cloudstack >>> >>> Commit: 919a358fcc951724c2364ee363a9a055258b1fc3 >>> >>> Source release (checksums and signatures are available at the same >>> location): >>> >>> https://dist.apache.org/repos/dist/dev/cloudstack/cloudstack-terraform-provider-0.6.0/ >>> >>> PGP release keys (signed using E03379CB066175FAC2BC9E027B3F1C5E93F97FAB): >>> >>> https://dist.apache.org/repos/dist/release/cloudstack/KEYS >>> >>> For testing/voting purposes, you can use the provider >>> >>> terraform { >>> required_providers { >>> cloudstack = { >>> source = "cloudstack/cloudstack" >>> version = "0.6.0-rc2" >>> } >>> } >>> } >>> >>> >>> >>> Vote will be open for 72 hours. >>> >>> For sanity in tallying the vote, can PMC members please be sure to indicate >>> "(binding)" with their vote? >>> >>> [ ] +1 approve >>> [ ] +0 no opinion >>> [ ] -1 disapprove (and reason why) >>>
