[ https://issues.apache.org/jira/browse/CLOUDSTACK-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931592#comment-15931592 ]
ASF GitHub Bot commented on CLOUDSTACK-9317: -------------------------------------------- Github user cloudmonger commented on the issue: https://github.com/apache/cloudstack/pull/1908 ### ACS CI BVT Run **Sumarry:** Build Number 3 Hypervisor xenserver NetworkType Advanced Passed=623 Failed=290 Skipped=58 _Link to logs Folder (search by build_no):_ https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0 **Failed tests:** * test_dhcp_dns_offload.py * ContextSuite context=TestDeployVMs>:setup Failed * test_affinity_groups_projects.py * test_01_deploy_vm_anti_affinity_group Failed * test_02_deploy_vm_anti_affinity_group_fail_on_not_enough_hosts Failed * test_01_list_aff_grps_for_vm Failed * test_02_list_multiple_aff_grps_for_vm Failed * test_01_update_aff_grp_by_ids Failed * test_ps_domain_limits.py * test_04_create_template_snapshot Failed * test_ss_domain_limits.py * test_04_create_template_delete_account Failed * test_04_create_template_delete_account Failing since 2 runs * test_01_multiple_domains_secondary_storage_limits Failed * test_01_multiple_domains_secondary_storage_limits Failing since 2 runs * test_02_multiple_domains_secondary_storage_counts Failed * test_deploy_vm_userdata_multi_nic.py * test_deployvm_multinic Failed * test_vpn_service.py * test_01_VPN_service Failed * test_organization_states.py * ContextSuite context=TestOrganizationStates>:setup Failed * test_escalations_snapshots.py * ContextSuite context=TestSnapshots>:setup Failed * test_dynamic_compute_offering.py * test_max_account_memory_scale_VM_1_ADMIN_ACCOUNT Failed * test_max_account_memory_scale_VM_2_USER_ACCOUNT Failed * test_deploy_VM_with_affinity_group_1_ADMIN_ACCOUNT Failed * test_deploy_VM_with_affinity_group_2_USER_ACCOUNT Failed * ContextSuite context=TestDynamicServiceOffering>:setup Failed * test_change_so_running_vm_dynamic_to_dynamic_1_ADMIN_ACCOUNT Failed * test_change_so_running_vm_dynamic_to_dynamic_2_USER_ACCOUNT Failed * test_change_so_running_vm_dynamic_to_static_1_ADMIN_ACCOUNT Failed * test_change_so_running_vm_dynamic_to_static_2_USER_ACCOUNT Failed * test_change_so_running_vm_static_to_dynamic_1_ADMIN_ACCOUNT Failed * test_change_so_running_vm_static_to_dynamic_2_USER_ACCOUNT Failed * test_change_so_running_vm_static_to_static_1_ADMIN_ACCOUNT Failed * test_change_so_running_vm_static_to_static_2_USER_ACCOUNT Failed * test_change_so_stopped_vm_dynamic_to_dynamic_1_ADMIN_ACCOUNT Failed * test_change_so_stopped_vm_dynamic_to_dynamic_2_USER_ACCOUNT Failed * test_change_so_stopped_vm_dynamic_to_static_1_ADMIN_ACCOUNT Failed * test_change_so_stopped_vm_dynamic_to_static_2_USER_ACCOUNT Failed * test_change_so_stopped_vm_static_to_dynamic_1_ADMIN_ACCOUNT Failed * test_change_so_stopped_vm_static_to_dynamic_2_USER_ACCOUNT Failed * test_change_so_stopped_vm_static_to_static_1_ADMIN_ACCOUNT Failed * test_ps_resource_limits_volume.py * test_attach_volume_exceeding_primary_limits Failed * test_vpc.py * test_07_restart_network_vm_running Failed * test_08_delete_vpc Failed * test_11_deploy_vm_wo_network_netdomain Failed * test_13_deploy_vm_with_vpc_netdomain Failed * test_14_deploy_vm_1 Failed * test_15_deploy_vm_2 Failed * test_16_deploy_vm_for_user_by_admin Failed * test_21_deploy_vm_with_gateway_ip Failed * test_22_vpn_customer_gw_with_hostname Failed * test_non_contiguous_vlan.py * test_01_add_non_contiguous_ranges Failed * test_project_usage.py * ContextSuite context=TestLBRuleUsage>:setup Failed * ContextSuite context=TestNatRuleUsage>:setup Failed * ContextSuite context=TestPublicIPUsage>:setup Failed * ContextSuite context=TestSnapshotUsage>:setup Failed * ContextSuite context=TestVmUsage>:setup Failed * ContextSuite context=TestVolumeUsage>:setup Failed * ContextSuite context=TestVpnUsage>:setup Failed * test_ps_max_limits.py * test_01_deploy_vm_domain_limit_reached Failed * test_02_deploy_vm_account_limit_reached Failed * test_03_deploy_vm_project_limit_reached Failed * test_project_resources.py * test_07_associate_public_ip Failed * test_vpc_network_staticnatrule.py * test_01_VPC_StaticNatRuleCreateStoppedState Failed * test_03_VPC_StopCreateMultipleStaticNatRuleStopppedState Failed * test_04_VPC_CreateMultipleStaticNatRule Failed * test_05_network_services_VPC_DeleteAllPF Failed * test_06_network_services_VPC_DeleteAllMultiplePF Failed * test_redundant_router_services.py * test_enableVPNOverRvR Failed * test_resource_limits.py * test_01_vm_per_account Failed * test_02_publicip_per_account Failed * test_03_snapshots_per_account Failed * test_04_volumes_per_account Failed * test_05_templates_per_account Failed * test_escalations_vpncustomergateways.py * test_01_list_vpncustomergateways_pagination Failed * test_02_update_vpncustomergateways Failed * test_multiple_public_interfaces.py * test_iptable_rules Failed * ContextSuite context=TestPortForwarding>:setup Failing since 8 runs * ContextSuite context=TestRouting>:setup Failed * ContextSuite context=TestStaticNat>:setup Failed * test_network_services_VPC_CreatePF Failed * test_network_services_VPC_CreatePF Failing since 2 runs * test_cpu_project_limits.py * test_01_project_counts_start_stop_instance Failed * test_02_project_counts_migrate_instance Failed * test_03_project_counts_delete_instance Failed * test_routers.py * test_01_RouterStopCreateFW Failed * test_overcommit.py * test_01_cluster_settings Failed * test_02_Overcommit_factor Failed * test_03_cluste_capacity_check Failed * test_04_zone_capacity_check Failed * test_custom_hostname.py * test_01_custom_hostname_instancename_false Failed * test_snapshots.py * test_01_createVM_snapshotTemplate Failed * ContextSuite context=TestSnapshotEvents>:setup Failed * test_01_volume_from_snapshot Failed * test_02_snapshot_data_disk Failed * test_03_snapshot_detachedDisk Failed * test_04_delete_snapshot Failed * test_07_template_from_snapshot Failed * test_egress_fw_rules.py * test_01_1_egress_fr1 Failed * test_01_2_egress_fr1 Failed * test_01_egress_fr1 Failed * test_02_1_egress_fr2 Failed * test_02_2_egress_fr2 Failed * test_02_egress_fr2 Failed * test_03_1_egress_fr3 Failed * test_03_egress_fr3 Failed * test_04_1_egress_fr4 Failed * test_04_egress_fr4 Failed * test_06_1_egress_fr6 Failed * test_06_egress_fr6 Failed * test_07_1_egress_fr7 Failed * test_07_egress_fr7 Failed * test_09_1_egress_fr9 Failed * test_redundant_router_cleanups.py * test_restart_network_with_destroyed_masterVR Failed * test_ss_max_limits.py * test_01_deploy_vm_domain_limit_reached Failing since 2 runs * test_02_deploy_vm_account_limit_reached Failing since 2 runs * test_03_deploy_vm_project_limit_reached Failing since 2 runs * test_ss_project_limits.py * test_01_register_template_with_project Failed * test_memory_limits.py * test_01_stop_start_instance Failed * test_02_migrate_instance Failed * test_03_delete_instance Failed * test_01_stop_start_instance Failing since 2 runs * test_02_migrate_instance Failing since 2 runs * test_vpc_network_pfrules.py * test_01_network_services_VPC_StopCreatePF Failed * test_02_network_services_VPC_CreatePF Failed * test_03_network_services_VPC_StopCreateMultiplePF Failed * test_04_network_services_VPC_CreateMultiplePF Failed * test_05_network_services_VPC_StopDeletePF Failed * test_06_network_services_VPC_DeletePF Failed * test_07_network_services_VPC_StopDeleteAllPF Failed * test_08_network_services_VPC_DeleteAllPF Failed * test_09_network_services_VPC_StopDeleteAllMultiplePF Failed * test_10_network_services_VPC_DeleteAllMultiplePF Failed * test_ip_reservation.py * test_reservation_after_router_restart Failed * test_vm_create_after_reservation Failed * test_vm_create_outside_cidr_after_reservation Failed * test_ip_reservation_in_multiple_networks_same_account Failed * test_nat_rules_1_nat_rule Failed * test_nat_rules_2_static_nat_rule Failed * test_update_cidr_multiple_vms_not_all_inclusive Failed * test_update_cidr_single_vm_not_inclusive Failed * test_vm_create_after_reservation Failing since 2 runs * test_vm_create_outside_cidr_after_reservation Failing since 2 runs * test_restart_network_with_cleanup_1_True Failed * test_vpc_vms_deployment.py * test_01_deploy_vms_in_network Failed * test_02_deploy_vms_delete_network Failed * test_03_deploy_vms_delete_add_network Failed * test_04_deploy_vms_delete_add_network_noLb Failed * test_05_create_network_max_limit Failed * test_06_delete_network_vm_running Failed * test_07_delete_network_with_rules Failed * test_vpc_network.py * ContextSuite context=TestVPCNetworkGc>:setup Failed * test_mm_project_limits.py * test_01_project_vmlifecycle_start_stop_instance Failed * test_02_project_vmlifecycle_migrate_instance Failed * test_03_project_vmlifecycle_delete_instance Failed * test_escalations_instances.py * test_13_attach_detach_iso Failed * test_14_vm_snapshot_pagination Failed * test_15_revert_vm_to_snapshot Failed * test_16_list_vm_volumes_pagination Failed * test_18_stopped_vm_change_service Failed * test_19_create_reset_vm_sshkey Failed * test_20_update_vm_displayname_group Failed * test_21_restore_vm Failed * test_22_deploy_vm_multiple_networks Failed * test_24_deploy_vm_with_static_ip_ES1662 Failed * test_25_ip_reallocation_ES1377 Failed * test_27_VM_restore_ES3467 Failed * test_28_VM_restore_ES3467 Failed * test_01_list_instances_pagination Failed * test_02_list_Running_vm Failed * test_03_list_Stopped_vm Failed * test_04_list_Destroyed_vm Failed * test_05_list_vm_by_id Failed * test_06_list_vm_by_name Failed * test_07_list_vm_by_name_state Failed * test_10_list_vm_by_zone_name_state Failed * test_11_register_reset_vm_sshkey Failed * test_12_running_vm_change_service Failed * test_13_vm_nics Failed * test_acl_isolatednetwork.py * ContextSuite context=TestIsolatedNetwork>:setup Failed * test_cpu_domain_limits.py * test_01_multiple_core_vm_start_stop_instance Failed * test_02_multiple_core_vm_migrate_instance Failed * test_03_multiple_core_vm_delete_instance Failed * test_04_deploy_multiple_vm_with_multiple_core Failed * test_01_multiple_child_domains Failed * test_escalations_networks.py * test_03_list_vpc_pagination Failed * test_04_create_vpc_with_networkdomain Failed * test_06_create_network_in_vpc Failed * test_10_list_networks_in_vpc Failed * test_11_update_vpc Failed * test_12_list_create_delete_networkACL Failed * test_19_create_list_reset_delete_vpnconnections Failed * test_accounts.py * ContextSuite context=TestAddVmToSubDomain>:setup Failed * test_DeleteDomain Failed * test_forceDeleteDomain Failed * test_01_user_remove_VM_running Failed * ContextSuite context=TestTemplateHierarchy>:setup Failed * test_LoginApiDomain Failed * test_LoginApiUuidResponse Failed * test_ps_limits.py * test_create_template_snapshot_1_root_domain_admin Failed * test_create_template_snapshot_2_child_domain_admin Failed * test_mm_max_limits.py * test_02_deploy_vm_account_limit_reached Failing since 3 runs * test_04_deployVm__account_limit_reached Failed * test_cpu_max_limits.py * test_02_deploy_vm_account_limit_reached Failing since 4 runs * test_04_deployVm__account_limit_reached Failing since 2 runs * test_escalation_listTemplateDomainAdmin.py * test_listtemplate Failed * test_portable_ip.py * test_associate_ip_address_services_enable_disable Failed * test_list_portable_ip_range_non_root_admin Failed * test_affinity_groups.py * test_03_delete_aff_grp_with_vms Failed * test_01_deploy_vm_without_aff_grp Failed * test_02_deploy_vm_by_aff_grp_name Failed * test_03_deploy_vm_by_aff_grp_id Failed * test_04_deploy_vm_anti_affinity_group Failed * test_05_deploy_vm_by_id Failed * test_08_deploy_vm_multiple_aff_grps Failed * test_09_deploy_vm_multiple_aff_grps Failed * test_01_list_aff_grps_for_vm Failing since 2 runs * test_02_list_multiple_aff_grps_for_vm Failing since 2 runs * test_07_list_all_vms_in_aff_grp Failed * test_02_update_aff_grp_by_names Failed * test_03_update_aff_grp_for_vm_with_no_aff_grp Failed * test_05_update_aff_grp_on_running_vm Failed * test_acl_listvolume.py * ContextSuite context=TestVolumeList>:setup Failed * test_mm_domain_limits.py * test_01_change_service_offering Failed * test_02_migrate_vm Failed * test_03_delete_vm Failed * test_04_deploy_multiple_vm Failed * test_01_multiple_child_domains Failing since 2 runs * test_blocker_bugs.py * ContextSuite context=TestDataPersistency>:setup Failed * ContextSuite context=TestNATRules>:setup Failed * ContextSuite context=TestRouterRestart>:setup Failed * test_01_list_routers_admin Failed * test_01_create_template Failed * test_vpc_network_internal_lbrules.py * test_02_internallb_rules_traffic Failed * test_03_internallb_rules_vpc_network_restarts_traffic Failed * test_04_internallb_appliance_operations_traffic Failed * test_add_remove_network.py * ContextSuite context=TestAddNetworkToVirtualMachine>:setup Failed * ContextSuite context=TestFailureScenariosAddNetworkToVM>:setup Failed * ContextSuite context=TestFailureScenariosRemoveNicFromVM>:setup Failed * ContextSuite context=TestFailureScenariosUpdateVirtualMachineNIC>:setup Failed * ContextSuite context=TestRemoveNetworkFromVirtualMachine>:setup Failed * ContextSuite context=TestUpdateVirtualMachineNIC>:setup Failed * test_escalations_ipaddresses.py * test_11_create_delete_firewallrule Failed * test_lb_secondary_ip.py * test_01_lb_rule_for_primary_ip Failed * test_02_lb_rule_for_primary_ip Failed * test_03_lb_rule_for_primary_and_secondary_ip Failed * test_04_lb_rule_primary_secondary_multiple_vms Failed * test_05_lb_rule_wrong_vm_id Failed * test_06_lb_rule_wrong_vm_ip Failed * test_08_lb_rule_remove_used_ip Failed * test_11_ssh_to_secondary_ip Failed * test_13_delete_lb_rule Failed * test_14_remove_lb_rule_secondary_ip Failed * test_15_remove_lb_rule_primary_ip Failed * test_16_delete_vm_from_lb_rule Failed * test_20_destroy_recover_vm Failed * test_21_modify_lb_rule_algorithm Failed * test_09_lbvmips_true Failed * test_10_lbvmips_false Failed * test_17_restart_router Failed * test_18_restart_network_cleanup_true Failed * test_19_restart_network_cleanup_false Failed * test_22_network_gc Failed * test_volumes.py * test_01_volume_attach_detach Failed * test_01_volume_attach_detach Failing since 2 runs * test_02_root_volume_attach_detach Failed * ContextSuite context=TestAttachVolume>:setup Failed * ContextSuite context=TestAttachVolumeISO>:setup Failed * ContextSuite context=TestMigrateVolume>:setup Failed * ContextSuite context=TestVolumes>:setup Failing since 12 runs * test_acl_listsnapshot.py * ContextSuite context=TestSnapshotList>:setup Failed * test_stopped_vm.py * test_deploy_vm_password_enabled Failed * test_snapshot_limits.py * ContextSuite context=TestSnapshotLimit>:setup Failed * test_escalations_templates.py * test_01_list_templates_pagination Failed * test_02_download_template Failed * test_03_edit_template_details Failed * test_ss_limits.py * test_01_register_template_1_root_domain_admin Failed * test_01_register_template_2_child_domain_admin Failed * test_ps_resize_volume.py * test_02_increase_volume_size_above_account_limit Failed * test_03_increase_volume_size_above_domain_limit Failed * test_usage.py * ContextSuite context=TestLBRuleUsage>:setup Failing since 2 runs * ContextSuite context=TestNatRuleUsage>:setup Failing since 2 runs * ContextSuite context=TestPublicIPUsage>:setup Failing since 2 runs * ContextSuite context=TestSnapshotUsage>:setup Failing since 2 runs * ContextSuite context=TestVmUsage>:setup Failing since 2 runs * ContextSuite context=TestVolumeUsage>:setup Failing since 2 runs * ContextSuite context=TestVpnUsage>:setup Failing since 2 runs * test_vpc_vm_life_cycle.py * test_06_migrate_instance_in_network Failed * test_07_migrate_instance_in_network Failed * test_project_limits.py * test_06_volumes_per_project Failed * test_07_templates_per_project Failed * test_templates.py * test_01_create_template Failing since 2 runs * test_assign_vm.py * test_13_move_across_subdomain_vm_snapshot Failed * test_15_move_across_subdomain_account_limit Failed * test_16_move_across_subdomain_volume_and_account_limit Failed **Skipped tests:** test_03_copy_template ContextSuite context=TestvGPUWindowsVm>:setup ContextSuite context=TestTemplateUsage>:setup test_02_create_vpc_from_offering_with_regionlevelvpc_service_capability test_03_deploy_vms_in_vpc_with_regionlevelvpc test_02_cancel_host_maintenace_with_migration_jobs ContextSuite context=TestInstanceNameFlagTrue>:setup test_04_deploy_multiple_vm_with_multiple_cpus test_05_1_egress_fr5 test_05_egress_fr5 test_08_1_egress_fr8 test_08_egress_fr8 test_04_vpn_tag ContextSuite context=TestAffinityRules>:setup ContextSuite context=TestAntiAffinityRules>:setup test_02_create_vpc_from_offering_with_distributedrouter_service_capability test_03_deploy_vms_in_vpc_with_distributedrouter test_04_deploy_multiple_vm ContextSuite context=TestProjectCreationNegative>:setup ContextSuite context=TestProjectInviteRequiredTrue>:setup ContextSuite context=TestProjectInviteTimeout>:setup test_RVR_network test_01_create_network_2_network_offering_vpcns test_02_create_network_fail_2_network_offering_vpcns test_04_create_multiple_networks_with_lb_1_network_offering test_04_create_multiple_networks_with_lb_2_network_offering_vpcns test_06_create_network_with_rvr test_09_create_network_shared_nwoff_2_network_offering_vpcns test_01_create_network_outside_range_2_network_offering_vpcns test_03_create_network_inside_range_2_network_offering_vpcns test_04_create_network_overlapping_range_2_network_offering_vpcns test_05_create_network_diff_account_2_network_offering_vpcns test_02_concurrent_snapshot_global_limit test_01_migrate_vm_with_volume test_02_migrate_volume test_02_migrate_vm test_03_delete_vm test_01_deploy_vm_with_implicit_planner test_destroy_recover_vm_1_root_domain_admin test_destroy_recover_vm_2_child_domain_admin test_04_copy_iso test_04_update_aff_grp_remove_all test_01_host_ha_with_nfs_storagepool_with_vm test_02_host_ha_with_local_storage_and_nfs test_03_host_ha_with_only_local_storage ContextSuite context=TestTemplates>:setup test_07_lb_rule_used_ip test_12_ssh_to_primary_secondary_ip test_09_stop_vm_migrate_vol test_04_copy_template test_04_copy_template_1_root_domain_admin test_04_copy_template_2_child_domain_admin test1_attach_volume_ide test2_attach_ISO_in_CentOSVM ContextSuite context=TestTemplateUsage>:setup test_createSharedNetwork_usedVlan test_01_addLdapConfiguration ContextSuite context=TestTemplates>:setup **Passed test suits:** None test_redundant_router_upgrades.py test_VirtualRouter_alerts.py test_regions_accounts.py test_acl_isolatednetwork_delete.py test_invalid_gw_nm.py test_haproxy.py test_vpc_offerings.py test_snapshots_improvement.py test_vpn_users.py test_vpc_routers.py test_acl_listvm.py test_simultaneous_volume_attach.py test_deploy_vm_userdata_reg.py test_escalations_routers.py test_vpc_network_lbrules.py test_update_vm.py test_escalations_volumes.py test_regions.py test_projects.py test_acl_sharednetwork.py test_allocation_states.py test_snapshot_gc.py test_vm_passwdenabled.py test_network_offering.py None > Disabling static NAT on many IPs can leave wrong IPs on the router > ------------------------------------------------------------------ > > Key: CLOUDSTACK-9317 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9317 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, Virtual Router > Affects Versions: 4.7.0, 4.7.1, 4.7.2 > Reporter: Jeff Hair > > The current behavior of enabling or disabling static NAT will call the apply > IP associations method in the management server. The method is not > thread-safe. If it's called from multiple threads, each thread will load up > the list of public IPs in different states (add or revoke)--correct for the > thread, but not correct overall. Depending on execution order on the virtual > router, the router can end up with public IPs assigned to it that are not > supposed to be on it anymore. When another account acquires the same IP, this > of course leads to network problems. > The problem has been in CS since at least 4.2, and likely affects all > recently released versions. Affected version is set to 4.7.x because that's > what we verified against. -- This message was sent by Atlassian JIRA (v6.3.15#6346)