[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user karuturi commented on the issue: https://github.com/apache/cloudstack/pull/2003 ok. Thanks everyone. I am merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @karuturi we have 3 LGTMs, I think we're good to merge this. tag:mergeready --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 I think #2011 fixes the observed issue, but this PR improves the IP ordering fix originally added to the strongswan pr. This implementation ensures (again) that duplicate public ips will not be saved to the data bag while ensuring the source nature ip is primary on its nic. In short, I think both PRs are worth merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @karuturi to be honest, I think PR#2011 fixes the issue, not this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @rhtyd I think this was introduced with the StrongSwan implementation which was 4.10 I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user rhtyd commented on the issue: https://github.com/apache/cloudstack/pull/2003 LGTM, does this affect 4.9 as well @swill ? /cc @borisstoyanov --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user karuturi commented on the issue: https://github.com/apache/cloudstack/pull/2003 @ustcweizhou can you also review this fix? Is pr #2011 still required? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @DaanHoogland you could review it as well it addressed in https://github.com/apache/cloudstack/pull/2011/files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user DaanHoogland commented on the issue: https://github.com/apache/cloudstack/pull/2003 code LGTM but @ustcweizhou 's patch still makes sense to me. Maybe you can add that separately? travis still fails! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 ping @rhtyd @DaanHoogland @abhinandanprateek @PaulAngus for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/2003 Trillian test result (tid-958) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 31840 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2003-t958-kvm-centos7.zip Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py Test completed. 46 look ok, 3 have error(s) Test | Result | Time (s) | Test File --- | --- | --- | --- test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 369.05 | test_vpc_redundant.py test_04_rvpc_privategw_static_routes | `Failure` | 335.61 | test_privategw_acl.py test_02_list_snapshots_with_removed_data_store | `Error` | 0.04 | test_snapshots.py test_01_vpc_site2site_vpn | Success | 145.03 | test_vpc_vpn.py test_01_vpc_remote_access_vpn | Success | 66.12 | test_vpc_vpn.py test_01_redundant_vpc_site2site_vpn | Success | 250.53 | test_vpc_vpn.py test_02_VPC_default_routes | Success | 275.54 | test_vpc_router_nics.py test_01_VPC_nics_after_destroy | Success | 547.92 | test_vpc_router_nics.py test_05_rvpc_multi_tiers | Success | 508.38 | test_vpc_redundant.py test_04_rvpc_network_garbage_collector_nics | Success | 1403.12 | test_vpc_redundant.py test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Success | 543.21 | test_vpc_redundant.py test_02_redundant_VPC_default_routes | Success | 750.68 | test_vpc_redundant.py test_09_delete_detached_volume | Success | 156.45 | test_volumes.py test_08_resize_volume | Success | 156.39 | test_volumes.py test_07_resize_fail | Success | 161.50 | test_volumes.py test_06_download_detached_volume | Success | 156.27 | test_volumes.py test_05_detach_volume | Success | 150.72 | test_volumes.py test_04_delete_attached_volume | Success | 151.19 | test_volumes.py test_03_download_attached_volume | Success | 156.32 | test_volumes.py test_02_attach_volume | Success | 94.54 | test_volumes.py test_01_create_volume | Success | 711.06 | test_volumes.py test_03_delete_vm_snapshots | Success | 275.20 | test_vm_snapshots.py test_02_revert_vm_snapshots | Success | 95.66 | test_vm_snapshots.py test_01_create_vm_snapshots | Success | 163.76 | test_vm_snapshots.py test_deploy_vm_multiple | Success | 247.58 | test_vm_life_cycle.py test_deploy_vm | Success | 0.03 | test_vm_life_cycle.py test_advZoneVirtualRouter | Success | 0.02 | test_vm_life_cycle.py test_10_attachAndDetach_iso | Success | 26.64 | test_vm_life_cycle.py test_09_expunge_vm | Success | 125.24 | test_vm_life_cycle.py test_08_migrate_vm | Success | 45.95 | test_vm_life_cycle.py test_07_restore_vm | Success | 0.15 | test_vm_life_cycle.py test_06_destroy_vm | Success | 125.83 | test_vm_life_cycle.py test_03_reboot_vm | Success | 125.85 | test_vm_life_cycle.py test_02_start_vm | Success | 10.16 | test_vm_life_cycle.py test_01_stop_vm | Success | 40.31 | test_vm_life_cycle.py test_CreateTemplateWithDuplicateName | Success | 30.38 | test_templates.py test_08_list_system_templates | Success | 0.03 | test_templates.py test_07_list_public_templates | Success | 0.04 | test_templates.py test_05_template_permissions | Success | 0.06 | test_templates.py test_04_extract_template | Success | 5.13 | test_templates.py test_03_delete_template | Success | 5.13 | test_templates.py test_02_edit_template | Success | 90.17 | test_templates.py test_01_create_template | Success | 25.33 | test_templates.py test_10_destroy_cpvm | Success | 161.59 | test_ssvm.py test_09_destroy_ssvm | Success | 163.56 | test_ssvm.py test_08_reboot_cpvm | Success | 101.50 | test_ssvm.py test_07_reboot_ssvm | Success | 133.54 | test_ssvm.py test_06_stop_cpvm | Success | 131.72 | test_ssvm.py test_05_stop_ssvm | Success | 193.66 | test_ssvm.py test_04_cpvm_internals | Success | 1.22 | test_ssvm.py test_03_ssvm_internals | Success | 3.25 | test_ssvm.py test_02_list_cpvm_vm | Success | 0.12 | test_ssvm.py test_01_list_sec_storage_vm | Success | 0.13 | test_ssvm.py test_01_snapshot_root_disk | Success | 11.12 | test_snapshots.py test_04_change_offering_small | Success | 239.60 | test_service_offerings.py test_03_delete_service_offering | Success | 0.04 | test_service_offerings.py test_02_edit_service_offering | Success | 0.05 | test_service_offerings.py test_01_create_service_offering | Success | 0.11 | test_service_offerings.py test_02_sys_template_ready | Success | 0.12 | test_secondary_storage.py test_01_sys_vm_start | Success | 0.18 | test_secondary_storage.py test_09_reboot_router | Success | 35.33
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @blueorangutan test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 thank you sir. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @blueorangutan test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/2003 Packaging result: âcentos6 âcentos7 âdebian. JID-594 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 not sure about the CI tests @swill, I think the easiest way to kick Travis tests is the close/reopen the PR. I'll pick it up as soon as possible. I'll rebuild and run the smoketests. @blueorangutan package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov alright, this is ready for you to start testing. Can you kick off CI on this as well? I will be doing testing of this locally as well. This implementation is very similar to how it was implemented before my original change. See: https://github.com/apache/cloudstack/pull/1741/files#diff-a7d6f7150cca74029f23c19b72ad0622L19 The only change from the original was that if the IP is a `source_nat` IP and there are already IPs associated with that dev, it will `prepend` instead of `append` so the source nat ip is set as the primary ip on the nic (required for VPN). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 Thanks @remibergsma. I am going to use `source_nat` as it better represents what it is I am trying to establish and it seems to always be in the object where `first_i_p` does not always seem to be present (look at @borisstoyanov's output above for example). @borisstoyanov I will work on getting you the new implementation today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user remibergsma commented on the issue: https://github.com/apache/cloudstack/pull/2003 @swill FYI if you look for `first_i_p` as seen in the json on the Python side, then on the Java side it's called `firstIP`, as the gson lib replaces every capital with an underscore and then the lowercase letter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov could you please test with my suggestion? I think it should fix the duplicated nics in VR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 Hi @swill, could you please let us know when do you expect to address the changes you have mentioned, so I could schedule the testing accordingly. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 I am considering changing my implementation to be the same as the old implementation (which removed the IP from the dbag in the initial loop of the merge), but if `source_nat` is present and it is `true` and the `len(dbag[ip['device']]) > 0` (with all the checking required), then prepend else append. This should ensure that the source nat ip is the primary ip on the nic which the StrongSwan feature depends on. If I do this and there is a bug elsewhere that will duplicate the IPs in the databag, then this will clean that up (as I assume it was doing before). Not sure this will fix @borisstoyanov's current problem, but I think it is probably a safer implementation to keep the source nat IP as the first IP. I will update this PR with that change later today... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov I suspect this is caused by the name of your public interface (p55p1). can you please try the following change ``` diff --git a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingReso index d925297..706f7d9 100755 --- a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java +++ b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java @@ -1481,7 +1481,7 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv private String getBroadcastUriFromBridge(final String brName) { final String pif = matchPifFileInDirectory(brName); -final Pattern pattern = Pattern.compile("(\\D+)(\\d+)(\\D*)(\\d*)"); +final Pattern pattern = Pattern.compile("(\\D+)(\\d+)(\\D*)(\\d*)(\\D*)(\\d*)"); final Matcher matcher = pattern.matcher(pif); s_logger.debug("getting broadcast uri for pif " + pif + " and bridge " + brName); if(matcher.find()) { @@ -1489,7 +1489,9 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv return BroadcastDomainType.Vxlan.toUri(matcher.group(2)).toString(); } else{ -if (!matcher.group(4).isEmpty()) { +if (!matcher.group(6).isEmpty()) { +return BroadcastDomainType.Vlan.toUri(matcher.group(6)).toString(); +} else if (!matcher.group(4).isEmpty()) { return BroadcastDomainType.Vlan.toUri(matcher.group(4)).toString(); } else { //untagged or not matching (eth|bond|team)#.# ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 Thanks @ustcweizhou here's the StartCommand log ``` 2017-03-16 08:48:46,068 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:be1f1987) Request:Seq 1-2088825802169778189: { Cmd , MgmtId: 6629449860900, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":5,"name":"r-5-VM","type":"DomainRouter","cpus":1,"minSpeed":250,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-5-VM eth2ip=10.1.35.83 eth2mask=255.255.224.0 gateway=10.1.63.254 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24 dhcprange=10.1.1.1 eth1ip=169.254.2.157 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=QtUSkBDvXbuIN02ekUDrlpkmPzc5XXTVAOV1_L3Nvozwp46Xyq7PnGvYYiPWctsrMj2cgIIxFIUzXOXLntsomA baremetalnotificationapikey=3cvNCQ5aBbRVT2a1vjuO0WKF3cELdiS86t_J_5GJnpPTobkyHZPjmxIKuDNHZHRpzfWooLi6pms0QPiEhpiq5A host=1 0.2.2.44 port=8080","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"aFTlD_W3vJCMrO9fjc_GFA","params":{"cpuOvercommitRatio":"2.0","memoryOvercommitRatio":"1.0"},"uuid":"469106c8-107d-46b9-9bd0-237221c97ffe","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2844d31e-edd9-40a1-8e2d-0394df1b2d1d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a86067db-e2df-3c95-819c-254d6d60f4bd","id":2,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-106-k-cs410-bstoyanov/ref-trl-106-k-cs410-bstoyanov-kvm-pri2","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-106-k-cs410-bstoyanov/ref-trl-106-k-cs410-bstoyanov-kvm-pri2/?ROLE=Primary=a86067db-e2df-3c95-819c-254d6d60f4bd","isManaged":false}},"name":"ROOT-5","size":348984320,"path":"2844d31e-edd9-40a1-8e2d-0394df1b2d1d","volumeId":5,"vmName":"r-5-VM","accountId":2,"format":"QCOW2","prov isioningType":"THIN","id":5,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"2844d31e-edd9-40a1-8e2d-0394df1b2d1d","type":"ROOT","_details":{"storageHost":"10.2.0.16","managed":"false","storagePort":"2049","volumeSize":"348984320"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"8bc36a5d-e51a-4f93-b764-89d452a89617","uuid":"268a2858-a1ac-4aec-aa3f-365c4ea97d18","ip":"10.1.35.83","netmask":"255.255.224.0","gateway":"10.1.63.254","mac":"06:17:c6:00:00:03","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://7","isolationUri":"vlan://7","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"pxeDisable":true,"nicUuid":"cf5bcb72-96e2-47a2-a5b2-9187d8922ae5","uuid":"9046a22a-ac92-45c9-9757-107ca10e38d4","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:46:fd:00:03","d ns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://884","isolationUri":"vlan://884","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"8cc9d552-6238-4add-9497-9b3d61ca938a","uuid":"9a24aaa5-e041-460c-9954-660fb4a41404","ip":"169.254.2.157","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:9d","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.2.5.1","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.2.157","port":3922,"interval":6,"retries":100,"name":"r-5-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.ip":"169.254.2.157","router.name":"r-5-VM"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"10.1.1.1","router.ip":"169.254.2.157","router.name":"r
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov I think this is caused by KVM plugin. can you please perform the following on KVM host ? (1) change INFO to DEBUG in /etc/cloudstack/agent/log4j-cloud.xml, and restart cloudstack-agent (2) restart the VR (on UI), and post the StartCommand in /var/log/cloudstack/agent/agent.log (3) post the xml definition of VR (virsh dumpxml r--VM) (4) post bridge information (brctl show) (5) post /etc/cloudstack/agent/agent.properties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @swill first_i_p in json -> firstIP in java. This is formatted by google.gson it is defined in api/src/com/cloud/agent/api/to/IpAddressTO.java, and used in server/src/com/cloud/network/router/CommandSetupHelper.java so, the issue might be caused by commit baac747089ef48ea6627a6aacf27156222862352 not sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 Does anyone know what `first_i_p` represents in the `ips.json` dbag? I am assuming it is the Source NAT IP, but I can't find it represented anywhere in the code (other than in a test). If it does represent the Source NAT, then we can use it to maintain the order and keep the Source NAT IP as the primary IP on the Nic. That was the bug I was trying to fix in my #1741 PR. Example with: 1) source nat 2) port forwarding 3) static nat 4) load balancer ``` "eth2": [ { "add": true, "broadcast": "192.168.80.255", "cidr": "192.168.80.8/24", "device": "eth2", "first_i_p": true, "gateway": "192.168.80.5", "netmask": "255.255.255.0", "network": "192.168.80.0/24", "new_nic": false, "nic_dev_id": 2, "nw_type": "public", "one_to_one_nat": false, "public_ip": "192.168.80.8", "size": "24", "source_nat": true, "vif_mac_address": "06:00:44:00:00:23" }, { "add": true, "broadcast": "192.168.80.255", "cidr": "192.168.80.9/24", "device": "eth2", "first_i_p": false, "gateway": "192.168.80.5", "netmask": "255.255.255.0", "network": "192.168.80.0/24", "new_nic": false, "nic_dev_id": 2, "nw_type": "public", "one_to_one_nat": false, "public_ip": "192.168.80.9", "size": "24", "source_nat": false, "vif_mac_address": "06:34:d1:00:00:23" }, { "add": true, "broadcast": "192.168.80.255", "cidr": "192.168.80.10/24", "device": "eth2", "first_i_p": false, "gateway": "192.168.80.5", "netmask": "255.255.255.0", "network": "192.168.80.0/24", "new_nic": false, "nic_dev_id": 2, "nw_type": "public", "one_to_one_nat": true, "public_ip": "192.168.80.10", "size": "24", "source_nat": false, "vif_mac_address": "06:34:d1:00:00:23" }, { "add": true, "broadcast": "192.168.80.255", "cidr": "192.168.80.11/24", "device": "eth2", "first_i_p": false, "gateway": "192.168.80.5", "netmask": "255.255.255.0", "network": "192.168.80.0/24", "new_nic": false, "nic_dev_id": 2, "nw_type": "public", "one_to_one_nat": false, "public_ip": "192.168.80.11", "size": "24", "source_nat": false, "vif_mac_address": "06:34:d1:00:00:23" } ], ``` @ustcweizhou @murali-reddy @remibergsma --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 nics in database seems ok, so it looks some issue in creating ip_associations.json --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @ustcweizhou - I was able to restart the network with cleanup. New VR came up with duplicated eth2 and eth3. ![screen shot 2017-03-15 at 9 48 33 am](https://cloud.githubusercontent.com/assets/13551960/23938354/b2cb8398-0964-11e7-8690-4cb0d635ec9c.png) - Not sure what you mean by what is my last commit, currently this PR has been built and deployed on the physical hosts. - There are 3 nics in the DB for that router. ``` *** 8. row *** id: 8 uuid: 9a2d5343-fbff-487e-a48d-509ae0503687 instance_id: 4 mac_address: 02:00:19:89:00:02 ip4_address: 10.1.1.1 netmask: 255.255.255.0 gateway: NULL ip_type: NULL broadcast_uri: vlan://884 network_id: 204 mode: Dhcp state: Reserved strategy: Start reserver_name: ExternalGuestNetworkGuru reservation_id: 2f428f23-afae-4916-a423-4a7452a33138 device_id: 0 update_time: 2017-03-14 13:41:23 isolation_uri: vlan://884 ip6_address: NULL default_nic: 0 vm_type: DomainRouter created: 2017-03-14 09:31:29 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 *** 9. row *** id: 9 uuid: bb36a0a5-ac45-45ac-976f-341e2d2c72be instance_id: 4 mac_address: 0e:00:a9:fe:00:44 ip4_address: 169.254.0.68 netmask: 255.255.0.0 gateway: 169.254.0.1 ip_type: Ip4 broadcast_uri: NULL network_id: 202 mode: Static state: Reserved strategy: Start reserver_name: ControlNetworkGuru reservation_id: 2f428f23-afae-4916-a423-4a7452a33138 device_id: 1 update_time: 2017-03-14 13:41:23 isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: DomainRouter created: 2017-03-14 09:31:29 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 *** 10. row *** id: 10 uuid: a4e7691b-b290-4bfe-905e-ddb3c5b1c466 instance_id: 4 mac_address: 06:bb:c8:00:00:03 ip4_address: 10.1.35.83 netmask: 255.255.224.0 gateway: 10.1.63.254 ip_type: NULL broadcast_uri: vlan://7 network_id: 200 mode: Static state: Reserved strategy: Managed reserver_name: PublicNetworkGuru reservation_id: NULL device_id: 2 update_time: 2017-03-14 13:41:23 isolation_uri: vlan://7 ip6_address: NULL default_nic: 1 vm_type: DomainRouter created: 2017-03-14 09:31:29 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov moreover, can you please restart the network with cleanup, to see if the new VRs can start ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 Ya, I am not sure where the problem is stemming from. The fact that my code is not defensive around this problem is a problem. That is for sure. But given my understanding of the moving parts here, I feel like there is something else going on as well. I will see if I can get an environment up and running tomorrow so I can do some more testing on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov what's the last commit in your code ? could you check how many nics attached to the VR in database, select * from nics where instance_id= ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 it seems nicDevId is not set correctly in java code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 yes, agree with you @swill, it seems eth3 is duplicated eth2. The VR we're trying to bring up isn't supposed to have eth3 at all... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 So... Looking at this with a different perspective. If there is a bug elsewhere which would cause an IP to be duplicated in the `dbag`, the old implementation would have cleaned up this bug and would have removed all the duplicates (including the real one) and then would have only added back the real one. My implementation did not assume that I was having to clean up duplicates in the `dbag` variable and only focused on adding if it didn't exist or updating the existing entry if it did exist. I will have to think about this more... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 This `ips.json` file confuses the hell out of me. From what I can tell both `eth2` and `eth3` both have the ip `10.1.35.83` twice for each (for a total of the IP being configured 4 times over two interfaces). How was this IP configured? ``` "eth2": [ { "add": true, "broadcast": "10.1.63.255", "cidr": "10.1.35.83/19", "device": "eth2", "gateway": "10.1.63.254", "netmask": "255.255.224.0", "network": "10.1.32.0/19", "nic_dev_id": "2", "nw_type": "public", "one_to_one_nat": false, "public_ip": "10.1.35.83", "size": "19", "source_nat": false }, { "add": true, "broadcast": "10.1.63.255", "cidr": "10.1.35.83/19", "device": "eth2", "gateway": "10.1.63.254", "netmask": "255.255.224.0", "network": "10.1.32.0/19", "nic_dev_id": "2", "nw_type": "public", "one_to_one_nat": false, "public_ip": "10.1.35.83", "size": "19", "source_nat": false } ], "eth3": [ { "add": true, "broadcast": "10.1.63.255", "cidr": "10.1.35.83/19", "device": "eth3", "first_i_p": true, "gateway": "10.1.63.254", "netmask": "255.255.224.0", "network": "10.1.32.0/19", "new_nic": true, "nic_dev_id": 3, "nw_type": "public", "one_to_one_nat": false, "public_ip": "10.1.35.83", "size": "19", "source_nat": true, "vif_mac_address": "06:ee:46:00:00:03" }, { "add": true, "broadcast": "10.1.63.255", "cidr": "10.1.35.83/19", "device": "eth3", "first_i_p": true, "gateway": "10.1.63.254", "netmask": "255.255.224.0", "network": "10.1.32.0/19", "new_nic": true, "nic_dev_id": 3, "nw_type": "public", "one_to_one_nat": false, "public_ip": "10.1.35.83", "size": "19", "source_nat": true, "vif_mac_address": "06:a4:80:00:00:03" } ], ``` I will see if I can get to the bottom of this, but this is pretty confusing. Do you have a sequence of events which produced this configuration? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov I have a suspicion that the ip_association.json is a transient file which may be short lived. I am not sure because I have not seen it either, but it was the file being processed when you had your original error. I will review your ips.json and see what I can find. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @ustcweizhou no VPC --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @swill thanks, I found the ips.json, but I couldn't find the ip_association.json on the VR. I guess it's not processed yet. Can you advise how to force it to generate? Hope this helps. [ips.txt](https://github.com/apache/cloudstack/files/842199/ips.txt) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov we are still trying to understand how the IP was found when looping through the `dbag` but it is not associated with the expected dev when we try to update the index which it was found on. The only way I can see this could happen is if the dev requested it be configured on is different from the dev it is actually configured on. I still have not figured out how that case exists. @ustcweizhou has some good ideas on this front, but I don't think we have gotten to a point where we understand how this problem is happening. @borisstoyanov is there a chance you can post the `ips.json` and `ip_association.json` files in your environment so we can start to understand what the config is that is causing this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @borisstoyanov are you using vpc ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user borisstoyanov commented on the issue: https://github.com/apache/cloudstack/pull/2003 @swill we're using one IP range for the physical guest public network. I think we could set this up if required? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 On the Jira ticket, Wei Zhou mentioned that this could potentially happen if you have two different public IP ranges. So if you associate a public IP which is in a different range than the source nat, a new nic will be created. If you disassociate it, that nic will be removed. I guess my question is. Why is it not on the`dbag`variable then associated with that dev? I think Wei is probably right in terms of context, but I am not sure how exactly to solve this yet. @borisstoyanov can you confirm that you have two different public IP ranges in your setup? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 I have to admit, I am not sure how the IP was found when looping through the list initially, but then when trying to update the index for the IP that was found, it is not there. The only way I can think of that would cause this is if the IP <-> dev mapping is broken somehow. It found the matching IP, but apparently that IP is not on the dev defined by the `nic_dev_id` (which comes from either: `nic_dev_id = address['nic_dev_id']` (from the databag), which can then be overridden by `nic_dev_id = ip['nic_dev_id']` (from the passed in IP) ). Yes, I think you are right that the `else` case does not work right now. This is because we are appending the IP to this dev, but we never removed it from the other dev it was apparently found in. This logic is SUPPOSED to find the existing IP's index and then if found, update that index with the updated info. The old logic which just removed all "found" IPs and then re-added them, did not preserve the order, so the source nat IP could get reconfigured as a secondary IP on a nic instead of it being primary. This reordering of the IPs on VR reboot caused the VPN to fail because the source nat IP was no longer the primary IP. So we know that removing all found IPs and then adding them does not work because it changes the order of the IPs causing the source nat IP to become a secondary IP on the nic. The error described in CLOUDSTACK-9811 seems to be a situation where the IP is found on one nic, but then for some reason the dev which the IP is associated with is changed (I don't know why this would be happening), causing the index where the IP was found to not be valid because the IP is actually on a different dev. To know how to fix this correctly, we need to understand why/how an IP can be associated with one dev (index found) and then get changed to be associated with a different dev. We need to find a way to preserve IP order while being able to update the IP configurations. @remibergsma do you have any ideas on this? Anyone else have ideas here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user ustcweizhou commented on the issue: https://github.com/apache/cloudstack/pull/2003 @swill what I mean is , if you do not want to change the logic, do you need to change 'else:' to 'elif index == -1:' ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #2003: CLOUDSTACK-9811: fixed an issue if the dev is not in...
Github user swill commented on the issue: https://github.com/apache/cloudstack/pull/2003 @ustcweizhou the `else` case did not change from the original code: https://github.com/apache/cloudstack/pull/1741/files#diff-a7d6f7150cca74029f23c19b72ad0622L49 @karuturi I have updated the PR title and I updated the commit message to include the jira ticket. @borisstoyanov do we have a run of these tests just against master recently to review the difference in output? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---