Nice, but we cannot enable this until all the tests pass or disabled. There is no point in broken or flaky functional tests.
On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <[email protected]> wrote: > So its not stable. It won't block merges and at least give us report after > each merge. It takes really long time to run it (because of the tests > themselves. Lago things takes maximum 15minutes, but the run last for more > than 2hrs right now and I suspect functional/storageTests.py gets stuck) > > Bellow you can see where we stand (before I added python-rtslib package). > > Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ - > which enables this run after merges, and I still want you to consider the > addition of Automation CI flag to our gerrit so that developer will be able > to use it as a trigger for the check-merged.sh script run, just to see if > their patch fixes\brakes something realted to the functional tests > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is > an example of how the run looks like. I still work to improve the output > > > Please reply and let me know if the idea around the automation flag is > acceptable by you.. and please review the patch for comments and acks. > We can ask dcaro to add the flag until Friday, otherwise we'll need to delay > this effort after the holiday.. > > > functional.sosPluginTests.SosPluginTest > testSosPlugin OK > functional.vmRecoveryTests.RecoveryTests > test_vm_recovery FAIL > functional.vmQoSTests.VMQosTests > testSmallVMBallooning FAIL > functional.virtTests.VirtTest > testComplexVm FAIL > testHeadlessVm OK > testSimpleVm FAIL > testVmDefinitionGraphics('spice') FAIL > testVmDefinitionGraphics('vnc') OK > testVmDefinitionLegacyGraphics('qxl') FAIL > testVmDefinitionLegacyGraphics('vnc') OK > testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL > testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL > testVmWithCdrom('self') FAIL > testVmWithCdrom('specParams') FAIL > testVmWithCdrom('vmPayload') FAIL > testVmWithDevice('hotplugDisk') FAIL > testVmWithDevice('hotplugNic') FAIL > testVmWithDevice('smartcard') FAIL > testVmWithDevice('virtioNic') FAIL > testVmWithDevice('virtioRng') FAIL > testVmWithSla FAIL > testVmWithStorage('iscsi') SKIP: > python-rtslib is not installed. > testVmWithStorage('localfs') FAIL > testVmWithStorage('nfs') FAIL > functional.storageTests.StorageTest > testCreatePoolErrors OK > testStorage('glusterfs', 0) ERROR > testStorage('glusterfs', 3) ERROR > testStorage('iscsi', 0) SKIP: > python-rtslib is not installed. > testStorage('iscsi', 3) SKIP: > python-rtslib is not installed. > testStorage('localfs', 0) FAIL > testStorage('localfs', 3) FAIL > testStorage('nfs', 0) FAIL > testStorage('nfs', 3) FAIL > functional.networkTests.NetworkTest > testAddVlanedBridgeless ERROR > testAddVlanedBridgeless_oneCommand ERROR > testAfterNetworkSetupHook ERROR > testBeforeNetworkSetupHook ERROR > testBondHwAddress(False) ERROR > testBondHwAddress(True) ERROR > testBrokenNetworkReplacement(False) ERROR > testBrokenNetworkReplacement(True) ERROR > testDelNetworkBondAccumulation ERROR > testDelNetworkWithMTU(False) ERROR > testDelNetworkWithMTU(True) ERROR > testDelWithoutAdd ERROR > testDhclientLeases(4, 'default') ERROR > testDhclientLeases(4, 'local') ERROR > testDhclientLeases(6, None) ERROR > testDhcpReplaceNicWithBridge ERROR > testFailWithInvalidBondingName(False) ERROR > testFailWithInvalidBondingName(True) ERROR > testFailWithInvalidBridgeName ERROR > testFailWithInvalidIpConfig ERROR > testFailWithInvalidNic(False) ERROR > testFailWithInvalidNic(True) ERROR > testFailWithInvalidParams(False) ERROR > testFailWithInvalidParams(True) ERROR > testGetRouteDeviceTo ERROR > testHonorBlockingDhcp SKIP: Slow > tests are disabled > testIpLinkWrapper ERROR > testLowerMtuDoesNotOverride FAIL > testNoBridgeLeftovers FAIL > testReconfigureBrNetWithVanishedPort ERROR > testRedefineBondedNetworkIPs ERROR > testRemovingBridgeDoesNotLeaveBridge ERROR > testReorderBondingOptions(False) ERROR > testReorderBondingOptions(True) ERROR > testRestoreNetworksOnlyRestoreUnchangedDevices ERROR > testRestoreToBlockingDHCP ERROR > testSafeNetworkConfig(False) ERROR > testSafeNetworkConfig(True) ERROR > testSelectiveRestoreDuringUpgrade ERROR > testSelectiveRestoreIgnoresVdsmRegParams FAIL > testSetupNetworkOutboundQos(False) ERROR > testSetupNetworkOutboundQos(True) FAIL > testSetupNetworksActiveSlave ERROR > testSetupNetworksAddBadParams(False) ERROR > testSetupNetworksAddBadParams(True) FAIL > testSetupNetworksAddBondWithManyVlans(False) ERROR > testSetupNetworksAddBondWithManyVlans(True) FAIL > testSetupNetworksAddDelBondedNetwork(False) ERROR > testSetupNetworksAddDelBondedNetwork(True) FAIL > testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR > testSetupNetworksAddDelDhcp(False, (4,)) ERROR > testSetupNetworksAddDelDhcp(False, (6,)) ERROR > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL > testSetupNetworksAddDelDhcp(True, (4,)) FAIL > testSetupNetworksAddDelDhcp(True, (6,)) FAIL > testSetupNetworksAddManyVlans(False) ERROR > testSetupNetworksAddManyVlans(True) FAIL > testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL > testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR > testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR > testSetupNetworksAddOverExistingBond(False) ERROR > testSetupNetworksAddOverExistingBond(True) ERROR > testSetupNetworksAddVlan(False) ERROR > testSetupNetworksAddVlan(True) FAIL > testSetupNetworksConnectivityCheck FAIL > testSetupNetworksConvertVlanNetBridgeness ERROR > testSetupNetworksDelOneOfBondNets ERROR > testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow > tests are disabled > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow > tests are disabled > testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL > testSetupNetworksMtus(False) ERROR > testSetupNetworksMtus(True) SKIP: This > test fails because the 2 different networks are getting configured with the > same MTU. The test should assert that the reported MTUs are equal to the > requested ones. > testSetupNetworksMultiMTUsOverBond(False) ERROR > testSetupNetworksMultiMTUsOverBond(True) FAIL > testSetupNetworksMultiMTUsOverNic(False) ERROR > testSetupNetworksMultiMTUsOverNic(True) FAIL > testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL > testSetupNetworksNicless FAIL > testSetupNetworksNiclessBridgeless ERROR > testSetupNetworksOverDhcpIface ERROR > testSetupNetworksOverExistingBridge FAIL > testSetupNetworksReconfigureBridge FAIL > testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR > testSetupNetworksRemoveSlavelessBond ERROR > testSetupNetworksResizeBond(False) ERROR > testSetupNetworksResizeBond(True) ERROR > testSetupNetworksStableBond(False) ERROR > testSetupNetworksStableBond(True) ERROR > testStaticNetworkConfig((4, 6)) FAIL > testStaticNetworkConfig((4,)) FAIL > testStaticNetworkConfig((6,)) FAIL > testStaticSourceRouting(False) ERROR > testStaticSourceRouting(True) FAIL > testTwiceAdd(False) ERROR > testTwiceAdd(True) ERROR > testVolatileConfig(False) ERROR > testVolatileConfig(True) ERROR > test_drop_initial_bond_slaves_ip_config FAIL > test_getVdsStats ERROR > test_remove_bond_under_network FAIL > test_remove_initial_network_nic_ip_config FAIL > test_rollback FAIL > test_setupNetworks_bond_with_custom_option ERROR > test_setupNetworks_on_external_bond ERROR > test_setupNetworks_on_external_vlaned_bond ERROR > functional.networkTestsOVS.OVSNetworkTest > testHonorBlockingDhcp SKIP: Slow > tests are disabled > testLowerMtuDoesNotOverride OK > testNoBridgeLeftovers FAIL > testSelectiveRestoreIgnoresVdsmRegParams FAIL > testSetupNetworkOutboundQos(True) OK > testSetupNetworksAddBadParams(True) OK > testSetupNetworksAddBondWithManyVlans(True) FAIL > testSetupNetworksAddDelBondedNetwork(True) FAIL > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL > testSetupNetworksAddDelDhcp(True, (4,)) FAIL > testSetupNetworksAddDelDhcp(True, (6,)) FAIL > testSetupNetworksAddManyVlans(True) OK > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL > testSetupNetworksAddVlan(True) FAIL > testSetupNetworksConnectivityCheck OK > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow > tests are disabled > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow > tests are disabled > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL > testSetupNetworksMtus(True) SKIP: This > test fails because the 2 different networks are getting configured with the > same MTU. The test should assert that the reported MTUs are equal to the > requested ones. > testSetupNetworksMultiMTUsOverBond(True) FAIL > testSetupNetworksMultiMTUsOverNic(True) FAIL > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL > testSetupNetworksNicless OK > testSetupNetworksOverExistingBridge OK > testSetupNetworksReconfigureBridge FAIL > testStaticNetworkConfig((4, 6)) FAIL > testStaticNetworkConfig((4,)) FAIL > testStaticNetworkConfig((6,)) FAIL > testStaticSourceRouting(True) FAIL > test_drop_initial_bond_slaves_ip_config FAIL > test_ovirtmgmtm_to_ovs FAIL > test_remove_bond_under_network SKIP: OVS > does not support bridgeless networks > test_remove_initial_network_nic_ip_config OK > test_rollback FAIL > > > On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <[email protected]> > wrote: >> >> I like the idea but I have the same feelings as Francesco. I think that we >> need to make sure that functional tests for each vertical are stable before >> enabling this process. >> >> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <[email protected]> wrote: >>> >>> adding also infra team for visibility on the change in CI. >>> also inline. >>> >>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <[email protected]> >>> wrote: >>>> >>>> ----- Original Message ----- >>>> > From: "Yaniv Bronheim" <[email protected]> >>>> > To: [email protected], "Francesco Romani" <[email protected]>, "Nir >>>> > Soffer" <[email protected]>, "Piotr Kliczewski" >>>> > <[email protected]> >>>> > Cc: "danken" <[email protected]>, "David Caro" <[email protected]>, >>>> > "Eyal Edri" <[email protected]> >>>> > Sent: Thursday, December 10, 2015 6:46:37 PM >>>> > Subject: Automation CI for vdsm >>>> >>>> [...] >>>> > We want to allow developers to trigger the script once reviews and >>>> > verification are ready (last step before merge). To do so we agreed to >>>> > add >>>> > Continues Integration flag for each vdsm patch. >>> >>> >>> This flag will be called 'Workflow' or we can name it otherwise, we just >>> need to choose what makes sense. >>> David/Yaniv - Please correct me if I'm wrong. >>> >>>> >>>> Once this flag will be >>>> > signed with +1 it will trigger Jenkins CI to run the check-merged >>>> > script >>>> > (adding new button to gerrit is not an option - you can image that >>>> > flag as >>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail >>>> > we'll get -1 and once new patchset is ready the developer will remove >>>> > the >>>> > +1 and add it back to the Continues Integration flag to re-trigger the >>>> > job. >>>> > >>>> > Please ack the process before we move on with that >>>> >>>> Sounds good, even though I'm a little scared (just gut feeling, no >>>> evidence >>>> whatsoever) that this could add even more complexity and fragility to >>>> the jenkins >>>> fleet. >>>> >>>> In the long run, when this is reliable, it will help greatly. >>>> In the short term, I'm scared because this can lead to false positives >>>> and bogus >>>> failures. >>>> >>>> Let me stress I don't have concrete item to share or specific flaws. >>>> >>>> As action item on me, I will find some time next week to check virt >>>> functional tests, >>>> to see if they need some fixes, work reliably and so forth >>>> >>>> > The patch for those scripts still under review and testing - >>>> > https://gerrit.ovirt.org/#/c/48268 >>>> >>>> Will review asap. >>>> >>>> -- >>>> Francesco Romani >>>> RedHat Engineering Virtualization R & D >>>> Phone: 8261328 >>>> IRC: fromani >>> >>> >>> >>> >>> -- >>> Eyal Edri >>> Supervisor, RHEV CI >>> EMEA ENG Virtualization R&D >>> Red Hat Israel >>> >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> > > > > -- > Yaniv Bronhaim. _______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
