Ok, Nigel helped me in understanding the trace back time is not something we should look and the right way to dig through this problem is by looking at glusto logs. As per the last rebalance instance from the log I see the following:
volume rebalance: testvol_distributed-dispersed: success: Rebalance on testvol_distributed-dispersed has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 96c645c7-710c-4c4c-a434-4157cbb75753 2017-08-28 15:13:58,952 INFO (test_expanding_volume_when_io_in_progress) Successfully started rebalance on the volume testvol_distributed-dispersed 2017-08-28 15:13:58,952 INFO (test_expanding_volume_when_io_in_progress) Checking Rebalance status 2017-08-28 15:13:58,953 INFO (run) [email protected] (cp): gluster volume rebalance testvol_distributed-dispersed status 2017-08-28 15:13:58,953 DEBUG (_get_ssh_connection) Retrieved connection from cache: [email protected] 2017-08-28 15:13:58,993 INFO (_log_results) RETCODE ([email protected]): 1 2017-08-28 15:13:58,994 INFO (_log_results) STDERR ([email protected])... volume rebalance: testvol_distributed-dispersed: failed: error Again here I see the rebalance status failure was logged at 15:13:58,994 where as rebalance start was triggered at 15:13:58,952. @Shwetha - could you help me in understanding how do we log the rebalance status ret code in glusto log? On Wed, Aug 30, 2017 at 4:07 PM, Atin Mukherjee <[email protected]> wrote: > *14:15:57* # Start Rebalance*14:15:57* g.log.info("Starting > Rebalance on the volume")*14:15:57* ret, _, _ = > rebalance_start(self.mnode, self.volname)*14:15:57* > self.assertEqual(ret, 0, ("Failed to start rebalance on the volume > "*14:15:57* "%s", self.volname))*14:15:57* > g.log.info("Successfully started rebalance on the volume > %s",*14:15:57* self.volname)*14:15:57* *14:15:57* > # Check Rebalance status*14:15:57* g.log.info("Checking Rebalance > status")*14:15:57* ret, _, _ = rebalance_status(self.mnode, > self.volname)*14:15:57* self.assertEqual(ret, 0, ("Failed to get > rebalance status for the "*14:15:57* > > "volume %s", self.volname))*14:15:57* E AssertionError: ('Failed to get > rebalance status for the volume %s', 'testvol_distributed-dispersed') > > > The above is the snip extracted from https://ci.centos.org/view/ > Gluster/job/gluster_glusto/377/console > > If we had gone for rebalance status checks multiple times, I should have > seen multiple entries of rebalance_status failure or at least a difference > in time, isn't it? > > > On Wed, Aug 30, 2017 at 3:39 PM, Shwetha Panduranga <[email protected]> > wrote: > >> Case: >> >> 1) add-brick when IO is in progress , wait for 30 seconds >> >> 2) Trigger rebalance >> >> 3) Execute: 'rebalance status' ( there is no time delay b/w 2) and 3) ) >> >> 4) wait_for_rebalance_to_complete ( This get's the xml output of >> rebalance status and keep checking for rebalance status to be 'complete' >> for every 10 seconds uptil 5 minutes. 5 minutes wait time can be passed as >> parameter ) >> >> At every step we check the exit status of the command output. If the exit >> status is non-zero we fail the test case. >> >> -Shwetha >> >> On Wed, Aug 30, 2017 at 6:06 AM, Sankarshan Mukhopadhyay < >> [email protected]> wrote: >> >>> On Wed, Aug 30, 2017 at 6:03 AM, Atin Mukherjee <[email protected]> >>> wrote: >>> > >>> > On Wed, 30 Aug 2017 at 00:23, Shwetha Panduranga <[email protected]> >>> > wrote: >>> >> >>> >> Hi Shyam, we are already doing it. we wait for rebalance status to be >>> >> complete. We loop. we keep checking if the status is complete for '20' >>> >> minutes or so. >>> > >>> > >>> > Are you saying in this test rebalance status was executed multiple >>> times >>> > till it succeed? If yes then the test shouldn't have failed. Can I get >>> to >>> > access the complete set of logs? >>> >>> Would you not prefer to look at the specific test under discussion as >>> well? >>> _______________________________________________ >>> Gluster-devel mailing list >>> [email protected] >>> http://lists.gluster.org/mailman/listinfo/gluster-devel >>> >> >> >> _______________________________________________ >> Gluster-devel mailing list >> [email protected] >> http://lists.gluster.org/mailman/listinfo/gluster-devel >> > >
_______________________________________________ Gluster-devel mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-devel
