I too was speculating, but if there’s anything to it not just the OS but the available primary and secondary storage would presumably play a role, so it would be down to the individual computers and how they are configured. I suspect you are more generous with your own machines than the taxpayer is with the ones at NASA.
Mike ------------------------------------------------------------------------------------------------------------------------------------ Michael Stark [email protected] (301) 286-8571 "I don't give them hell, I tell the truth and they think it's hell!" -- Harry S. Truman ------------------------------------------------------------------------------------------------------------------------------------ From: Darrel Conway <[email protected]> Organization: Thinking Systems, Inc. Reply-To: Darrel Conway <[email protected]> Date: Friday, September 21, 2018 at 16:08 To: "Stark, Michael E. (GSFC-5830)" <[email protected]>, "Hughes, Steven P. (GSFC-5950)" <[email protected]>, John McGreevy <[email protected]>, "Cooley, D S. (GSFC-5950)" <[email protected]>, "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [Gmat-buildtest] GMT-6055-GMATFunctionLargeArrayCrash That is what I had thought as well. This test uses a big chunk of memory. The test system for a nightly run (presumably) does as well, and the combination is more than the run can bear. Speculation, of course, I have no evidence (but then, I also don't see the issue on Linux...) On 09/21/2018 09:59 AM, Stark, Michael E. (GSFC-5830) wrote: I had it pass on Mac today, after I had moved and deleted over 100GB of old test builds & results. I think this may point to a memory issue, but it would have to be memory as used by the test system as a whole, not this individual script, which runs just fine on its own. Thoughts? Mike ------------------------------------------------------------------------------------------------------------------------------------ Michael Stark [email protected]<mailto:[email protected]> (301) 286-8571 "I don't give them hell, I tell the truth and they think it's hell!" -- Harry S. Truman ------------------------------------------------------------------------------------------------------------------------------------ From: Darrel Conway <[email protected]><mailto:[email protected]> Reply-To: Darrel Conway <[email protected]><mailto:[email protected]> Date: Friday, September 21, 2018 at 11:22 To: "Hughes, Steven P. (GSFC-5950)" <[email protected]><mailto:[email protected]>, John McGreevy <[email protected]><mailto:[email protected]>, "Stark, Michael E. (GSFC-5830)" <[email protected]><mailto:[email protected]>, "Cooley, D S. (GSFC-5950)" <[email protected]><mailto:[email protected]>, "[email protected]"<mailto:[email protected]> <[email protected]><mailto:[email protected]> Cc: "[email protected]"<mailto:[email protected]> <[email protected]><mailto:[email protected]> Subject: Re: [Gmat-buildtest] GMT-6055-GMATFunctionLargeArrayCrash Why would it pass on Linux, then? Here's the .tc file contents that I see: Bugs: [] Categories: [System] Requirements: [] OutputFiles: [ { File: GMT-6055-GMATFunctionLargeArrayCrash.report, Truth: GMT-6055-GMATFunctionLargeArrayCrash.truth, Comparator: PercentDiffComparator, RegressionComparator: FileDiffComparator, Tolerances: [1e-10], } ] I also do not have a FileDiffComparator. - DJC On 09/21/2018 08:00 AM, Hughes, Steven P. (GSFC-5950) wrote: I suspect that is a typo in .tc the file. I created that test, and I do not have a comparator named FileDiffComparator on my machine that I forgot to commit. You can change the comparator name and commit. From: gmat-nav <[email protected]><mailto:[email protected]> On Behalf Of John McGreevy Sent: Friday, September 21, 2018 10:46 AM To: Stark, Michael E. (GSFC-5830) <[email protected]><mailto:[email protected]>; Cooley, D S. (GSFC-5950) <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [gmat-nav] [Gmat-buildtest] GMT-6055-GMATFunctionLargeArrayCrash The GMT-6055-GMATFunctionLargeArrayCrash has a separate regression comparator which I've been able to reproduce the crash in. There's two things going on here, one is why it's hard to reproduce, and the other is why it's crashing: 1. This test uses a regression comparator, instead of just a truth comparator, which wouldn't run until I added 'regression' to my RunDef.Comparisons, along with setting a folder name to RunDef.RegressionBuild. 2. The regression comparator is the FileDiffComparator, which does not exist in the repo. If I change this comparator to the DiffComparator, the test passes. Was the comparator just renamed and the .tc file not updated? If so, I can push the updated name to the test repo. John On 9/17/2018 13:55, Stark, Michael E. (GSFC-5830) via Gmat-buildtest wrote: The answer is "a while", I notice the repetitiveness of it about a month ago, but I didn't trace back to see how far before that. I suspect it matches how far back it goes on Windows; I can't prove it though. ------------------------------------------------------------------------------------------------------------------------------------ Michael Stark [email protected]<mailto:[email protected]> (301) 286-8571 "I don't give them hell, I tell the truth and they think it's hell!" -- Harry S. Truman ------------------------------------------------------------------------------------------------------------------------------------ On 9/17/18, 13:17, "Cooley, D S. (GSFC-5950) via Gmat-buildtest" <[email protected]><mailto:[email protected]> wrote: Hi Mike, Do you have an idea on how long its been failing on the Mac? steve -----Original Message----- From: Cooley, D S. (GSFC-5950) Sent: Monday, September 17, 2018 12:58 PM To: '[email protected]<mailto:[email protected]>' <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: RE: [Gmat-buildtest] GMT-6055-GMATFunctionLargeArrayCrash I focus on the Nav tests but from a look back, it appears to fail every run for the last 3 months or so. -----Original Message----- From: Darrel Conway <[email protected]><mailto:[email protected]> Sent: Monday, September 17, 2018 11:56 AM To: Cooley, D S. (GSFC-5950) <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Gmat-buildtest] GMT-6055-GMATFunctionLargeArrayCrash Question: when you say "It seems to fail a lot on the nightly runs.", is that a recent occurrence, something that has happened ever since you took over the test machine chores, or something else? I seem to remember looking at this test a while ago and thinking it was a stack size issue on Windows (the comparator running out of memory for the compare, given other data buffered in memory during a full test run). But if that were the case, I'd expect it to work fine on Mac, unless MATLAB is somehow cutting the default stack size down. - Darrel On 09/15/2018 11:24 AM, Cooley, D S. (GSFC-5950) via Gmat-buildtest wrote: > Good News. We only had one Windows regression test fail: > > GMT-6055-GMATFunctionLargeArrayCrash > > The test passes locally. It seems to fail a lot on the nightly runs. Anyone have any ideas on how to make this test more stable? > > > > -----Original Message----- > From: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> > Sent: Saturday, September 15, 2018 2:52 AM > To: Cooley, D S. (GSFC-5950) <[email protected]><mailto:[email protected]>; > [email protected]<mailto:[email protected]> > Subject: Test results: 2018-09-14 > (GS580W-GMAT-T6/Win7-64/GMAT-64/M2016a/VS2013) > > ----------------------------------------------------------------- > Summary > ----------------------------------------------------------------- > System configuration: Windows 10/MATLAB R2017a/VS 2013 Tested build: > 2018-09-14 Regression build: 2018-09-13 > > Total number of scripts: 13853 (+83 deferred) Percent Passed: 99.99% New issues: 12 > Output changes: 12 > New truth failures: 0 > New crashes: 0 > New script errors: 0 > New comparator errors: 0 > Fixed issues: 26 > Total issues: 1 > > Test system warnings: 39 > New deferred issues: 11 > Fixed deferred issues: 0 > Total deferred issues: 55 > > Run Time: 13 hours, 45 minutes, 49 seconds > > ----------------------------------------------------------------- > New Issues > ----------------------------------------------------------------- > New truth failures: > None > > Output changes (but still pass): > BatchEstimator_ReportFile_MeasurementModeling > Ephem_GMAT_Code500_ACE_OpsPrototype_v13 > Ephem_GMAT_Code500_CoordinateSystem > Ephem_GMAT_Code500_ManeuverAtEnd > Ephem_GMAT_Code500_OutputFormat > Ephem_GMAT_Code500_TimeInteractions > Ephem_GMAT_Code500_TimeInteractions > Ephem_GMAT_Code500_TimeInteractions > Ephem_GMAT_Code500_Toggle > Ex_FirstVersionNumericSettings > SNOPT_OverrideSpecsFile > SNOPT_SpecsFile > > Output changes (but still fail): > None > > New crashes/hangs/etc.: > None > > New script errors: > None > > New comparator errors: > None > > ----------------------------------------------------------------- > Fixed Issues > ----------------------------------------------------------------- > Fixed truth failures: > fmincon_Report > fmincon_ShowProgress > Optimize_fmincon_cvg_status > Optimize_fmincon_cvg_status > Vary_optimizer_fmincon__AdditiveScaleFactor_Variable > Vary_optimizer_fmincon_Additive_and_Multiplicative_ScaleFactor_RealNumber > Vary_optimizer_fmincon_AdditiveScaleFactor_ArrayElement > Vary_optimizer_fmincon_AdditiveScaleFactor_RealNumber > Vary_optimizer_fmincon_MultiplicativeScaleFactor_ArrayElement > Vary_optimizer_fmincon_MultiplicativeScaleFactor_RealNumber > Vary_optimizer_fmincon_MultiplicativeScaleFactor_Variable > > Fixed crashes/hangs/etc.: > None > > Fixed script errors: > Vary_optimizer_fmincon_InitialGuess_ForLoopVariable_2 > Vary_optimizer_fmincon_InitialGuess_ForLoopVariable > Vary_optimizer_fmincon_Perturbation_ArrayElement > Vary_optimizer_fmincon_Perturbation_RealNumber > Vary_optimizer_fmincon_Perturbation_Variable > Vary_optimizer_fmincon_UpperBound_ArrayElement > Vary_optimizer_fmincon_UpperBound_Variable > Vary_optimizer_fmincon__InitialValue_Array > Vary_optimizer_fmincon__InitialValue_RealNumber > Vary_optimizer_fmincon__InitialValue_Variable > Vary_optimizer_fmincon__LowerBound_Array > Vary_optimizer_fmincon__LowerBound_RealNumber > Vary_optimizer_fmincon__LowerBound_Variable > completed_steps_1_thru_10_of_FRC_1_using_fmincon_Optimizer > fmincon_ConstraintTolerance > > Fixed comparator errors: > None > > ----------------------------------------------------------------- > All Known Issues > ----------------------------------------------------------------- > Crashes/hangs/etc.: > None > > Script errors: > None > > Comparator errors: > GMT-6055-GMATFunctionLargeArrayCrash: (reg) > ERROR MESSAGE: Undefined function > 'comparator.FileDiffComparator.Compare' for input arguments of type > 'char'. > > Truth failures: > None > > ----------------------------------------------------------------- > New Issues (Deferred) > ----------------------------------------------------------------- > New truth failures (deferred): > None > > Output changes (but still pass) (deferred): > Ephem_GMAT_Code500_InterpolationOrder [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_InterpolationOrder [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_StepSize [GMT-4112] > > Output changes (but still fail) (deferred): > Ephem_GMAT_Code500_InterpolationOrder [pos err 0.0173914, tol 0.002] [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_InterpolationOrder [pos err 0.0463976, tol 0.002] [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_StepSize [pos err 305.165, tol 0.002] [GMT-4112] > EphemStressTests_Default_and_ABM_Integrator_Code500_60Sec_MultipleSCs_ForwardDirection [pos err 15.2466, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_ForwardDirection [pos err 9.57664, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_SolverTypes_ForwardDirect [pos err 14.3693, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_ForwardDirection [pos err 0.000216981, tol 1e-12] [GMT-4112] > EphemStressTests_PD78_ABM_Prop_Code500_60Sec_Step_ForwardDirection > [pos err 21.5632, tol 1e-12] [GMT-4112] > > New crashes/hangs/etc. (deferred): > None > > New script errors (deferred): > None > > New comparator errors (deferred): > None > > ----------------------------------------------------------------- > Fixed Issues (Deferred) > ----------------------------------------------------------------- > Fixed truth failures (deferred): > None > > Fixed crashes/hangs/etc. (deferred): > None > > Fixed script errors (deferred): > None > > Fixed comparator errors (deferred): > None > > ----------------------------------------------------------------- > All Known Issues (Deferred) > ----------------------------------------------------------------- > Crashes/hangs/etc. (deferred): > GMAT_LunarTransfer_AllPlanets_DE421_BS > GMAT_MarsTransfer_4_BS > > Script errors (deferred): > BatchEstimator_EstimationEpoch [GMT-5708] > Math_Shadowed_Builtins [GMT-3043] > Python_IO_String [5317] > ScriptLanguage_WhiteSpace_AfterArrayName [GMT-3522] > THF_Sim_Accel_ConfigD_None_NA_2 [GMT-6451] > THF_Sim_Accel_ConfigD_None_NA_3 [GMT-6451] > THF_Sim_Accel_ConfigD_None_NA_4 [GMT-6451] > THF_Sim_Accel_ConfigD_None_NA [GMT-6451] > THF_Sim_Thrust_CSF2_None_NA_ZeroTSF [GMT-6451] > TrkFile_SimEst_DSN_DS_Range_Solve_Bias_apriori [GMT-5709] > > Comparator errors (deferred): > GroundStation_Iono_Range_Kband (truth) [GMT-5644] > GroundStation_Iono_RangeRate_Kband (truth) [GMT-5644] > GroundStation_TropoIono_Range_Kband (truth) [GMT-5644] > > Truth failures (deferred): > AcceptFilter_Validation_RecordNumbers [#5 failed] [GMT-6034] > Achieve_Validation_LHS_Field [ptrn 1 failed] [GMT-3687] > Achieve_Validation_RHS_Field [no error or exception] [GMT-3687] > BatchEstimator_ReportFile_InitialState [#5 failed] [GMT-6412] > Ephem_GMAT_Code500_InterpolationOrder/Ephem_GMAT_Code500_InterpolationOrder_1.eph [pos err 0.0173914, tol 0.002] [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_InterpolationOrder/Ephem_GMAT_Code500_InterpolationOrder_3.eph [pos err 0.0463976, tol 0.002] [GMT-4075, GMT-4112] > Ephem_GMAT_Code500_StepSize/Ephem_GMAT_Code500_StepSize_1.eph [pos err 305.165, tol 0.002] [GMT-4112] > EphemStressTests_Default_and_ABM_Integrator_Code500_60Sec_MultipleSCs_ForwardDirection [pos err 15.2466, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_ForwardDirection [pos err 9.57664, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_SolverTypes_ForwardDirect [pos err 14.3693, tol 1e-12] [GMT-4077] > EphemStressTests_Default_Prop_Code500_60Sec_MixedStopTypes_LogicTypes_ForwardDirection [pos err 0.000216981, tol 1e-12] [GMT-4112] > EphemStressTests_PD78_ABM_Prop_Code500_60Sec_Step_ForwardDirection [pos err 21.5632, tol 1e-12] [GMT-4112] > EphemStressTests_SPK_Integrator_Code500_60Sec_ForwardDirection [pos err 0.000427977, tol 1e-12] [GMT-4112] > Epoch_LeapSecond_G2MJD [max mjd diff 1.000259s > tol 0.001000s] [2561] > Epoch_LeapSecond_In [max greg. diff 1.000005s > tol 0.001000s] [2561] > Epoch_LeapSecond_Out [max mjd diff 1.000146s > tol 0.001000s] [2561] > Formation_Validation_MissingComma [no error or exception] [GMT-3672] > GMAT_ISS_Earth_JGM2_0_0_NutationUpdateInterval [pos err 0.00081496, tol 1e-06] [GMT-2567] > GMAT_LEO_PropSynchronization_BS [] > GMAT_Molniya_EarthSunLuna_JGM2_JR_SRP_BS [pos err 0.0756285, tol 0.002] > Math_Inf [GMAT: 53 lines, Truth: 69 lines] [GMT-3195] > Minimize_Validation_Field [no warn or err] [GMT-3687] > Nonlinear_Validation_LHS_Field [no warn or err] [GMT-3687] > Nonlinear_Validation_RHS_Field [no warn or err] [GMT-3687] > Optimize_Yukon_Minimize_Height_on_circle_2 [diff 497, tol 0.001] > Optimize_Yukon_Minimize_Height_on_circle_4 [diff 497, tol 0.001] > RejectFilter_Validation_RecordNumbers [#5 failed] [GMT-6034] > ScriptLanguage_Comments_AfterControlFlow [] [GMT-3332] > ScriptLanguage_LineContinuation_BeginningOfLine [] [GMT-3756] > ScriptLanguage_LiteralArrays_String [] [GMT-3680] > ScriptLanguage_ReferenceArrays_ForceModel_7 [] [GMT-3541] > ScriptLanguage_Validation_Semicolons_1 [no error or exception] [GMT-3545] > Simulator_MoonOccultation [GMAT: 11299 lines, Truth: 11306 lines] [GMT-6134] > SpacecraftNavigation_Validation_Mode [#1 failed] [GMT-5841] > String_AssignNumericParam [] [GMT-2888] > THF_Sim_Accel_ConfigD_None_NA_FixedStep [ Observable Error: 100.8 m.] [FixedStep 1 second, Sim, constant thrust over entire interval, should it pass] > TrkFile_SimEst_TDRS_range_Solve_Cr [] [GMT-5416] > Vary_Validation_RHS_Field [no error or exception] [GMT-3687] > Write_Multiple_ReportFile_Concise_2 [#69 failed] [GMT-5801] > Write_Multiple_ReportFile_Verbose_2 [#69 failed] [GMT-5801] > > ----------------------------------------------------------------- > Test System Warnings > ----------------------------------------------------------------- > Passing tests tagged with bugs: > (To fix, remove bug ID from Bugs key in .tc file) > Comet_Validation_OrientationEpoch_3 [GMT-3252] > CoordSysTest_Luna_MJ2000Eq_To_ICRF [GMT-3227] > CoordinateSystem_Validation_Axes_4 [GMT-3155] > EphemPropagator_Code500_CentralBody_Jupiter [GMT-5795] > EphemPropagator_Code500_CentralBody_Saturn [GMT-5795] > EphemPropagator_Code500_ForwardProp_Interp [GMT-5760] > EphemPropagator_SatAssigmentToNewSatThenProp [GMT-3434] > EphemStressTests_Default_Prop_STK_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_ForwardDirection [GMT-6090] > EphemStressTests_Default_Prop_STK_60Sec_MixedStopTypes_LogicTypes_DiscontinuityTypes_SolverTypes_ForwardDirect [GMT-6090] > Ephem_GMAT_SPK_60sec_backward [GMT-5286] > Ephem_GMAT_SPK_DefaultIntegrationStep_User_Defined_LagrangeOrder > [GMT-5286] > Ephem_GMAT_STK_60sec_IterativeProcess_Optimizer [GMT-6090] > Ephem_GMAT_STK_60sec_IterativeProcess_Optimizer_NoBoundaries [GMT-6090] > Events_ContactLocator_Ephemeris_DE405AllPlanets [GMT-5377] > Events_ContactLocator_Ephemeris_DE424AllPlanets [GMT-5377] > Events_ContactLocator_Heo2 [GMT-5302] > Events_ContactLocator_UseEntireInterval_False_MissionSequence [GMT-5292] > Ex_GmatFunction [GMT-3604] > Formation_Validation_PropFormSatInTwoProps [GMT-3581] > GMAT_GEO_EarthSunMoon_0_0_0_Relativity [3405] > GMAT_GEO_EarthSun_0_0_0_Relativity [3405] > GMAT_ISS_EarthSunMoon_0_0_0_Relativity [3405] > GMAT_ISS_EarthSunMoon_0_0_0_Relativity_Cmd [3405] > GMAT_ISS_EarthSun_0_0_0_Relativity [3405] > Math_Validation_Cross_WrongDim2 [GMT-5676] > ScriptLanguage_Validation_DataTypes_ReferenceArray_Formation_5 [GMT-6064] > Simulator_Propagator_EarthPointMass_Drag [GMT-5417] > Simulator_Propagator_EarthPointMass_SRP_Drag [GMT-5417] > Simulator_Propagator_Sun_EM [GMT-6040] > Simulator_Propagator_Sun_EM_AllPlanets [GMT-6040] > Simulator_Propagator_Sun_EM_AllPlanets_SRP_GR [GMT-6040] > Simulator_Propagator_Sun_EM_GR [GMT-6040] > SpacePointParams_CelestialBody_EpochOutput [3939] > THF_AccelMass_CSC_None_None [GMT-5981] > THF_Accel_CSC_None_None [GMT-5981] > THF_ThrustMass_CSC_None_None [GMT-5981] > THF_Thrust_CSC_None_None [GMT-5981] > THF_Thrust_CSC_None_None_2 [GMT-5981] > Thruster_Finite_Burn_Permutations_Separate_Propagators_2 [GMT-3455] > > > > > _______________________________________________ > Gmat-buildtest mailing list > [email protected]<mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/gmat-buildtest > > -- Darrel J. Conway, Ph.D. Thinking Systems, Inc. Senior Scientist and CEO 437 W Thurber Road, Suite 6 Phone: (623) 298-4530 Tucson, AZ 85705 FAX: (520) 232-2533 www.thinksysinc.com<http://www.thinksysinc.com> Cell: (520) 425-3626 [email protected]<mailto:[email protected]> _______________________________________________ Gmat-buildtest mailing list [email protected]<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/gmat-buildtest _______________________________________________ Gmat-buildtest mailing list [email protected]<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/gmat-buildtest -- Darrel J. Conway, Ph.D. Thinking Systems, Inc. Senior Scientist and CEO 437 W Thurber Road, Suite 6 Phone: (623) 298-4530 Tucson, AZ 85705 FAX: (520) 232-2533 www.thinksysinc.com<http://www.thinksysinc.com> Cell: (520) 425-3626 [email protected]<mailto:[email protected]> -- Darrel J. Conway, Ph.D. Thinking Systems, Inc. Senior Scientist and CEO 6441 N Camino Libby Phone: (623) 298-4530 Tucson, AZ 85718 FAX: (520) 232-2533 www.thinksysinc.com<http://www.thinksysinc.com> Cell: (520) 425-3626 [email protected]<mailto:[email protected]>
_______________________________________________ Gmat-buildtest mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gmat-buildtest
