Naoki, yes, I believe that should probably be updated. It's OK to have multiple benchmarks, but for something that we are going to base blockers on, they should be well-established per release. For example:
v2.2 flame-kk 319MB v2.5 flame-kk 319MB flame-kk 1024MB aries 2048MB v3.0 aries 2048MB aries-l 2048 Not that these are the actual baseline benchmarks, just that they should be well-defined, and there should be a common denominator between 2 adjacent releases. Thanks, Eli Perelman On Tue, Sep 15, 2015 at 3:28 PM, Naoki Hirata <[email protected]> wrote: > It sounds like : https://wiki.mozilla.org/Firefox_OS/Performance/Benchmark > needs to be updated to include memory allocations? or are we talking about > the same thing? > > On Tue, Sep 15, 2015 at 12:01 PM, Eli Perelman <[email protected]> > wrote: > >> Raptor's performance tests also use 319MB as it is the only real baseline >> we tracked from v2.2. In order to compare the performance between the two >> branches of v2.2 and v2.5, we need a comparable baseline. We have been >> aggregating data for the Flame 1GB for future comparisons, but I would >> suggest proposing a new baseline sooner rather than later so Raptor can >> begin to aggregate data against that device and make it trackable between >> branches. >> >> Thanks, >> >> Eli Perelman >> >> On Tue, Sep 15, 2015 at 12:48 PM, Martijn <[email protected]> >> wrote: >> >>> On Tue, Sep 15, 2015 at 6:32 PM, Naoki Hirata <[email protected]> >>> wrote: >>> > >>> > I believe we still do care. >>> > >>> > Running Engineering build will make some difference as the 319 MB >>> takes into account the camera but not Marionette (adb + devtools) which >>> the engineering build runs. >>> > >>> > Having said that we run automation tests through jenkins, I'll check >>> with mwargers in regards to the issue to see if he already filed a bug or >>> not. >>> >>> >>> Yes, we're running the Gaia UI tests with the 319MB configuration and >>> we have various tests that are failing (intermittently) and disabled >>> because they just don't work with 319MB: >>> 1172167 [Flame][Cards View] Cards view loses the list of recently opened >>> apps >>> 1176502 Failure in test_homescreen_column_layout.py on the Flame >>> device, because Settings app gets killed >>> 1178859 test_browser_bookmark.py: "IOError: Connection to Marionette >>> server is lost." >>> 1181344 [Window Management] Settings will be a blank white screen when >>> returning from the "Built-in Keyboard" settings >>> 1188080 test_rocketbar_offline_behavior.py: "NoSuchWindowException: None" >>> 1189262 Failure in test_browser_share_link.py using Flame 319MB >>> 1204280 [Flame] Intermittent failure in test_sms_contact.py in >>> tap_send_sms using Flame 319MB >>> 1188603 [Settings][Ringtones] Share activity closes when attempting to >>> add custom ringtone >>> >>> I think these bugs need to be fixed, because they stand for lost >>> functionality in those low memory conditions. >>> >>> All the bugs currently out there, regarding supporting 319MB memory >>> configuration: >>> >>> https://bugzilla.mozilla.org/buglist.cgi?list_id=12549261&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=[319MB-Flame-Support] >>> >>> Regards, >>> Martijn >>> >>> >>> > >>> > >>> > We still have not resolved the series of initial bugs going into >>> 2.5/3.0: >>> > https://bugzilla.mozilla.org/show_bug.cgi?id=1155854 >>> > https://bugzilla.mozilla.org/show_bug.cgi?id=1162535 >>> > >>> > On Tue, Sep 15, 2015 at 2:13 AM, Ting-Yu Chou <[email protected]> >>> wrote: >>> >> >>> >> A bit off-topic. >>> >> >>> >> Raptor [1] just records a serious memory regression though, not sure >>> is it related to the issue you see. >>> >> >>> >> Ting >>> >> >>> >> [1] >>> http://raptor.mozilla.org/#/dashboard/script/apps-memory.js?device=flame-kk&branch=master&memory=319 >>> >> >>> >> On Tue, Sep 15, 2015 at 4:56 PM, Gabriele Svelto <[email protected]> >>> wrote: >>> >>> >>> >>> I was doing some testing in the past few days on a Flame using the >>> >>> 319MiB memory switch and found that our core apps are now almost >>> >>> unusable on it. Opening a single app seems to end up killing the >>> >>> homescreen and keeping two open at the same time is almost >>> impossible. >>> >>> >>> >>> So I've got two questions: the first one is, did we significantly >>> >>> regress memory consumption again? Or is it just gecko having grown >>> >>> significantly in the past few weeks making our minimum process size >>> >>> larger? The second question is obviously: do we still care about >>> 256MiB >>> >>> devices? Because if we still do then it looks like we're not in a >>> good >>> >>> spot for a 2.5 release running on those. If we don't care I suppose >>> it's >>> >>> fine though we should still keep our memory consumption under >>> control. >>> >>> If we move our minimum to 512MiB we'll have plenty of room and this >>> >>> might cause us to regress even further almost without noticing (*). >>> >>> >>> >>> I'm running an engineering build BTW, but I don't think this should >>> make >>> >>> much of a difference. >>> >>> >>> >>> Gabriele >>> >>> >>> >>> *) Unless the screen has a rather high resolution, in which case >>> 512MiB >>> >>> might still yield only a small amount of usable memory for apps as >>> most >>> >>> of it will go to the framebuffer and accompanying structures (layers >>> & co). >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> dev-fxos mailing list >>> >>> [email protected] >>> >>> https://lists.mozilla.org/listinfo/dev-fxos >>> >>> >>> >> >>> >> >>> >> _______________________________________________ >>> >> dev-fxos mailing list >>> >> [email protected] >>> >> https://lists.mozilla.org/listinfo/dev-fxos >>> >> >>> > >>> > >>> > _______________________________________________ >>> > dev-fxos mailing list >>> > [email protected] >>> > https://lists.mozilla.org/listinfo/dev-fxos >>> > >>> _______________________________________________ >>> dev-fxos mailing list >>> [email protected] >>> https://lists.mozilla.org/listinfo/dev-fxos >>> >> >> >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
