Hi all,
Sorry to jump in late. My general comment is : THANK YOU.
That all looks very good. A couple of things though:
- we already have some tests for hosted apps install/update.
- we already have some tests for unsigned packaged apps install.
- the "old" tests (in dom/tests/mochitests/webapps) should be converted
to the new style, but it's lower priority.
Jason, if you still have the list of bugs you skimmed through, maybe we
could add a keyword on them to track that we don't forget tests for all
of them.
Fabrice
On 05/30/2013 05:22 AM, David Clarke wrote:
> Jason,
> I believe your P1 list is in agreement, I have updated the list to reflect
> your concerns.
>
>
> Packaged Apps:
> * [P1] Install/launch/uninstall signed/unsigned packaged app
> * [P1] Install failure for packaged app not signed with type privileged
> (Security)
> * [P1] Install failure for packaged app with type certified (Security)
> * [P1] Cancel download of packaged app
> * [P1] Restart download of packaged app
> * [P>1] Download failure for packaged app for having a bad packaged app zip
>
> Updated Apps:
> * [P1] Updated Manifest
> * [P1] Updated Manifest with preloaded appcache
> * [P1] General pieces of download portions of API on install/update
> * [P1] Web activities app install/update
>
> Preloaded Appcache:
>
> Appcache:
> * [P1] Install/launch/uninstall hosted app with appcache
> * [P>1] Download failure for hosted app with appcache for running out of
> space
> * [P1] Restart download of hosted app with appcache
>
> Hosted Apps:
> * [P1] Install/launch/uninstall hosted app
> * [P1] Install failure for hosted app with type as privileged or certified
> * [P1] Cancel download of hosted app with appcache
> * [P>1] Download failure for hosted app with appcache for having a bad
> appcache manifest
>
>
>
>
> -David
>
> ----- Original Message -----
> From: "Jason Smith" <[email protected]>
> To: "David Clarke" <[email protected]>
> Cc: "Fernando Jiménez" <[email protected]>, "Fabrice Desre"
> <[email protected]>, "Andrew McKay" <[email protected]>, "Gregor Wagner"
> <[email protected]>, "dev-b2g" <[email protected]>, "Julien
> Wajsberg" <[email protected]>, "David Chan" <[email protected]>
> Sent: Tuesday, May 28, 2013 10:33:50 PM
> Subject: Re: Platfrom Apps Automation Testing
>
>
> Hi David,
>
> I think your feedback makes sense. Some final comments:
>
> When I refer to download API, I'm referring to what was originally defined
> here - https://bugzilla.mozilla.org/show_bug.cgi?id=790558 , now included in
> this file -
> http://hg.mozilla.org/mozilla-central/file/495b385ae811/dom/interfaces/apps/nsIDOMApplicationRegistry.idl
> . In this API, I'm meaning things such as the download attributes (e.g.
> downloadAvailable, downloading), the event handlers (e.g. ondownloadsuccess),
> and the download operations (e.g. checkForUpdate, download, cancelDownload).
>
> To address your feedback about priorities, I conducted a quick study of the
> blocker bugs (either basecamp+ or tef+) that showed up in core dom:apps.
> Here's the areas that came up as noticeable:
>
>
> * Web activities registration on app install and changes during app
> updates - evidence of regressions here a few times
> * Bad attribute state with download attributes (e.g. downloadAvailable)
> depending on the state of the app install/update was evident across a few
> blocker bugs
> * Assurance is needed to we cannot rename an app on an update, even
> through a locale override - a few blocker bugs were present in this area
> * On a successful app update, we need to ensure that none of the
> properties of the old version of the app is still present - some bugs present
> here
> * Some blocker bugs were evident in not finding out updates were
> available when they were via checkForUpdate and vice versa (finding that an
> update was available when it wasn't) - seen on a few bugs
> * Generally saw bugs on general cancel download & restart download cases
> (sanity flows)
> * Insufficient storage error at install or download time triggers
> correctly (evident on a few blocker bugs)
> * Sanity flow installing a privileged app to ensure it works (but nothing
> more than that appears to be critically needed)
> * Some bugs were evident with preinstalled apps in how they are installed
> and updated (this is a different flow to how the apps are installed from the
> web, but out of scope here since it's part of the build system)
> * A couple of bugs on mini-manifest to webapp manifest matching (certain
> properties such as name have to match on both manifests)
> * General bugs about the one app per origin rule
> * Correct app status & permissions setup on install of an app (although
> this is out of scope for this automation being targeted here)
>
>
> Based on the analysis above, I would say that the critical areas appear to be
> general pieces of the download portions of the API (attribute state,
> completed update, checkForUpdate, cancelDownload, restartDownload) is one
> area to definitely to target either during initial install vs. update of an
> existing app. For download errors, the only interesting area I see to
> probably get automation for is the insufficient storage case on install vs.
> download. For webapp & mini manifests, the only interesting areas I see is
> web activities, cannot rename apps on update, and mini-manifest to webapp
> manifest matching. I'd also slip sanity tests in for a privileged app. I'd
> also double check we've got enough coverage on the one app per origin rule
> management.
>
>
>
> That about paragraph then means:
>
>
> * [P1] General pieces of download portions of API on install/update
> * [P1] Sanity privileged app install/update test
> * [P1] Web activities app install/update
> * [P2] Insufficient storage error handling
> * [P2] Cannot rename apps on update
> * [P3] Mini-manifest to webapp manifest matching
> * [P3] One app per origin rule
>
>
> Note - Web activities hits the P1 level mainly because if that breaks, that
> can lead to the phone experiencing severe failures, given that many
> foundational pieces of the phone rely on web activities.
>
>
>
> Does that analysis align with your understanding?
> Sincerely,
> Jason Smith
>
> Desktop QA Engineer
> Mozilla Corporation https://quality.mozilla.com On 5/28/2013 9:31 PM, David
> Clarke wrote:
>
>
>
> Comments highlighted by section:
>
> Could you point to the spec of the "download" API ? I think I am unfamiliar ?
>
>
> * Update hosted app, hosted app with appcache, packaged app
> <--- updated hosted app, is just caused by updating the end point, updating
> of appcache is really an html5 feature, not necessarily an apps based feature
> so i don't believe this is in scope.
>
> Drilling at the specifics appcache isn't important for this, but ensuring the
> integration into the download API definitely is. That's a fundamental piece
> of the download portion of the API. So that's definitely in scope. If that
> breaks (which it has in the past), we're in trouble - including potentially
> breaking partner apps in the process.
>
> ** Generally the mechanism that is in place is that you call
> app.checkForUpdate() and if the downloadAvailable flag has been set, then you
> would call navigator.mozApps.install.
> If you were attempting to write automation, it doesn't really make sense to
> test the "Update" failure scenario as you are suggesting, as this is the same
> as the install Failure.
> I understand the scenario you are suggesting, but I think if you treat the
> update as being, checkForUpdate(), then that makes more sense.
> The general failure scenarios are all covered in the install Failure
> scenarios which I listed below as being important.
>
>
> * Fail to update hosted app, hosted app with appcache, packaged app
> <-- unsure what failure to update hosted app means?
>
> Failure to update means you couldn't complete the update due to an error. The
> download API on failure to update needs to fail gracefully, not cause data
> loss, and fail when you expect it to fail. An example test case would be
> ensuring that trying to update a hosted app with an app type of web -->
> certified should fail during download.
>
> ** Unsure what the download api is.
>
> <-- failure to update a hosted app with appcache should be covered by
> appcache testing.
>
> Actually, there's integration points in the download API that directly
> integrates with appcache logic. So appcache testing doesn't cover this
> entirely. You need to ensure that the download API integrates correctly with
> appcache. The specifics of appcache is out of scope though.
>
> <-- failure to update a packaged app is reasonable, but would be considered a
> P3. The failure scenario to update a packaged app, would be to have a bad
> packaged app / diretory structure.
>
> I don't think all error cases are low priority, especially because this plays
> a fundamental piece of the download API and certain error cases can have
> severe implications if those test cases don't fail correctly. Other example
> failure cases such as:
>
>
> ** General Expected functionality would be the primary goal, next iteration
> would be to re-review / prioritize.
> ** Is the anticipation such that apps will be poorly signed ? There are
> plenty of things that could go wrong, and I'm in agreement that those tests
> need to be worked on. However first goal is to make sure we have good general
> coverage across all areas.
>
> * Network connection loss
> * Bad signing
> * Incorrect app type on update (web --> privileged for hosted, web -->
> certified for hosted/packaged)
> * etc. (for the many things that can incorrectly happen on an update)
>
> Some of these bad updates have put us into a bad state in certain cases in
> the past. A key characteristic of the download API is being able to ensure
> that you know when you fail and expect a failure when you should get one,
> have an ability to recover from a restart, and not suffer from data loss.
> Error cases are actually quite common with packaged apps given it's
> complexity. If we don't fail when we should fail, that could be quite bad in
> certain cases - especially in the case if say, I allowed an update of a
> hosted app with an app type of web to certified. Or another example would be
> trying to update a signed privileged packaged app to another packaged app
> without a proper signature. There's security holes that open up if that ends
> up not failing.
> * Restart update of hosted app, hosted app with appcache, packaged app
> <-- unsure what an update of a hosted app means, are you saying a manifest
> update, or content update ?
>
> ** Some of this I agree with, however this is handled by permissions testing
> / david chan should be able to speak to this. But I imagine that is where
> this would fall. Would love to hear what he is doing in this regard as well.
>
> * [general hosted app case] Updating a hosted app without appcache means
> changing the app manifest
> * [appcache involved for hosted app] Updating the appcache for a hosted app
> preloading appcache
>
> (This doesn't look to be in scope for this round )
>
> * Other more higher level areas of analysis could focus on:
>
> * Webapp manifest analysis
> * Drilling at the specifics of the download API
> * Mini-manifest analysis
>
>
> I wouldn't necessarily throw this out without analyzing this. The automation
> here drills at the gecko API level, which if it doesn't work correctly, will
> cause end up affecting Gaia negatively by caveat with the many bugs we've
> seen. However, there's certainly areas of the API that end-users aren't
> necessarily affected by as much. The manifest pieces (webapp manifest &
> mini-manifest) additionally play an important role in how you install and
> update an app, so those flows are important to analyze (example really bad
> bug that happened - we had a bug where a certain app update that changed a
> web activity in the manifest on an update caused the phone to fail to
> startup). I'd still apply perspectives on these pieces to see what's needed
> here. Sincerely,
>
>
> ** Agreed with you here. But this test case does seem more of an edge case,
> than a target for automation.
>
>
>
>
>
>
> Jason Smith
>
> Desktop QA Engineer
> Mozilla Corporation https://quality.mozilla.com On 5/25/2013 12:37 AM, David
> Clarke wrote:
>
>
> Jason,
> Great list, went through and organized things somewhat into the areas
> outlined before.
> Areas that I was unsure as to the meaning of the test I left out. I believe
> Fernando is working on https://bugzilla.mozilla.org/show_bug.cgi?id=821589 ,
> so adding him directly to the thread.
>
>
>
>
> Packaged Apps:
> * Install/launch/uninstall packaged app
> * Install failure for packaged app not signed with type privileged
> * Install failure for packaged app with type certified
> * Download failure for packaged app for running out of space (P3)
> * Cancel download of packaged app
> * Restart download of packaged app
> * Download failure for packaged app for having a bad packaged app zip (P3)
> * Install/launch/uninstall/update signed privileged packaged app
>
> Updated Apps:
> * Updated Manifest
> * Updated Manifest with preloaded appcache
> * Updated packaged app install
> Preloaded Appcache:
>
> Appcache:
> * Install/launch/uninstall hosted app with appcache
> * Download failure for hosted app with appcache for running out of space (P3)
> * Restart download of hosted app with appcache
>
> Hosted Apps:
> * Install/launch/uninstall hosted app
> * Install failure for hosted app with type as privileged or certified
> * Cancel download of hosted app with appcache
> * Download failure for hosted app with appcache for having a bad appcache
> manifest (P3)
>
> Notifications:
> * Verification that notification of installs / upgrades / pause / cancel /
> restart events are propagated correctly
>
> Permissions:
>
>
> -------------------------
> Areas where I feel where we need more clarity before we add it tot he list.
>
>
>
>
> * Update hosted app, hosted app with appcache, packaged app
> <--- updated hosted app, is just caused by updating the end point, updating
> of appcache is really an html5 feature, not necessarily an apps based feature
> so i don't believe this is in scope.
>
> * Fail to update hosted app, hosted app with appcache, packaged app
> <-- unsure what failure to update hosted app means?
> <-- failure to update a hosted app with appcache should be covered by
> appcache testing.
> <-- failure to update a packaged app is reasonable, but would be considered a
> P3. The failure scenario to update a packaged app, would be to have a bad
> packaged app / diretory structure.
>
>
> * Restart update of hosted app, hosted app with appcache, packaged app
> <-- unsure what an update of a hosted app means, are you saying a manifest
> update, or content update ?
>
> (This doesn't look to be in scope for this round )
>
> * Other more higher level areas of analysis could focus on:
>
> * Webapp manifest analysis
> * Drilling at the specifics of the download API
> * Mini-manifest analysis
>
>
> --David
> ----- Original Message -----
>
> From: "Jason Smith" <[email protected]> To: "David Clarke"
> <[email protected]> Cc: "Fabrice Desre" <[email protected]> , "Andrew
> McKay" <[email protected]> , "Gregor Wagner" <[email protected]> ,
> "Fernando Jiménez" <[email protected]> , "dev-b2g"
> <[email protected]> , "Julien Wajsberg" <[email protected]> Sent:
> Thursday, May 23, 2013 6:26:29 PM
> Subject: Re: Platfrom Apps Automation Testing
>
>
> My comments in summary:
>
>
> * I'm in agreement with your approach you are suggesting in #1
> * I'd suggest getting feedback from Julien on how Gaia integrates with the
> gecko layers to restart and cancel workflows. I'd suggest simulating what he
> does there. We should aim for sanity tests here, as the restart case
> (especially with packaged apps) is actually quite common (many problems can
> happen to cause a download to fail)
> * David Chan is already focusing on app permissions, so I'd keep that out of
> scope for now
> * The next step in analysis that I think would be helpful to get is to take
> the end to end analysis you've done and apply a gecko perspective on it
> referencing the underlying APIs involved
> * We need signed privileged packaged app test cases in this list (e.g.
> install a privileged app)
> * Mostly in agreement with the theme of the priorities. My comments
> specifically would suggest that these themes are important to consider in
> comparing with your list:
>
>
> * Install/launch/uninstall hosted app
> * Install failure for hosted app with type as privileged or certified
> * Install/launch/uninstall packaged app
> * Install failure for packaged app not signed with type privileged
> * Install failure for packaged app with type certified
> * Install/launch/uninstall hosted app with appcache
> * Download failure for packaged app for running out of space
> * Download failure for hosted app with appcache for running out of space
> * Cancel download of hosted app with appcache
> * Cancel download of packaged app
> * Restart download of hosted app with appcache
> * Restart download of packaged app
> * Download failure for packaged app for having a bad packaged app zip
> * Download failure for hosted app with appcache for having a bad appcache
> manifest
> * Update hosted app, hosted app with appcache, packaged app
> * Fail to update hosted app, hosted app with appcache, packaged app
> * Restart update of hosted app, hosted app with appcache, packaged app
> * Install/launch/uninstall/update signed privileged packaged app
> * Other more higher level areas of analysis could focus on:
>
>
> * Webapp manifest analysis
> * Drilling at the specifics of the download API
> * Mini-manifest analysis
> Sincerely,
> Jason Smith
>
> Desktop QA Engineer
> Mozilla Corporation https://quality.mozilla.com On 5/23/2013 3:08 PM, Jason
> Smith wrote:
>
>
> ==> Moving to dev-b2g (don't think there's anything confidential here).
> +Fernando
> +Julien
>
> Sincerely,
> Jason Smith
>
> Desktop QA Engineer
> Mozilla Corporation https://quality.mozilla.com On 5/23/2013 2:26 PM, David
> Clarke wrote:
>
>
> Navigator.mozApps Developers:
>
> I have been reviewing the automation suites that are currently in place, and
> wanted to propose a plan for organizing test cases going forward, and
> preparing for 1.1 / 1.2 features.
>
> #1 Cleanup / Stabilization:
> - http://mzl.la/Ywh6rL .There are currently some amount of tests which are
> failing intermittently of the current mozApps chrome test suite.
> The current mozApps test suite is not run on B2G Emulator as it is a
> mochitest-chrome based set of tests.
> My proposal would be to move the current mozApps test suite to a
> mochitest-plain based test suite, and use
> SpecialPowers.autoConfirmAppInstall, which was not available
> at the writing of the chrome based test.
> The double benefit here is that we will remove one area of intermittent
> failure, and also allow us to run the test suite on the B2G emulator.
> (imho this is the main cause of the test timeouts you are seeing in the above
> link )
>
>
> #2 Extend coverage:
> - Identify areas of coverage that are needed and high priority, and attempt
> to write test cases for them.
> - http://bit.ly/16aGkPS I have imported the manual test cases into the above
> google spreadsheet and attempted to rank by both priority / difficulty to get
> some sense of what test cases will be both easy / a high priority to test.
> - Good news is that packagedApp test cases are on the way, but there are lots
> of test cases that are not on the automation roadmap.
> Example:.... stop / cancel and restart packaged app installs.
> It would be good to separate the test cases listed above and discuss from a
> feasibility standpoint what is capable of being supported, and then make sure
> the hooks are in the platform that will allow for the automation to be
> written.
> The general priority listings I have organized around:
> P1: refers to Primary User Flows
> P2: Secondary user flows, a little further from the beaten path
> P3: Edge Cases, Error conditions, that are not explicitly part of a user
> story.
>
> I am only considering P1 issues for this round, but what I would like to do
> is to start breaking down the list further, and seeing what would be
> beneficial for gecko automation.
> If there are specific areas that we can agree on for automation, then we can
> focus around those specific use cases / test cases. The main areas for apps
> testing are listed below, please feel free to update with any thoughts or
> specific test cases that you think are relevant. You can also edit the google
> spreadsheet that is linked to from above, and add comments wherever
> necessary.
>
>
> Packaged Apps:
>
> Updated Apps:
>
> Preloaded Appcache:
>
> Appcache:
>
> Hosted Apps:
>
> Notifications:
>
> Permissions:
>
>
> Thanks all,
>
>
> --David
>
>
> _______________________________________________
> b2g-internal mailing list [email protected]
> https://mail.mozilla.org/listinfo/b2g-internal
>
--
Fabrice Desré
b2g team
Mozilla Corporation
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
