Hello everybody, we had a discussion about the process we could use for reviewing the incoming apps until an automated review mechanism is in place. Here's a quick summary. Please weigh in, so we can document this and start with the implementation.
Goal ==== The ultimate goal is to have a fully automated process, so that manual review is only the exception in those cases where there are needs for conflict resolution or upload revocation (e.g. failure to comply with the ToS). However, in the interim, we will need a team of human reviewers to approve app uploads until full automation has been implemented. Lessons learnt ============== * Human review does not scale very well. * Arduous task. * Ensuring Debian best practices are upheld is tiresome. * Relaxing review policy was only possible up to a small degree. * Feedback ping-pong exhausts everybody’s patience, both reviewers and developers. Requirements ============ * App upload should be a fully automated process * A human team of reviewers should only be required for maintenance and conflict resolution. * App review process: temporary human review period. Queue handling ============== Having automated review tools is a must. Review policy ------------- * ARB review guidelines https://wiki.ubuntu.com/AppReviewBoard/Review/Guidelines * Implementation idea: have a tool which downloads everything from unreviewed queue, run lint tool over all of them, spit out list of apps for approval and rejection. * David’s idea: “listadmin for click packages” (Daniel: +1) Checks we could run =================== * General: - Description + app details - Price of the app. - Do we want to check the screenshot? yes - Check icon/screenshot for noticable trademark infringement (ie, adobe icon or facebook icon, etc) - Description? yes - Contact line / support URL - copyright? * Submission - Tarball? Other file we don’t accept? - Submission should be defined click format only - no other option - Verify developer signature * Lint tool - Manifest - Size - Architectures - Security profile - are the specified perms supported by our tools/policy? (automatable) - do the specified perms make sense relative to description/developer justification? (manual review) * QA - App is installable - App is startable (perhaps just handle by user reviews? - this is how android does it) - App operates correctly - doesn’t scale, handle by user reviews * Non-automatable - Are the requested security features necessary? - Is it a demo app? - Are apps generating revenue? (In-app purchases.) Review tools ============ We wrote arb-lint (https://launchpad.net/arb-lint) and some of its functionality has been merged into lintian or could be used from there. (Mail from Niels Thykier, one of its Debian maintainers: https://lists.ubuntu.com/archives/app-review-board/2012-November/002569.html) Staffing of the review team =========================== Daniel: It’d be good if we could have working in shifts. So we’d have people assigned for each week day, much like the archive admins (https://wiki.ubuntu.com/ArchiveAdministration#Archive_days) do. Plan ahead ========== Daniel: it’d be good if we planned ahead and kept the period with manual review to an absolute minimum. This of course depends on how much the automated checking tools can do for us and how easily they can be integrated into an automatic review queue handler in the app store. Future review team: just deal with conflict resolution. What needs to be done to automate review ---------------------------------------- 1. Collect problematic cases during reviews, note them down, work on automating their reviews. 2. Accept uploads, put them into a queue. 3. Process queue with automatic review tools. 4. Feedback by email, available on web as well. Open questions ============== - Resolve namespacing discussion - Public open queue? (Developers might not enjoy this.) - Need to create better guidance on rejection of apps vs simply asking for changes to fix problems. - Current review process lacks methods for adequate informative dialogue. Comments and feedback welcome! Have a great day, Daniel -- Get involved in Ubuntu development! developer.ubuntu.com/packaging Follow @ubuntudev on identi.ca/twitter.com/facebook.com/G+ -- Mailing list: https://launchpad.net/~ubuntu-appstore-developers Post to : [email protected] Unsubscribe : https://launchpad.net/~ubuntu-appstore-developers More help : https://help.launchpad.net/ListHelp
