Quality Summary ------------------------------------------- QA complete Crosswalk App Tools Testing based on Commit ID 625835539744dd61dffc95f40586998abec674e3. Glad to see that newly added feature: manifest orientation, offline operation, crosswalk command line args, packaging of extensions, required minimum API target version on Android works fine. However, One major issue found: Invalid app has been built on iOS platform(XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233>),two regression issue found: The APK size is incorrect when the 'xwalk_android_webp' set the correct value(REG XWALK-5264<https://crosswalk-project.org/jira/browse/XWALK-5264>), Fail to remove project when xwalk_app_version is set to invalid value(REG XWALK-5255<https://crosswalk-project.org/jira/browse/XWALK-5255>). Additionally, three issues filed: Build project success when the display or orientation set the invalid value(XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262>), project success when the webp format set out of the range(XWALK-5254), Fail to build the success with the project path on Mac OS(XWALK-5263<https://crosswalk-project.org/jira/browse/XWALK-5263>). The overall test results:
Test Scope Android (Host:Ubuntu 14 ) Android (Host:Windows) Deepin (Host:Deepin) iOS (Host:OS X) Comments Total TCs Pass Rate Diff W/last commit ID(*) Total TCs Pass Rate Diff W/last commit ID(*) Total TCs Pass Rate Diff W/last commit ID(*) Total TCs Pass Rate Diff W/last commit ID(*) Crosswalk App Tools/CLI (Embedded x86) 120 83% -9% 120 83% -9% 38 89% +2% 39 82% -9% Bugs on Android: XWALK-3779<https://crosswalk-project.org/jira/browse/XWALK-3779>, XWALK-3782<https://crosswalk-project.org/jira/browse/XWALK-3782>, XWALK-4056<https://crosswalk-project.org/jira/browse/XWALK-4056>, XWALK-5254<https://crosswalk-project.org/jira/browse/XWALK-5254>, XWALK-5255<https://crosswalk-project.org/jira/browse/XWALK-5255>, XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262>, XWALK-5264<https://crosswalk-project.org/jira/browse/XWALK-5264>, Bugs on Linux: XWALK-4013<https://crosswalk-project.org/jira/browse/XWALK-4013>, XWALK-4785<https://crosswalk-project.org/jira/browse/XWALK-4785>, XWALK-5256<https://crosswalk-project.org/jira/browse/XWALK-5256>, XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262> Bugs on iOS: XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262>,XWALK-5263<https://crosswalk-project.org/jira/browse/XWALK-5263>,XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233> Ubuntu/Windows pass rate got downgrade due to case increase and fail reason due to due to new issue XWALK-5254<https://crosswalk-project.org/jira/browse/XWALK-5254>, XWALK-5255<https://crosswalk-project.org/jira/browse/XWALK-5255>, XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262>, XWALK-5264<https://crosswalk-project.org/jira/browse/XWALK-5264>, OS X pass rate got downgrade due to case increase and fail reason due to due to new issue XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262>,XWALK-5263<https://crosswalk-project.org/jira/browse/XWALK-5263> and one regression issue XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233> Deepin pass rate got improvement due to case increase Crosswalk App Tools/CLI (Embedded arm) 120 83% -9% 120 83% -9% Crosswalk App Tools/CLI (Shared x86) 120 83% -9% 120 83% -9% Crosswalk App Tools/CLI (Shared arm) 120 83% -9% 120 83% -9% Crosswalk App Tools/Common Module(Embedded x86) 103 100% -- 103 100% -- 54 100% -- 54 100% -- Got good quality Crosswalk App Tools/Common Module(Embedded arm) 103 100% -- 103 100% -- Crosswalk App Tools/Common Module(Shared x86) 103 100% -- 103 100% -- Crosswalk App Tools/Common Module(Shared arm) 103 100% -- 103 100% -- Crosswalk App Tools/Sample Apps 2 100% -- 2 100% -- 2 100% -- 2 0% -100% Got good quality on Linux,Windows,Deepin. OS X pass rate got downgrade due to regression issue XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233> Crosswalk App Tools/Usecase 2 100% -- 2 100% -- 1 100% -- 2 0% -100% Got good quality on Linux,Windows,Deepin. OS X pass rate got downgrade due to regression issue XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233> Total 896 81% -- 896 81% -- 95 96% -1% 97 89% -8% (*) instead of the previous test commit id: 7a767b913a729e16325e36ebeb9b42eed9b234b0 New Issues: ----------------------- XWALK-5233<https://crosswalk-project.org/jira/browse/XWALK-5233> P1 [app-tools] Invalid app has been built on iOS platform XWALK-5264<https://crosswalk-project.org/jira/browse/XWALK-5264> P2 [REG][app-tools]The APK size is incorrect when the 'xwalk_android_webp' set the correct value XWALK-5263<https://crosswalk-project.org/jira/browse/XWALK-5263> P2 Fail to build the success with the project path on Mac OS XWALK-5262<https://crosswalk-project.org/jira/browse/XWALK-5262> P2 Build project success when the display or orientation set the invalid value XWALK-5255<https://crosswalk-project.org/jira/browse/XWALK-5255> P2 [REG][app-tools]Fail to remove project when xwalk_app_version is set to invalid value XWALK-5254<https://crosswalk-project.org/jira/browse/XWALK-5254> P3 Build project success when the webp format set out of the range Verified Issues: ----------------------- XWALK-5135<https://crosswalk-project.org/jira/browse/XWALK-5135> [app-tools] No support for setting xwalk command line args XWALK-5049<https://crosswalk-project.org/jira/browse/XWALK-5049> [app-tools] Fail to build project when set webp conversion from manifest XWALK-4129<https://crosswalk-project.org/jira/browse/XWALK-4129> Incorrect help info is displayed on linux platform Crosswalk-app-tools Repositories: ----------------------- https://github.com/crosswalk-project/crosswalk-app-tools Release Tag or Commit ID: 625835539744dd61dffc95f40586998abec674e3 Debian Backend: https://github.com/crosswalk-project/crosswalk-app-tools-deb Release Tag or Commit ID: c5212e631dc37807be394dd2c86fa0dad9c42767 iOS Backend: https://github.com/crosswalk-project/crosswalk-app-tools-ios Release Tag or Commit ID: 7054a66fa7c0c48d9be12a7ced4645a6d62a7f3d Test Suite Repositories: ----------------------- Command-line Interface: https://github.com/crosswalk-project/crosswalk-test-suite/tree/master/apptools/apptools-android-tests https://github.com/crosswalk-project/crosswalk-test-suite/tree/master/apptools/apptools-ios-tests https://github.com/crosswalk-project/crosswalk-test-suite/tree/master/apptools/apptools-linux-tests https://github.com/crosswalk-project/crosswalk-test-suite/tree/master/apptools/apptools-sampleapp-tests https://github.com/crosswalk-project/crosswalk-test-suite/blob/master/usecase/usecase-apptools-tests/ Common module: https://github.com/crosswalk-project/crosswalk-app-tools/tree/master/test Crosswalk Binary on Android: https://download.01.org/crosswalk/releases/crosswalk/android/beta/15.44.384.12/crosswalk-15.44.384.12.zip Test Environment: ----------------------- Android host: Ivy Bridge Desktop, Ubuntu 14, Windows 8 Android device: Android IA 4.2.2 ZTE Geek, Memopad 8, Nexus 4 iOS host: MacBook Pro 10.10 with XCode 6.3 iOS device: iPad mini 8.2 iOS backend: https://github.com/crosswalk-project/crosswalk-app-tools-ios Linux/Deepin desktop: Deepin 2014.2 (http://www.linuxdeepin.com/download.cn.html) x86_64 with Chinese Linux/Deepin backend:https://github.com/crosswalk-project/crosswalk-app-tools-deb Thanks, Crosswalk QA team
_______________________________________________ Crosswalk-dev mailing list [email protected] https://lists.crosswalk-project.org/mailman/listinfo/crosswalk-dev
