Hi all,
This is test report for ConnMan upstream 0.53.13. In this testing, we
ran 331 cases. 306 cases passed and 25 case failed because of known/new
bug. The pass rate is 92%. We found 2 new bugs, which were soon fixed.
The testing for more recent package shows state issues mostly got fixed,
but we see some new issues, which need to be double check next week.
Verified Bugs
=============================================
[437] ConnMan needs to connect hidden service automatically
http://bugs.meego.com/show_bug.cgi?id=437
New Bugs
=============================================
[2846] Switch between two hidden wifi services, sometimes one of them will
fail to be connected anymore
http://bugs.meego.com/show_bug.cgi?id=2846
[2850] ConnMan easily fail to connect to hidden WIFI services for the first
time after reboot
http://bugs.meego.com/show_bug.cgi?id=2850
Testing Environment
==============================================
Hardware: netbook ACER AspireOne NAV50/Eeepc 1005PE/AspireOne ZG5
Image: MeeGo release 1.0/Build 20100524.1
ConnMan: connman-0.53.13.g1c18278-1.1
3G: Unicom Sim card/Dell 5400
BT: AnyCom AP/CSR dongle (disable the onboard BT device)
Wireless AP: Linksys/Asus/Cisco 1252
Test Execution Summary
==============================================
>From Carrick UI:
Feature Total Pass Fail Comments
Wire 2 2
WiFi 55 50 5 #437 #610 #2640
3G 14 14
BT 5 5
VPN 7 6 1 #2318
Switch 6 6
Signal 13 13
Flight 5 5
Launch 6 6
Info 19 18 1 #1409
Interface 14 14
Device 12 11 1 #2310
NetworkSel 6 3 3 #2846
Autoconnect 18 13 5 #2850
Advance 9 8 1 #1409
State 13 9 4 #2461 #2465 #2467,#1993,#1490
Total 204 183 21
>From test script
Total PASS FAIL ToCheck
TotalNumCase 127 123 4 0
connman 127 123 4 0
bt 14 12 2 0 #2514
ethernet 14 13 1 0 #2514
wifi 57 57 0 0
Guest 12 12 0 0
Functional 11 11 0 0
security 22 22 0 0
Advanced 6 6 0 0
Negative 6 6 0 0
signal 27 27 0 0
connman 14 14 0 0
manager 8 8 0 0
service 5 5 0 0
offline 0 0 0 0
profile 6 5 1 0 #2513
flight 15 15 0 0
Bug refered here is bug at bugs.meego.com, for example, bug #437 refer to
http://bugs.meego.com/show_bug.cgi?id=437
For detailed test results, please see attached file
Quality indicator
==============================================
Feature Score Comments
WiFi Buggy Bug 437, 610, 1578, 2312, 2640
Intel WiMAX N/A QA does not have WiMAX test environment
Bluetooth PAN Good Bug 1378, 269, 2310,
3G data Buggy Bug 392, 1993, 1950, 2312, 2323, 2324 , 2503
Wired (LAN) Good
State Bad Online/ready state is confusing: Bug 1490,
157, 1993, 2461, 2465, 2467, 2514
Network Selection Good Bug 2846
wpa-suplicant D-BUS Good
profile management Good Bug 2513
manual switch connect Good
Wired/WiFi co-exexist Good
Remember device status Good
connect to Broadcast AP Good
connect to Hidden AP Buggy MeeGo bug 437, ConnMan needs to connect
hidden service automatically
Hidden Group AP Good
auto connect reboot Buggy Bug 2850
auto connect Offline Good
auto connect suspend Good
favorite connection Good
auto connectivity order Good
WEP 40/128-bit key Good
WEP 128-bit passphrase Good
WPA personal Good
WPA2 personal Good
Open system Good
Shared Key Good
Remember passwd Good
forgot password Good
AP security detect Good
auto del non-exist AP Good
VPN Buggy Bug 2318
Plugin to oFono Good
DNS Proxy Good
DNS support via DHCP Good
Static IP Good
PPP 3G N/A
WPA-enterprise Good
Bests
Jeff
ConnMan_3G/
NETWORK_3G_Coexist_001 pass
NETWORK_3G_Coexist_002 pass
NETWORK_3G_DeviceControl_001 pass
NETWORK_3G_DeviceControl_002 pass
NETWORK_3G_ICON_001 pass
NETWORK_3G_Item_001 pass
NETWORK_3G_Item_002 pass
NETWORK_3G_Item_003 pass
NETWORK_3G_Item_003 pass
NETWORK_3G_Service_001 pass
NETWORK_3G_Service_002 pass
NETWORK_3G_Service_003 pass
ConnMan_Advance/
CONNMAN_Advance_Hostname pass
CONNMAN_Advance_ProfileCreate pass
CONNMAN_Advance_ProfileName pass
CONNMAN_Advance_StaticIP_BT pass
CONNMAN_Advance_StaticIP_reboot pass
CONNMAN_Advance_StaticIP_WIFI pass
CONNMAN_Advance_StaticIP_Wired pass
CONNMAN_profile_delete pass
CONNMAN_Set_Nameservers_Negtive pass
CONNMAN_Set_ Nameservers_Positivepass
ConnMan_AutoConn/
CONNMAN_AutoConn_FavoriteSetting pass
CONNMAN_AutoConn_LostConn pass
CONNMAN_AutoConn_Reboot3G pass
CONNMAN_AutoConn_RebootWIFI pass
CONNMAN_AutoConn_RebootWired pass
CONNMAN_AutoConn_RememNoSecu pass
CONNMAN_Autoconn_RememWEP128 pass
CONNMAN_AutoConn_RememWEP64 pass
CONNMAN_Autoconn_RememWPAPSKAES pass
CONNMAN_Autoconn_RememWPAPSKTKIP pass
CONNMAN_Autoconn_HIDDEN_Rem NoSecubug2850
CONNMAN_Autoconn_HIDDEN_Rem WEP128bug2850
CONNMAN_Autoconn_HIDDEN_Rem WEP64bug2850
CONNMAN_Autoconn_HIDDEN_Rem WPAPSKAESbug2850
CONNMAN_Autoconn_HIDDEN_RemWPAPSKTKIP bug2850
CONNMAN_AutoConn_SuspendMem3G pass
CONNMAN_AutoConn_SuspendMemWIFI pass
CONNMAN_AutoConn_SuspendMemWired pass
ConnMan_BT/
CONNMAN_BT_CheckProp_Powered pass
CONNMAN_BT_Check_State pass
CONNMAN_BT_Connect_Invalidnetwork pass
CONNMAN_BT_Connect_PAN pass
CONNMAN_BT_List_PAN pass
ConnMan_Device/
CONNMAN_Device_3G pass
CONNMAN_Device_3G_Conn pass
CONNMAN_Device_3G_Reboot pass
CONNMAN_Device_BT pass
CONNMAN_Device_BT_Conn pass
CONNMAN_Device_BT_Reboot fail(#2310)
CONNMAN_Device_WIFI pass
CONNMAN_Device_WIFI_Conn pass
CONNMAN_Device_WIFI_Reboot pass
CONNMAN_Device_Wired pass
CONNMAN_Device_Wired_Conn pass
CONNMAN_Device_Wired_Reboot pass
ConnMan_Info/
CONNMAN_INFO_BtConnInfo pass
CONNMAN_INFO_BTDevInfo pass
CONNMAN_INFO_EthConnInfo pass
CONNMAN_INFO_EthDevInfo pass
CONNMAN_INFO_Exit_FlightMode pass
CONNMAN_INFO_ManagerStateBTOffline pass
CONNMAN_INFO_ManagerStateBTOnline pass
CONNMAN_INFO_ManagerStateEthOffline pass
CONNMAN_INFO_ManagerStateEthOnline pass
CONNMAN_INFO_ManagerStateWiFiOffline pass
CONNMAN_INFO_ManagerStateWiFiOnline pass
CONNMAN_INFO_MultiConnection pass
CONNMAN_INFO_NoConnList fail(#1490)
CONNMAN_INFO_ProfileDefault pass
CONNMAN_INFO_SetIPBT pass
CONNMAN_INFO_TurnOn_FlightMode pass
CONNMAN_INFO_WiFiConnInfo pass
CONNMAN_INFO_WiFiDevInfo pass
ConnMan_Launch/
CONNMAN_LAUNCH_boot pass
CONNMAN_LAUNCH_BT pass
CONNMAN_LAUNCH_dhclient pass
CONNMAN_LAUNCH_OneConnmand pass
CONNMAN_LAUNCH_reboot pass
CONNMAN_LAUNCH_WpaSupplicant pass
ConnMan_NetworkSel/
CONNMAN_NetworkSel_BroadcastNoSec pass
CONNMAN_NetworkSel_BroadcastWEP pass
CONNMAN_NetworkSel_BroadcastWPA pass
CONNMAN_NetworkSel_HideNoSec fail(#2846)
CONNMAN_NetworkSel_HideWEP fail(#2846)
CONNMAN_NetworkSel_HideWPA fail(#2846)
ConnMan_OfflineMode/
CONNMAN_Offline_Conn3G pass
CONNMAN_Offline_ConnWIFI pass
CONNMAN_Offline_ConnWired pass
CONNMAN_Offline_Persistance pass
CONNMAN_Offline_set pass
ConnMan_Signal/
CONNMAN_SIGNAL_Change_Passphrase pass
CONNMAN_SIGNAL_ChgEthPolicy pass
CONNMAN_SIGNAL_DisableWiFiDev pass
CONNMAN_SIGNAL_List_Networks pass
CONNMAN_SIGNAL_MoveBT pass
CONNMAN_SIGNAL_MoveWiFi pass
CONNMAN_SIGNAL_MoveWiMAX pass
CONNMAN_SIGNAL_RmvBT pass
CONNMAN_SIGNAL_RmvEthCable pass
CONNMAN_SIGNAL_SetEthPriority pass
CONNMAN_SIGNAL_Set_Flightmode pass
CONNMAN_SIGNAL_Set_WiFiNetworkMode pass
CONNMAN_SIGNAL_StaticIPEth pass
ConnMan_state/
CONNMAN_STATE_ETHENET_001 pass
CONNMAN_STATE_ETHENET_002 pass
CONNMAN_STATE_ETHENET_003 fail(#2465)
CONNMAN_STATE_AvailableAP pass
CONNMAN_STATE_IsolateAP fail(#2461)
CONNMAN_STATE_Forbidden_service
CONNMAN_STATE_3G fail(#1993)
CONNMAN_STATE_System fail(#1490)
ConnMan_Switch/
CONNMAN_Switch_EthToWiFi pass
ConnMan_VPN/
CONNMAN_VPN_3G norun
CONNMAN_VPN_Hidden pass
CONNMAN_VPN_WEP pass
CONNMAN_VPN_Wired pass
CONNMAN_VPN_WPA pass
CONNMAN_VPN_WPA2 pass
CONNMAN_VPN_Disconnect fail(#2318)
CONNMAN_VPN_Reconnect pass
ConnMan_Wired/
CONNMAN_Wired_Conn pass
CONNMAN_Wired_Hotplug pass
WIFI/ConnMan_WiFi_CoExist/
CONNMAN_WIFI_CoExists_3G pass
CONNMAN_WIFI_CoExists_BT pass
CONNMAN_WIFI_CoExists_WIFI pass
CONNMAN_WIFI_CoExists_Wired pass
WIFI/ConnMan_WiFi_ConnectType/
CONNMAN_WIFI_Connect_NoSecuBroadcast pass
CONNMAN_WIFI_Connect_NoSecuHide pass
CONNMAN_WIFI_Connect_NoSecuHideGroup pass
CONNMAN_WIFI_Connect_OpenWEP128 pass
CONNMAN_WIFI_Connect_OpenWEP128Hide pass
CONNMAN_WIFI_Connect_OpenWEP64 pass
CONNMAN_WIFI_Connect_OpenWEP64Hide pass
CONNMAN_WIFI_Connect_SharedWEP128 pass
CONNMAN_WIFI_Connect_SharedWEP128Hide pass
CONNMAN_WIFI_Connect_SharedWEP64 pass
CONNMAN_WIFI_Connect_SharedWEP64Hide pass
CONNMAN_WIFI_Connect_WPA2 pass
CONNMAN_WIFI_Connect_WPA2Hide pass
CONNMAN_WIFI_Connect_WPAEnterprise fail(#610)
CONNMAN_WIFI_Connect_WPAPSKAES pass
CONNMAN_WIFI_Connect_WPAPSKAESHide pass
CONNMAN_WIFI_Connect_WPAPSKTKIP pass
CONNMAN_WIFI_Connect_WPAPSKTKIPHide pass
CONNMAN_WIFI_EnableDisable fail(#2640)
WIFI/ConnMan_WiFi_Refresh/
CONNMAN_WIFI_Refresh pass
CONNMAN_WiFi_Refresh_Disable pass
CONNMAN_WiFi_Refresh_Enable pass
WIFI/ConnMan_WiFi_Roaming/
CONNMAN_WIFI_Roaming_Broadcast fail(#?)
CONNMAN_WIFI_Roaming_Hidden fail(#?)
ConnMan_UI/3G/
NETWORK_3G_Coexist_001 pass
NETWORK_3G_Coexist_002 pass
NETWORK_3G_DeviceControl_001 pass
NETWORK_3G_DeviceControl_002 pass
NETWORK_3G_ICON_001 pass
NETWORK_3G_Item_001 pass
NETWORK_3G_Item_002 pass
NETWORK_3G_Item_003 pass
NETWORK_3G_Service_001 norun
NETWORK_3G_Service_002 norun
NETWORK_3G_Service_003 norun
ConnMan_UI/BT/
NETWORK_BT_DeviceControl_001 pass
NETWORK_BT_Item_001 pass
NETWORK_BT_Item_002 pass
NETWORK_BT_Item_003 pass
NETWORK_BT_Item_004 pass
NETWORK_BT_PANU pass
ConnMan_UI/Carrick
Hidden_Obsolete_Error pass
Hidden_Unavail_Tech pass
Too_Many_Notif pass
Visible_Connecting_Network pass
ConnMan_UI/ConfigAutoConnOrder/
NETWORK_profile_moveafter blocked#2247
NETWORK_profile_movebefore blocked#2247
ConnMan_UI/SaticIP/
NETWORK_StaticIP_BT pass
NETWORK_StaticIP_reboot pass
NETWORK_WIFI_StaticIP pass
NETWORK_Wired_StaticIP pass
ConnMan_UI/WIFI/
NETWORK_Coexist_WiFi_Wired_001 pass
NETWORK_Coexist_WiFi_Wired_002 pass
NETWORK_Coexist_WiFi_Wired_003 pass
NETWORK_Coexist_WiFi_Wired_004 pass
NETWORK_WIFI_setting_reboot pass
NETWORK_WiFi_Switch_004 pass
NETWORK_WiFi_Switch_005 pass
NETWORK_WiFi_Switch_006 pass
NETWORK_Wireless_005 pass
NETWORK_Wireless_007 pass
NETWORK_Wireless_008 pass
NETWORK_Wireless_011 pass
NETWORK_Wireless_012 pass
NETWORK_Wireless_013 pass
NETWORK_Wireless_014 pass
NETWORK_Wireless_015 pass
NETWORK_Wireless_016 pass
NETWORK_Wireless_017 pass
NETWORK_Wireless_017 pass
NETWORK_Wireless_018 pass
NETWORK_Wireless_019 pass
NETWORK_Wireless_024 pass
NETWORK_Wireless_025 pass
NETWORK_Wireless_026 pass
NETWORK_Wireless_027 pass
NETWORK_Wireless_028 pass
NETWORK_Wireless_029 pass
NETWORK_Wireless_030 pass
NETWORK_Wireless_031 pass
NETWORK_Wireless_032 pass
NETWORK_Wireless_033 pass
NETWORK_Wireless_034 pass
NETWORK_Wireless_035 pass
NETWORK_Wireless_037 fail(#610)
NETWORK_Wireless_038 pass
NETWORK_Wireless_039 pass
NETWORK_Wireless_80211 pass
NETWORK_Wireless_80211x_hidden pass
NETWORK_Wireless_HiddenSharedWEP128 pass
NETWORK_Wireless_HiddenSharedWEP64 pass
NETWORK_Wireless_SharedWEP128 pass
NETWORK_Wireless_SharedWEP64 pass
NETWORK_Wireless_UI_012 pass
NETWORK_Wireless_UI_014 pass
NETWORK_Wireless_WiFiSwitch_007 pass
ConnMan_UI/Wired/
NETWORK_Ethernet_001 pass
NETWORK_Ethernet_007 pass
NETWORK_Ethernet_009 pass
NETWORK_Ethernet_010 pass
==================== Test Report ===================================
======================================================================
Total PASS FAIL ToCheck
TotalNumCase 127 123 4 0
connman 127 123 4 0
bt 14 12 2 0
ethernet 14 13 1 0
wifi 57 57 0 0
Guest 12 12 0 0
Functional 11 11 0 0
security 22 22 0 0
Advanced 6 6 0 0
Negative 6 6 0 0
signal 27 27 0 0
connman 14 14 0 0
manager 8 8 0 0
service 5 5 0 0
offline 0 0 0 0
profile 6 5 1 0
flight 15 15 0 0
======================================================================
**********************************************************************
**********************************************************************
********************************** bt ******************************
BigBroadcastPing.py PASS
SmallBroadcastPing.py PASS
HasIP.py PASS
IsDHCP.py PASS
IsPoweredOn.py PASS
Disconnect.py PASS
Connect.py PASS
IsReady.py PASS
DeviceIsExist.py PASS
PoweredOff.py PASS
PoweredOn.py PASS
IsReEnableWorks.py PASS
ifconfig_wifi_down.py FAIL #2514
ifconfig_wifi_up.py FAIL #2514
Test report for connman/bt
CountNumCase 14
PASS 12
FAIL 2
ToCheck 0
********************************** ethernet ******************************
BigBroadcastPing.py PASS
HasIP.py PASS
Connect.py PASS
DeviceIsExist.py PASS
Disconnect.py PASS
IfconfigDown.py FAIL #2514
IfconfigUp.py PASS
IsDHCP.py PASS
IsPoweredOn.py PASS
IsReady.py PASS
IsReEnableWorks.py PASS
PoweredOff.py PASS
PoweredOn.py PASS
SmallBroadcastPing.py PASS
Test report for connman/ethernet
CountNumCase 14
PASS 13
FAIL 1
ToCheck 0
********************************** Guest ******************************
BigBroadcastPing.py PASS
SmallBroadcastPing.py PASS
HasIP.py PASS
IsDHCP.py PASS
IsPoweredOn.py PASS
Disconnect.py PASS
Connect.py PASS
IsReady.py PASS
DeviceIsExist.py PASS
PoweredOff.py PASS
PoweredOn.py PASS
IsReEnableWorks.py PASS
Test report for connman/wifi/Guest
CountNumCase 12
PASS 12
FAIL 0
ToCheck 0
********************************** Functional ******************************
Channel10.py PASS
Channel11.py PASS
Channel1.py PASS
Channel2.py PASS
Channel3.py PASS
Channel4.py PASS
Channel5.py PASS
Channel6.py PASS
Channel7.py PASS
Channel8.py PASS
Channel9.py PASS
Test report for connman/wifi/Functional
CountNumCase 11
PASS 11
FAIL 0
ToCheck 0
********************************** security ******************************
Broadcast.py PASS
Hidden.py PASS
OpenBroadcastRSN.py PASS
OpenBroadcastTKIP.py PASS
OpenBroadcastWEP128.py PASS
OpenBroadcastWEP40.py PASS
OpenHiddenRSN.py PASS
OpenHiddenTKIP.py PASS
OpenHiddenWEP128.py PASS
OpenHiddenWEP40.py PASS
Open.py PASS
PSK1.py PASS
PSK2.py PASS
RSN.py PASS
WEP128.py PASS
WEP40.py PASS
WPA.py PASS
SharedBroadcastWEP128.py PASS
SharedBroadcastWEP40.py PASS
SharedHiddenWEP128.py PASS
SharedHiddenWEP40.py PASS
Shared.py PASS
Test report for connman/wifi/security
CountNumCase 22
PASS 22
FAIL 0
ToCheck 0
********************************** Advanced ******************************
AutoConnectFromSuspend.py PASS
AutoDelete.py PASS
DetectSecurity.py PASS
Disconnect.py PASS
ForgotPassword.py PASS
RememberPasswd.py PASS
Test report for connman/wifi/Advanced
CountNumCase 6
PASS 6
FAIL 0
ToCheck 0
********************************** Negative ******************************
NegWEP128ascii.py PASS
NegWEP128Long.py PASS
NegWEP128Short.py PASS
NegWEP40ascii.py PASS
NegWEP40Long.py PASS
NegWEP40Short.py PASS
Test report for connman/wifi/Negative
CountNumCase 6
PASS 6
FAIL 0
ToCheck 0
********************************** connman ******************************
EnabledTechnologies.py PASS
Networks.py PASS
Powered.py PASS
Scanning.py PASS
PSK2.py PASS
Services.py PASS
RSN.py PASS
State.py PASS
Connected.py PASS
ConnectedTechnologies.py PASS
DefaultTechnology.py PASS
Favorite.py PASS
IPv4.py PASS
OfflineMode.py PASS
Test report for connman/signal/connman
CountNumCase 14
PASS 14
FAIL 0
ToCheck 0
********************************** manager ******************************
ConnectedTechnologies.py PASS
WEP128.py PASS
DefaultTechnology.py PASS
EnabledTechnologies.py PASS
Profiles.py PASS
Services.py PASS
State.py PASS
Strength.py PASS
Test report for connman/signal/manager
CountNumCase 8
PASS 8
FAIL 0
ToCheck 0
********************************** service ******************************
AutoConnect.py PASS
Favorite.py PASS
WEP40.py PASS
IPv4.py PASS
Passphrase.py PASS
Test report for connman/signal/service
CountNumCase 5
PASS 5
FAIL 0
ToCheck 0
********************************** offline ******************************
Test report for connman/signal/offline
CountNumCase 0
PASS 0
FAIL 0
ToCheck 0
********************************** profile ******************************
Add.py PASS
AddExist.py PASS
List.py PASS
Remove.py PASS
RemoveNoExist.py PASS
Test report for connman/profile
CountNumCase 5
PASS 5
FAIL 0
ToCheck 0
********************************** flight ******************************
WiFiGuest.py PASS
Offline.py PASS
Service.py PASS
EthNoPing.py PASS
EthIsPoweredOff.py PASS
WiFiNoPing.py PASS
WiFiIsPoweredOff.py PASS
EthPoweredOn.py PASS
WPA.py PASS
EthPoweredOff.py PASS
WiFiPoweredOn.py PASS
WiFiPoweredOff.py PASS
Online.py PASS
WiFiOnline.py PASS
EthOnline.py PASS
Test report for connman/flight
CountNumCase 15
PASS 15
FAIL 0
ToCheck 0
======================================================================
======================================================================
Test report for the whole test suite
TotalNumCase 127
PASS 123
FAIL 4
ToCheck 0
ToCheck
_______________________________________________
connman mailing list
connman@connman.net
http://lists.connman.net/listinfo/connman
