Hi Whoever related with IoTivity 1.2.1 release and CTT1.4,
This report should be shared by QA team originally, but I send it on behalf of QA team lead now. >From next time, I expect that it will be distributed from QA team. This is the IoTivity vs CTT testing result with the yesterday latest committed code on 1.2-rel branch of IoTivity and the latest CTT. * Test scope does not cover the collection scenario test yet. I expect from next testing, collection test result will be included Following issues are should be resolved. After resolving them, please let me know with IoTivity change_ID. >From the CTT perspective, Could you let me know the schedule of CTT with >resolving No7 issue? No TC Module Category 1 CT1.1.1 (both server & client) discovery request with rt filter, IoTivity respond with all resource Discovery&Connectivity-RI 2 CT1.1.1 & CT1.7.8.3 (both server & client) CTT sent a discovery request with invalid resource type query, Discovery&Connectivity-RI 3 CT1.1.1 & CT1.7.8.3 (both server & client) CTT sent a request to /oic/p with query rt=oic.wk.col Discovery&Connectivity-RI 4 CT1.1.1 & CT1.7.8.3 (both server & client) CTT expecting Array of links(valid for collection) for each unicast GET response. Discovery&Connectivity-RI 5 CT1.2.2 (only server) CON GET Discovery&Connectivity-CA 6 CT1.7.8.1 (only client) /oic/d is not a secured resource, it should not send error response. Security 7 CT1.7.8.3 (both server & client) CTT Problem (agreed) Security BR, Uze Choi From: Muhammad Mushfiqul Islam [mailto:[email protected]] Sent: Wednesday, November 23, 2016 8:20 PM To: 'Mitch Kettrick'; Sung-Kyu Ko; Uze Choi; MyeongGi Jeong Cc: DWARKAPRASAD DAYAMA; Jongsung Lee; 'Jacek Hryszkiewicz'; SOON HWANG CHOI; Gihong Kim; Jaehong, Jo; hyuna jo; 'Marek Szkowron'; 'Ed Agis'; 'Heldt-Sheller, Nathan'; Lee, Dongik; MoonYoung Choi; 'Brian Scriber'; 'Jason Smith'; jongmin choi; JiHwan Seo; Jaewook Jung; minji park; Byonggon Chun; Glen Youngjin Kim; Eunok Shin; Habib Virji; Ziran Sun; Joonghwan Lee; Chul Lee Subject: RE: [Security] CTT 1.3.18 vs. IoTivity 1.2-rel branch Hello Everyone, I am sharing the test result of CTT v1.3.18 against IoTivity 1.2-rel branch. This test was performed without collection resources. As currently vendors need to define collection response, I am working on to integrate it to IUT Simulator, and hope to provide a result including the collection resource when I am finished with it Now, there is 1 extra TC failing for 1.2-rel branch, which was not failing in 1.2.0-RC4(CT1.1.1). Here is a list of failing TCs: CT1.1.1 - both server & client TC CT1.2.2 - only server TC CT1.7.8.1 - only client TC CT1.7.8.3 - both server & client TC n TC Error Log Status Comment CT1.1.1 (both server & client) New Issue CTT sends Multicast discovery request with resource type query, IoTivity is sending response with all the available resource. This is a IoTivity Issue in 1.2-rel branch and should be fixed CT1.1.1 & CT1.7.8.3 (both server & client) New Issue CTT sent a discovery request with invalid resource type query, IoTivity sent success response instead of 4.04(resource not found) CT1.1.1 & CT1.7.8.3 (both server & client) New Issue CTT sent a request with to /oic/p resource(and other resources also) with resource type query rt=oic.wk.col As /oic/p resource does not have this resource type, it should have responded with 4.04 (IoTivity problem) CT1.1.1 & CT1.7.8.3 (both server & client) New Issue In the previous step as IoTivity responded positively for collection resource type query(for all; resources), CTT expecting Array of links(valid for collection) for each unicast GET response. This is a iotivity issue and would automatically fixed if previous problems are fixed. CT1.2.2 (only server) Old Issue Previously, CTT showed error for NON GET, now there is an error for CON GET also. Detailed log sows that 2 CON GET was sent consecutively, CTT is showing the 1st response had rt & if though the second response does not have it. CT1.7.8.1 (only client) Old Issue CTT is expecting error on GET request to client /oic/d resource, but as /oic/d is not a secured resource, it should not send error response. CT1.7.8.3 (both server & client) Old Issue Already discussed in last mail by Mr. Jongsung Lee - Thanks & Regards, Mushfiqul Islam Antu --------- Original Message --------- Sender : Jongsung Lee <js126.lee at samsung.com> S5/Senior Engineer/Security Lab./Samsung Electronics Date : 2016-11-23 13:55 (GMT+6) Title : [Security] CTT 1.3.18 vs. IoTivity 1.2-rel branch To : Uze Choi<uzchoi at samsung.com>, Sung-Kyu Ko<sungkyu.ko at samsung.com>, MyeongGi Jeong<myeong.jeong at samsung.com>, null<cpm at openconnectivity.org> CC : Muhammad Mushfiqul Islam<i.mushfiq at samsung.com>, DWARKAPRASAD DAYAMA<dwarka.dayama at samsung.com>, null<jacek.hryszkiewicz at comarch.com>, SOON HWANG CHOI<soonhwang.choi at samsung.com>, Gihong Kim<k1358.kim at samsung.com>, Jaehong, Jo<jaehong.jo at samsung.com>, hyuna jo<hyuna0213.jo at samsung.com>, null<marek.szkowron at comarch.com>, null<Ed.Agis at intel.com>, Jongsung Lee<js126.lee at samsung.com>, null<nathan.heldt-sheller at intel.com>, Lee, Dongik<dongik.lee at samsung.com>, MoonYoung Choi<moonyoung.choi at samsung.com>, null<b.scriber at cablelabs.com>, null<lab_mgr at openconnectivity.org>, jongmin choi<jminl.choi at samsung.com>, JiHwan Seo<jihwan.seo at samsung.com>, Jaewook Jung<jw0213.jung at samsung.com>, minji park<minjii.park at samsung.com>, Byonggon Chun<bg.chun at samsung.com>, Glen Youngjin Kim<glen.kim at samsung.com>, Eunok Shin<eunok.shin at samsung.com>, Habib Virji<habib.virji at samsung.com>, Ziran Sun<ziran.sun at samsung.com>, Joonghwan Lee<jh05.lee at samsung.com>, Chul Lee<chuls.lee at samsung.com> Dear Mitch and Jacek, I have checked the security server TCs of CTT v1.3.18 with IoTivity 1.2-rel branch based on the latest commit as below. This commit has been included <https://gerrit.iotivity.org/gerrit/#/c/14137/> https://gerrit.iotivity.org/gerrit/#/c/14137/ for CR0046. commit b85ff2c9b0a57e43f2694692f5283635198a0ac5 Author: Dave Thaler <dthaler at microsoft.com> Date: Mon Nov 21 16:49:57 2016 -0800 CT 1.7.8.3 is still failed due to below 2 issue. Other TCs were passed. (I attached CTT log) 1) Need to check which response is correct to "/oic/res?rt=oic.wk.col" query. 2) CTT is expecting response cod 2.05 to access "/oic/sec/cred & acl" via COAP. Please find below error log and comment. TC Error Log Status Comment CT1.7.8.3 1.195s 12:12:39 INFO:?Sending unicast NON RETRIEVE request with: /oic/res?rt=oic.wk.col 1.195s 12:12:39 VERBOSE:?-> 10.113.67.114:49176->10.113.67.115:42298 NON-GET ID=22574, Token=75C4938A, Options=[URI-Port=42298, URI-Path=oic, res, URI-Query=rt=oic.wk.col, Accept=application/cbor], Secured=true 1.203s 12:12:39 VERBOSE:?<- 10.113.67.115:42298->10.113.67.114:49176 NON-2.05 Content ID=22574, Token=75C4938A, Options=[URI-Path=oic, res, Content-Type=application/cbor], Secured=true 1.203s 12:12:39 VERBOSE:?<- [{"di":"6a757374-776f-726b-4465-765575696430","links":[{"href":"/oic/sec/doxm","rt":["oic.r.doxm"],"if":["oic.if.baseline"],"p":{"bm":1,"sec":true,"port":42298}},{"href":"/oic/sec/pstat","rt":["oic.r.pstat"],"if":["oic.if.baseline"],"p":{"bm":1,"sec":true,"port":42298}},{"href":"/oic/d","rt":["oic.wk.d","oic.d.airconditioner"],"if":["oic.if.baseline","oic.if.r"],"p":{"bm":1,"sec":false}},{"href":"/oic/p","rt":["oic.wk.p"],"if":["oic.if.baseline","oic.if.r"],"p":{"bm":1,"sec":false}},{"href":"/BinarySwitchResURI","rt":["oic.r.switch.binary"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":true,"port":42298}},{"href":"/TemperatureResURI","rt":["oic.r.temperature"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":true,"port":42298}},{"href":"/AirFlowResURI","rt":["oic.r.airflow"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":true,"port":42298}},{"href":"/Vendor/AirConditioner/TimerClock","rt":["x.com.vendor.timer"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":1,"sec":true,"port":42298}},{"href":"/Vendor/AirConditioner/Swinger","rt":["x.com.vendor.swing"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":1,"sec":false}}]}] 1.292s 12:12:39 INFO:?CT_Precondition: IUT respond to /oic/res?rt=oic.wk.col (NON-GET ID:22574) 1.292s 12:12:39 DEBUG:?Verification with ID:"CT_Precondition" ended with result: PASSED 1.298s 12:12:39 INFO:?Sending unicast NON RETRIEVE request with: /oic/d?rt=oic.wk.col 1.298s 12:12:39 VERBOSE:?-> 10.113.67.114:49176->10.113.67.115:42298 NON-GET ID=22575, Token=75C4938B, Options=[URI-Port=42298, URI-Path=oic, d, URI-Query=rt=oic.wk.col, Accept=application/cbor], Secured=true 1.304s 12:12:39 VERBOSE:?<- 10.113.67.115:42298->10.113.67.114:49176 NON-2.05 Content ID=22575, Token=75C4938B, Options=[URI-Path=oic, d, Content-Type=application/cbor], Secured=true 1.304s 12:12:39 VERBOSE:?<- {"rt":["oic.wk.d","oic.d.airconditioner"],"if":["oic.if.baseline","oic.if.r"],"di":"6a757374-776f-726b-4465-765575696430","n":"Vendor Smart Home AirCon Device","icv":"core.1.1.0","dmv":"res.1.1.0,SH.1.1.0"} 1.333s 12:12:39 INFO:?Prompting "Waiting for CoAP response... " 1.369s 12:12:39 DEBUG:?AgentBase multiple action complete 1.369s 12:12:39 DEBUG:?Starting verification with ID:"CT_Precondition"... 1.369s 12:12:39 INFO:?CT_Precondition: IUT respond to /oic/d?rt=oic.wk.col (NON-GET ID:22575) 1.369s 12:12:39 DEBUG:?Verification with ID:"CT_Precondition" ended with result: PASSED 1.370s 12:12:39 DEBUG:?Starting verification with ID:"CT_Precondition"... 1.371s 12:12:39 ERROR:?CT_Precondition: Improper format of link if in collection {'if': ['oic.if.baseline', 'oic.if.r'], 'di': '6a757374-776f-726b-4465-765575696430', 'rt': ['oic.wk.d', 'oic.d.airconditioner'], 'n': 'Vendor... 1.371s 12:12:39 DEBUG:?continued: ... Smart Home AirCon Device', 'dmv': 'res.1.1.0,SH.1.1.0', 'icv': 'core.1.1.0'} 1.371s 12:12:39 DEBUG:?Verification with ID:"CT_Precondition" ended with result: FAILED 1.371s 12:12:39 DEBUG:?Starting verification with ID:"CT_Precondition"... 1.371s 12:12:39 ERROR:?CT_Precondition: Improper format of link di in collection {'if': ['oic.if.baseline', 'oic.if.r'], 'di': '6a757374-776f-726b-4465-765575696430', 'rt': ['oic.wk.d', 'oic.d.airconditioner'], 'n': 'Vendor... 1.371s 12:12:39 DEBUG:?continued: ... Smart Home AirCon Device', 'dmv': 'res.1.1.0,SH.1.1.0', 'icv': 'core.1.1.0'} 1.371s 12:12:39 DEBUG:?Verification with ID:"CT_Precondition" ended with result: FAILED New Issue. In previous IoTivity version (1.2.0 TAG), IoTivity responded "Not Found" to "/oic/res?rt=oic.wk.col" query. But, the latest IoTivity responeds the same as "oic/res". Need to check which response is correct to "/oic/res?rt=oic.wk.col" query. 80.387s 12:13:58 INFO:?Sending unicast CON RETRIEVE request with /oic/sec/cred?if=oic.if.baseline 80.828s 12:13:59 INFO:?CT_Precondition: IUT respond to /oic/sec/cred?if=oic.if.baseline (CON-GET ID:22963) 80.828s 12:13:59 DEBUG:?Verification with ID:"CT_Precondition" ended with result: PASSED 80.830s 12:13:59 DEBUG:?Default interface: None 80.830s 12:13:59 DEBUG:?Starting verification with ID:"CT1.7.8.3_Precondition"... 80.830s 12:13:59 ERROR:?CT1.7.8.3_Precondition: For message: ID=22963 Token=409DB69C expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 80.830s 12:13:59 DEBUG:?Verification with ID:"CT1.7.8.3_Precondition" ended with result: FAILED 80.834s 12:13:59 ERROR:?Response for GET '/oic/sec/cred?if=oic.if.baseline' is not valid response. Could not process procedure for this resource Sending unicast CON RETRIEVE request with /oic/sec/acl?if=oic.if.baseline 81.294s 12:13:59 INFO:?CT_Precondition: IUT respond to /oic/sec/acl?if=oic.if.baseline (CON-GET ID:22964) 81.294s 12:13:59 DEBUG:?Verification with ID:"CT_Precondition" ended with result: PASSED 81.295s 12:13:59 DEBUG:?Default interface: None 81.295s 12:13:59 DEBUG:?Starting verification with ID:"CT1.7.8.3_Precondition"... 81.295s 12:13:59 ERROR:?CT1.7.8.3_Precondition: For message: ID=22964 Token=409DB69D expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 81.295s 12:13:59 DEBUG:?Verification with ID:"CT1.7.8.3_Precondition" ended with result: FAILED 81.295s 12:13:59 ERROR:?Response for GET '/oic/sec/acl?if=oic.if.baseline' is not valid response. Could not process procedure for this resource CTT Issue In order to aligns with CR0046, <https://gerrit.iotivity.org/gerrit/#/c/14137/> https://gerrit.iotivity.org/gerrit/#/c/14137/ is applied on IoTivity, the doxm and pstat can be accessed via CoAP among SVRs. But, CTT is expecting response cod 2.05 to access /oic/sec/cred & acl via COAP. Thank you, Jongsung Lee. -------------- next part -------------- HTML ?????? ??????????????... URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717539_F4AY25FV.png Type: image/png Size: 86499 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717546_HCKGEMU3.png Type: image/png Size: 5148 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0001.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717552_L801P3K2.png Type: image/png Size: 167521 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0002.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717558_8SVVMERX.png Type: image/png Size: 37521 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0003.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717564_B2LES0FC.png Type: image/png Size: 25212 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0004.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717569_96114O2X.png Type: image/png Size: 6770 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0005.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 201611231717578_Y56ATGRK.png Type: image/png Size: 80133 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment-0006.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: CTT1.3.18_IoTivity1.2-rel.zip Type: application/octet-stream Size: 4541593 bytes Desc: ?????? ?? ????????. URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161124/4982cbfa/attachment.obj>
