Hi all, Based on the internal discussion I had with Darshana, following changes were done in the implementation.
1. Use a proper JSON schema for data and the result. 2. Removed the sensitive information (ex: password) from data. 3. As the claims are also logged by the audit logger, remove them from the data. Here is the new audit log. Initiator : [email protected] | Action : bulk_user_import | Target : PRIMARY | Data : {"users":["name1","name2","nam e74","name3","name3","namsdsa","name2ssss","name","name83","name5","name522"]} | Result : {"operation":"bulk_user_import","performedBy":"admin"," userStore":"PRIMARY","successCount":0,"duplicateUsers":{" count":8,"users":["name1","name2","name74","name3"," name3","name","name83","name5"]},"failedUsers":{"count":3,"u sers":[{"name":"namsdsa","cause":"Invalid claim uri has been provided: http://wso2.org/claims/ctry"},{"name":"name2ssss","cause":"Invalid claim uri has been provided: http://wso2.org/claims/emaiaddress "},{"name":"name522","cause":"Claims and values are not in correct format"}]}} Thanks and Regards, Menaka On Mon, Mar 19, 2018 at 7:28 PM, Menaka Jayawardena <[email protected]> wrote: > Hi Denuwanthi, > > It's just a template of the audit log that is being used currently. > For Bulk user import the action would be "*Bulk User Import*". > > On Mon, Mar 19, 2018 at 7:02 PM, Denuwanthi De Silva <[email protected]> > wrote: > >> >> >> On Mon, Mar 19, 2018 at 3:46 PM, Menaka Jayawardena <[email protected]> >> wrote: >> >>> Hi, >>> >>> As we discussed in the meeting today[1] (19/03/2018), I modified the >>> summary log as follows. >>> >>> {"Bulk User Import Operation Performed by":"admin","User >>> Store":"PRIMARY","Duplicate Users":{"Duplicate User Count":8,"User >>> Names":[{"Name":"name1"},{"Name":"name2"},{"Name":"name74"}, >>> {"Name":"name3"},{"Name":"name3"},{"Name":"name"},{"Name":"n >>> ame83"},{"Name":"name5"}]},"Failed Users":{"Failed User >>> Count":2,"Failed Users List":[{"Name":"namsdsa","Cause":"Invalid claim >>> uri has been provided: http://wso2.org/claims/ctry"}, >>> {"Name":"name2ssss","Cause":"Invalid claim uri has been provided: >>> http://wso2.org/claims/emaiaddress"}]}} >>> >>> And also, we discussed to log the bulk user import summary to the audit >>> logs in the following format. >>> >>> Initiator : [email protected] | Action : Add Role | Target : admin | >>> Data : {} | Result >>> >> Does this audit log gives us the message that a bulk user import happened? >> Action 'Add Role' does not imply a bulk user import happened IMO. >> Is it possible to introduce an action which clearly conveys the actual >> operation that occurred? >> >> >>> >>> The data section will contain the importing user list. As in the >>> documentation, we support importing a maximum of 500,000 users at a time. >>> So, considering the worse case scenario, if we log these users as well, it >>> will eat up the storage very quickly and cause in threat conditions. >>> >>> So IMO, we do not need to log users that are being imported. Also with >>> the Megala's feature [2], as the information is also being logged, I think >>> it's enough if we only log the result of the operation with Initiator, >>> Action and the Target values. >>> >>> WDYT? >>> >>> [1] [IAM] [Discussion] Bulk User Import Improvements >>> [2] Discussion on Improving Audit logs Related with User Management >>> >>> Thanks and Regards, >>> Menaka >>> >>> On Tue, Mar 13, 2018 at 12:45 PM, Dimuthu Leelarathne <[email protected] >>> > wrote: >>> >>>> >>>> >>>> On Tue, Mar 13, 2018 at 11:47 AM, Menaka Jayawardena <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> @Denuwanthi: Yes. It can be done. Please find the summery below. >>>>> >>>>> SUMMERY : >>>>> Bulk User Import Operation Performed by: admin >>>>> User Store : PRIMARY >>>>> Duplicate user count : 8 >>>>> . >>>>> >>>>> >>>>> *UI Message Modification.* >>>>> >>>>> Currently, if an error occurred in the process of performing the Bulk >>>>> User Import Operation, the following Error message will be shown. >>>>> >>>>> *Error occurs while importing usernames. All usernames were not >>>>> imported. Last error was : Invalid claim uri has been provided: >>>>> http://wso2.org/claims/emaiaddress <http://wso2.org/claims/emaiaddress>* >>>>> >>>>> But there are multiple errors (Duplicate user etc). In this case, I >>>>> think it's better if we show a more generic error with a brief summery and >>>>> direct them to view the log file for more information. >>>>> >>>>> For an example: >>>>> Bulk User Import Completed with Errors. >>>>> Success user count: x Duplicate user count: y Failed user count: z >>>>> Please check the user import log for more detailed information. >>>>> >>>> >>>> +1 >>>> >>>> And in the detail log we can log errors and duplicates. >>>> >>>> thanks, >>>> Dimuthu >>>> >>>> >>>> >>>>> >>>>> Any ideas, suggestions are highly appreciated. >>>>> >>>>> Thanks and Regards, >>>>> Menaka >>>>> >>>>> On Tue, Mar 13, 2018 at 9:24 AM, Denuwanthi De Silva < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 12, 2018 at 4:29 PM, Menaka Jayawardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Here is an experimental user import summery. >>>>>>> >>>>>>> SUMMERY : >>>>>>> Bulk User Import Operation Performed by: admin >>>>>>> Duplicate user count: 8 >>>>>>> Duplicate Users : >>>>>>> name1, name2, name74, name3, name3, name, name83, name5, >>>>>>> >>>>>>> Failed User Count: 2 Failed Users: >>>>>>> Name : namsdsa >>>>>>> Cause : Invalid claim uri has been provided: >>>>>>> http://wso2.org/claims/ctry >>>>>>> Name : name2ssss >>>>>>> Cause : Invalid claim uri has been provided: >>>>>>> http://wso2.org/claims/emaiaddress >>>>>>> >>>>>> >>>>>> Hi Menaka, >>>>>> >>>>>> Is it possible to print the user domain in the summary as well? Then >>>>>> the information of the userstore the users were imported will be >>>>>> available >>>>>> as well. >>>>>> >>>>>> Thanks, >>>>>> >>>>>>> >>>>>>> >>>>>>> The cause string is the standard error which comes from the >>>>>>> exception. Do we need to print the stack trace here? >>>>>>> >>>>>>> Also, there are two BulkUserImport classes (CSVUserBulkImport[1] and >>>>>>> ExcelUserBulkImport[2]) and also an unused interface [3] (The classes >>>>>>> [1] >>>>>>> and [2] are concreet classes). >>>>>>> >>>>>>> @IAM Team: Is there any particular reason why it kept like this? >>>>>>> >>>>>>> IMO in this implementation, we could use it to avoid code and method >>>>>>> duplication. (By making it an Abstract class) >>>>>>> >>>>>>> [1] https://github.com/wso2/carbon-identity-framework/blob/maste >>>>>>> r/components/user-mgt/org.wso2.carbon.user.mgt/src/main/java >>>>>>> /org/wso2/carbon/user/mgt/bulkimport/CSVUserBulkImport.java >>>>>>> [2] https://github.com/wso2/carbon-identity-framework/blob/maste >>>>>>> r/components/user-mgt/org.wso2.carbon.user.mgt/src/main/java >>>>>>> /org/wso2/carbon/user/mgt/bulkimport/ExcelUserBulkImport.java >>>>>>> [3] https://github.com/wso2/carbon-identity-framework/blob/maste >>>>>>> r/components/user-mgt/org.wso2.carbon.user.mgt/src/main/java >>>>>>> /org/wso2/carbon/user/mgt/bulkimport/UserBulkImport.java >>>>>>> >>>>>>> Thanks and Regards, >>>>>>> Menaka >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 12, 2018 at 2:14 PM, Menaka Jayawardena <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> [- strategy +Architecture] >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 12, 2018 at 12:21 PM, Menaka Jayawardena < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Dimuthu, >>>>>>>>> >>>>>>>>> Are you going to add this log appender by default to the >>>>>>>>>> configuration? >>>>>>>>>> >>>>>>>>> We can add the log appender by default and keep it commented. So >>>>>>>>> when the user enables the Bulk User import, he also can enable the log >>>>>>>>> appender as well. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 12, 2018 at 12:07 PM, Dimuthu Leelarathne < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Menaka, >>>>>>>>>> >>>>>>>>>> Are you going to add this log appender by default to the >>>>>>>>>> configuration? >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> Dimuthu >>>>>>>>>> >>>>>>>>>> On Mon, Mar 12, 2018 at 11:48 AM, Dakshika Jayathilaka < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Ruwan, >>>>>>>>>>> >>>>>>>>>>> Do we need to log each success? IMO admin will more interest on >>>>>>>>>>> failures or duplicates. IMHO we can add detail log on failures and >>>>>>>>>>> duplicates and then log the summary which includes the success >>>>>>>>>>> count. >>>>>>>>>>> >>>>>>>>>>> WDYT? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> *Dakshika Jayathilaka* >>>>>>>>>>> PMC Member & Committer of Apache Stratos >>>>>>>>>>> Associate Technical Lead >>>>>>>>>>> WSO2, Inc. >>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>> 0771100911 <077%20110%200911> >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 12, 2018 at 11:35 AM, Ruwan Abeykoon < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Menaka, >>>>>>>>>>>> This is nice feature. >>>>>>>>>>>> I would suggest adding one line per each user, before adding, >>>>>>>>>>>> and one line for each success, failure(with reason). >>>>>>>>>>>> Also add a line who performs this operation. Any trackable >>>>>>>>>>>> information of the request for audit purposes. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Ruwan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 12, 2018 at 11:21 AM, Menaka Jayawardena < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Currently, when performing bulk user import operation in >>>>>>>>>>>>> Identity Server, users face following issues. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. To check import failed users, need to filter the carbon log >>>>>>>>>>>>> file. >>>>>>>>>>>>> 2. In UI, it shows only the last error that occurred when >>>>>>>>>>>>> importing users. >>>>>>>>>>>>> >>>>>>>>>>>>> *Requirement Description.* >>>>>>>>>>>>> There should be a user friendly way to view the import failed >>>>>>>>>>>>> users. >>>>>>>>>>>>> >>>>>>>>>>>>> As a solution for this, we will provide a new log appender >>>>>>>>>>>>> which will log the messages to a separate log file specific for >>>>>>>>>>>>> bulk user >>>>>>>>>>>>> import. This will help users to easily view the status of the >>>>>>>>>>>>> imported >>>>>>>>>>>>> users and all the error logs. >>>>>>>>>>>>> >>>>>>>>>>>>> Also currently, as the operation summery, we only have >>>>>>>>>>>>> >>>>>>>>>>>>> "Success count: " + successCount + ", Fail count: " + >>>>>>>>>>>>> failCount + ", Duplicate count: " + duplicateCount >>>>>>>>>>>>> >>>>>>>>>>>>> Instead, it would be much effective if we could list the >>>>>>>>>>>>> failed and duplicate user names as well. >>>>>>>>>>>>> >>>>>>>>>>>>> "Success count: " + successCount + ", Fail count: " + >>>>>>>>>>>>> failCount + ", Duplicate count: " + duplicateCount >>>>>>>>>>>>> "Failed Users : " + [Failed Users List] + "Duplicate Users : " >>>>>>>>>>>>> + [Duplicate Users List] >>>>>>>>>>>>> >>>>>>>>>>>>> WDYT? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>> Menaka >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Menaka Jayawardena* >>>>>>>>>>>>> *Software Engineer - WSO2 Inc* >>>>>>>>>>>>> *Tel : 071 350 5470 <071%20350%205470>* >>>>>>>>>>>>> *LinkedIn: https://lk.linkedin.com/in/menakajayawardena >>>>>>>>>>>>> <https://lk.linkedin.com/in/menakajayawardena>* >>>>>>>>>>>>> *Blog: https://menakamadushanka.wordpress.com/ >>>>>>>>>>>>> <https://menakamadushanka.wordpress.com/>* >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *Ruwan Abeykoon* >>>>>>>>>>>> *Associate Director/Architect**,* >>>>>>>>>>>> *WSO2, Inc. http://wso2.com <https://wso2.com/signature> * >>>>>>>>>>>> *lean.enterprise.middleware.* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Dimuthu Leelarathne >>>>>>>>>> Director, Rapid Response Team >>>>>>>>>> >>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>> email: [email protected] >>>>>>>>>> Mobile: +94773661935 <+94%2077%20366%201935> >>>>>>>>>> Blog: http://muthulee.blogspot.com >>>>>>>>>> >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Menaka Jayawardena* >>>>>>>>> *Software Engineer - WSO2 Inc* >>>>>>>>> *Tel : 071 350 5470 <071%20350%205470>* >>>>>>>>> *LinkedIn: https://lk.linkedin.com/in/menakajayawardena >>>>>>>>> <https://lk.linkedin.com/in/menakajayawardena>* >>>>>>>>> *Blog: https://menakamadushanka.wordpress.com/ >>>>>>>>> <https://menakamadushanka.wordpress.com/>* >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Menaka Jayawardena* >>>>>>>> *Software Engineer - WSO2 Inc* >>>>>>>> *Tel : 071 350 5470 <071%20350%205470>* >>>>>>>> *LinkedIn: https://lk.linkedin.com/in/menakajayawardena >>>>>>>> <https://lk.linkedin.com/in/menakajayawardena>* >>>>>>>> *Blog: https://menakamadushanka.wordpress.com/ >>>>>>>> <https://menakamadushanka.wordpress.com/>* >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Menaka Jayawardena* >>>>>>> *Software Engineer - WSO2 Inc* >>>>>>> *Tel : 071 350 5470 <071%20350%205470>* >>>>>>> *LinkedIn: https://lk.linkedin.com/in/menakajayawardena >>>>>>> <https://lk.linkedin.com/in/menakajayawardena>* >>>>>>> *Blog: https://menakamadushanka.wordpress.com/ >>>>>>> <https://menakamadushanka.wordpress.com/>* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Denuwanthi De Silva >>>>>> Senior Software Engineer; >>>>>> WSO2 Inc.; http://wso2.com, >>>>>> Email: [email protected] >>>>>> Blog: https://denuwanthi.wordpress.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Menaka Jayawardena* >>>>> *Software Engineer - WSO2 Inc* >>>>> *Tel : 071 350 5470 <071%20350%205470>* >>>>> *LinkedIn: https://lk.linkedin.com/in/menakajayawardena >>>>> <https://lk.linkedin.com/in/menakajayawardena>* >>>>> *Blog: https://menakamadushanka.wordpress.com/ >>>>> <https://menakamadushanka.wordpress.com/>* >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dimuthu Leelarathne >>>> Director, Rapid Response Team >>>> >>>> WSO2, Inc. (http://wso2.com) >>>> email: [email protected] >>>> Mobile: +94773661935 <+94%2077%20366%201935> >>>> Blog: http://muthulee.blogspot.com >>>> >>>> Lean . Enterprise . Middleware >>>> >>> >>> >>> >>> -- >>> *Menaka Jayawardena* >>> Software Engineer >>> WSO2 Inc. >>> >>> Phone : +94 71 350 5470 <+94%2071%20350%205470> >>> LinkedIn : https://lk.linkedin.com/in/menakajayawardena >>> Blog : https://menakamadushanka.wordpress.com/ >>> >>> >> >> >> -- >> Denuwanthi De Silva >> Senior Software Engineer; >> WSO2 Inc.; http://wso2.com, >> Email: [email protected] >> Blog: https://denuwanthi.wordpress.com/ >> > > > > -- > *Menaka Jayawardena* > Software Engineer > WSO2 Inc. > > Phone : +94 71 350 5470 > LinkedIn : https://lk.linkedin.com/in/menakajayawardena > Blog : https://menakamadushanka.wordpress.com/ > > -- *Menaka Jayawardena* Software Engineer WSO2 Inc. Phone : +94 71 350 5470 LinkedIn : https://lk.linkedin.com/in/menakajayawardena Blog : https://menakamadushanka.wordpress.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
