Hey Nick, The requirement is not to sum up bytes for all users globally. With the first profile we store sum of bytes per user to habse and to pull that information I need to query for each user providing username.
Instead of that I should able to query with single stellar command to pull upload bytes for all users (not the sum, it should be list of all users with upload bytes) available in the time window. Please let me know if it is clear now. Thanks, Anil. From: Nick Allen [mailto:[email protected]] Sent: Friday, April 12, 2019 10:30 AM To: [email protected] Subject: Re: Help to implement MAP kind of use case with metron profiler > I have an use case to sum the calculated values for each entity in the > profile and fetch the values for all the entities from profile in single > stellar command. Just create one profile that sums the bytes by user and create a separate profile that sums the bytes globally for all users. This seems close to what you have. Beyond that, I do not fully understand what you are trying to do. Can you describe the use case itself in more detail? On Thu, Apr 11, 2019 at 3:27 PM Anil Donthireddy <[email protected]<mailto:[email protected]>> wrote: Hi, I have an use case to sum the calculated values for each entity in the profile and fetch the values for all the entities from profile in single stellar command. As per my understanding, currently in metron we dont have such data structure to store the profile data to Hbase. If I store the profile the data with key as each entity and value, I need to iterate over the list of keys to fetch profile data for all keys. I would like to know if there is any better way to handle my use case with metron profiler. Please go through below illustration for more information about the usecase. Illustration: I need to calculate the upload bytes per each user in a profile window and retreive the list of all users with upload bytes ratio for specific duration. Currently with the existing scope, I have written a profile that aggregates the upload bytes per each users and stored to Hbase with key as user and value as sum of bytes for profiler window duration. Below is the profile definition for the same. ************ profile definition to calculate upload bytes per user *************** { "profile":"upbcpu", "onlyif":"(source.type == 'test1' || source.type == 'test2') and EXISTS(src_user_name) and (IS_EMPTY(src_user_name) == false) and EXISTS(ip_src_bytes_sent) and ip_src_bytes_sent > 0", "foreach":"src_user_name", "init": { "up_bytes" : 0.0 }, "update": { "up_bytes":"up_bytes + ip_src_bytes_sent" }, "result": { "profile": "up_bytes" } } *****************Profile definition end ****************************************** After storing the profile data, I can only fetch the upload bytes for all user only by querying for each user in separate stellar command by providing usernname as below PROFILE_GET('upbcpu', 'testuser', PROFILE_FIXED(20, 'DAYS')) The issue here is to fetch list of all users with their upload bytes for specific duration in single staller command, there is no way I can acheive it from profile definition. I am looking to have a profile something like below which stores a map to Hbase with key as username and value as upload bytes of that user and fetch all users information by querying with entity 'global' **************************Expected profile definition**************************** { "profile":"upbcpu", "onlyif":"(source.type == 'test1' || source.type == 'test2') and EXISTS(src_user_name) and (IS_EMPTY(src_user_name) == false) and EXISTS(ip_src_bytes_sent) and ip_src_bytes_sent > 0", "foreach":"'global'", "init": { "up_bytes_map" : <Initialize Map> }, "update": { "is_user_exists":"<Check by username as key in Map>", "up_bytes":"if(is_user_exists then ip_src_bytes_sent + <fetch existing upload bytes value for user from Map> else ip_src_bytes_sent", "up_bytes_map" : <update/add upload up_bytes value with key as user> }, "result": { "profile": "up_bytes_map" } } **************************Expected Profile definition End************************ With some definition like above, I can fetch the all users upload bytes for window using stellar command PROFILE_GET('upbcpu', 'global', PROFILE_FIXED(20, 'DAYS')) I would like to know the best way to implement the above illustrated usecase with metron profiler. Please help me if such use case can be implemented with existing stellar configuration. Looking forward to hear back with the suggestions. Thanks, Anil.
