I'd have to do testing on the times. The times were guestimates just to state there was a big difference, but not exact. 10 records today could take x minutes and tomorrow take y minutes. It just depends on what's coming in. Some records update no records and others update n records. I believe we had 904k records process in 55 minutes, which is pretty fast. If each record updates 10 other records than the process is still pretty fast because now you have 9 million records/updates processing in 8 hours. I'm not as worried about the validation process, which could take a while if done using a staging method. I was looking more performance from average users on the system by pushing that processing off to another server. It sounds like the 2 ways to do that is run the import on a backend admin type server or use a staging method and have escalations run on a backend admin type server. This would mean at minimum the server architecture should include 2 application servers, one for users and one for admin functions (which is ideal, but not always guaranteed). With escalations I could still have a user import to one server and run the escalations to trigger filters on the backend admin server based on server rankings. I couldn't run the import tool on server x though and fire the filters on server y, which would process the import in a quicker time because it's not waiting for all the actions to happen and have minimal impact on users that are hitting server x. I'm not sure if being able to specify where filters fire would be super useful, an impossible dream, or not really useful.
________________________________ From: ARSList <[email protected]> on behalf of LJ LongWing <[email protected]> Sent: Friday, January 12, 2018 2:58:30 PM To: ARSList Subject: Re: Filter Firing 1 million in 1 vs 8 hours though?....that's quite a bit of delay....I've done the 'staging' method on imports before, as you said, have them give you a file and process it in the background...notify them when the import is done, let them 'approve' the records, then move them into the actual form...yea, I've done that before and it helps with user perceptions of the system.....there must be something that can be done with the filters and indexes....going from .0036/record to .029 is QUITE a shift in amount of time to process. On Fri, Jan 12, 2018 at 12:48 PM, Brian Pancia <[email protected]<mailto:[email protected]>> wrote: Mike/LJ, Thanks for the quick response. That was my assumption of filter firing. Would be nice to be able to rank filters in the server ranking form and setup pools like escalations. Even better would be to setup server rankings based on pools. Currently I'm having a small set of users that can import data using the import tool. The difference can be 1 hour without firing the filters versus 8 hours firing filters. A bunch of variations in between depending on the data. A lot of checks and balances going on in the background. I could potentially standup a import server, so that everything runs on the import server which would eliminate any user impact from the mid-tier. That still leaves me with 1 hour versus 8 hours. That's where an escalation saves time for the end user doing the import. I could also setup a quick attachment field that they could save the file to the server and have workflow check for a file and import it or use AIE. The time to process is the same, but now instead of the user monitoring it, the process is done in the background. I would probably need to put in some type of validation notification then letting someone know that it was complete and everything is good. I've looked at the indexing and it looks pretty good. 1 million records just takes a little time to process. Thanks, Brian ________________________________ From: ARSList <[email protected]<mailto:[email protected]>> on behalf of Mike Galat <[email protected]<mailto:[email protected]>> Sent: Friday, January 12, 2018 2:06:27 PM To: 'ARSList' Subject: RE: Filter Firing Hi Brian – It has always been my experience that filters run on that initiated them. So, if initiated by an end user due to an action they took, filters would run on the AR server that they are connected to at the time. Filters that are initiated due to an escalation would run on the server where escalations are configured (via the ranking form for a server group). Thanks, Mike From: ARSList [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Brian Pancia Sent: Friday, January 12, 2018 13:55 To: [email protected]<mailto:[email protected]> Subject: Filter Firing Where do filters fire in a server group and can this be controlled? I'm assuming they fire on the server the user is attached to at the time. What about when an escalation triggers a filter. Does the filter fire on the same server as the escalation, which could then be controlled in the server ranking form. I never gave much thought to this. I'm sure some basic logging will give me the answer. However, looking to see if anyone has experience with this in a server group. This is looking at performance enhancements to integrations and data imports that could have a lot of updates, which could definitely impact the system(s). This seems to have more of an impact at the application layer rather than the database layer. One scenario I am looking at now is that 1 million records are imported into the system that may update multiple records throughout the system based on various conditions. If I fire the filters on import, the import could take a long time. If I surpress the filters then the import runs fairly quickly given the number of records somewhere around an hour for 1 million records. I could hold off on firing the filters until a time when utilization is low in order to have minimal impact on users. On import a record could say I need to update Bob, Sue, Kim, and Steve with the information or any combination. I'm sure an option is to have Bob as the keeper of info and have the others periodically check in with Bob through escalations in order to minimize filter firing. It still leaves me wondering were do filters fire and can this be controlled. I've always stay cleared of too many escalations. However, when your dealing with large data does it make sense to chunk it into buckets using escalations instead of running filters? Thanks Brian DISCLAIMER: The information contained in this e-mail and its attachments contain confidential information belonging to the sender, which is legally privileged. The information is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are notified that any disclosure, copying, distribution or action in reliance upon the contents of the information transmitted is strictly prohibited. If you have received this information in error, please delete it immediately. DISCLAIMER: The information contained in this e-mail and its attachments contain confidential information belonging to the sender, which is legally privileged. The information is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are notified that any disclosure, copying, distribution or action in reliance upon the contents of the information transmitted is strictly prohibited. If you have received this information in error, please delete it immediately. -- ARSList mailing list [email protected]<mailto:[email protected]> https://mailman.rrr.se/cgi/listinfo/arslist DISCLAIMER: The information contained in this e-mail and its attachments contain confidential information belonging to the sender, which is legally privileged. The information is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are notified that any disclosure, copying, distribution or action in reliance upon the contents of the information transmitted is strictly prohibited. If you have received this information in error, please delete it immediately.
-- ARSList mailing list [email protected] https://mailman.rrr.se/cgi/listinfo/arslist
