Actually, CPU idle time on the ApacheDS server never drops below around 80% (10 second sampling with vmstat), so I’m not convinced that is the problem. Also, I would argue that a RPi 4 is not that slow for a server deleting 50+ entries (RPi 3 possibly, RPi 2 absolutely). And the error always occurs within 60 seconds, so it is not a 60 second time-out.
However, after doing some more testing, I found that the copying the 200 entries to beneath another context, and then deleting 100 at a time from that new context, completes without error and is MUCH faster (takes just a few seconds), all using the same hardware. The difference? Something I should have thought about before: I have another server doing pull (consumer) replication from the original context. So I think that might be what is causing my bottleneck, specifically replicating changes — all of those deletions -- to another server. > On Jul 12, 2021, at 5:15 PM, Emmanuel Lécharny <elecha...@gmail.com> wrote: > > > > On 12/07/2021 16:37, Donald Lohr wrote: >> Dave, >> My recommendation is a test to determine if the issue is with Studio or >> Apache. I've done LDAP Admin for years and have used as an LDAP >> service/server four different vendor products. I have not found a GUI ldap >> admin tool provided by a vendor or in the free/opensource space that's worth >> a dang for doing most "mass changes" like it seems you are doing. > > The fact that you do such mass delete with a GUI is totally irrelevant with > the fact that you have trouble with those mass deletions. > > Here, ApacheDS is running on a pretty slow machine, and IMHO this is the root > cause of your troubles. > > Studio itself just send a DEL request to the server, wait for the Del > response, then iterates. It can't fail if the server does not fail. > > Here, I suspect that you get a timeout (60s) and that ends the process. > > Have you tried with ApacheDS running on a faster server ? > >> I exclusively use OpenLDAP's command line tools: >> ldapsearch >> ldapmodify >> ldapdelete >> ldapmodrdn >> I even develop .sh scripts that do ldapsearches, manipulate data with sed, >> cut, grep and etc and build .ldif files to import data (via ldapadd or >> ldapmodify) into other ldap servers for testing/dev and etc. And when done >> them mass delete those test users with ldapdelete and a .ldif file. >> You can install these OpenLDAP tools on both your mac and linux server. >> If you can successfully delete 100 user entries using OpenLDAP's ldapdelete >> command without errors, then the issue is with Studio, if you get errors >> then something on the backend (your ldap server) is causing the error you >> report. >> Don >> On 7/11/21 7:22 PM, David Filip wrote: >>> ------------------------------------------------------------------------ >>> Just curious whether this problem has been acknowledged and/or reproduced? >>> >>> Also, to be clear, the reason why I started doing smaller “batches” of >>> deletions is because if I try to delete a large number at once in Studio >>> (using ApacheDS as the LDAP server) I also receive an error (from one large >>> batch vs. successive small batches). >>> >>> For example, I just tried deleting one hundred (100) entries beneath a >>> single sub-context, and received the error: >>> >>> >>> After reaching what looked like around 47%, and it deleting 47 entries. No >>> other updates were occurring on the LDAP server (ApacheDS) while I was >>> performing the deletions. >>> >>> As you may have guessed — or maybe not — I am writing a utility to >>> load/synchronize an LDAP server (outside of Studio) from a non-LDAP >>> directory, so I am doing large loads / deletions / rinse and repeat during >>> development and testing. Therefore, I am trying to delete large batches of >>> LDAP entries — all beneath the same sub-context -- after each test run. So >>> perhaps not something that I will be doing on a regular basis once I am >>> done with development. >>> >>> Nonetheless, I would not expect to see these types of errors from Studio. >>> And no, I have not tried mass deletions programmatically (JNDI), as I am >>> using Studio as my workbench for viewing and managing the LDAP directory >>> after each test load. >>> >>> Regards, >>> >>> Dave. >>> >>>> On Jul 4, 2021, at 8:03 PM, David Filip <dfi...@colornet.com >>>> <mailto:dfi...@colornet.com>> wrote: >>>> >>>> Stefan, >>>> >>>> Answers below. >>>> >>>> Regards, >>>> >>>> Dave. >>>> >>>>> On Jul 4, 2021, at 8:59 AM, Stefan Seelmann <m...@stefan-seelmann.de >>>>> <mailto:m...@stefan-seelmann.de>> wrote: >>>>> >>>>> Hi Dave, >>>>> >>>>> On 7/4/21 2:00 AM, David Filip wrote: >>>>>> Greetings, >>>>>> >>>>>> I have noticed that when deleting entries too fast using Apache >>>>>> Directory Studio Version: 2.0.0.v20210213-M16 (ApacheDS 2.0.0.AM26), I >>>>>> receive the following error: >>>>>> >>>>>> ERR_04169_RESPONSE_QUEUE_EMPTIED The response queue has been emptied, no >>>>>> response was found >>>>>> >>>>>> >>>>>> >>>>>> I am not sure if this is a Studio (frontend) or ApacheDS (backend) issue? >>>>> >>>>> Can you please give detailed information about your environment? >>>>> >>>>> Studio: Which operating system and which Java version do you use? >>>>> >>>> >>>> Running Apache Directory Studio (Version: 2.0.0.v20210213-M16) on macOS >>>> Catalina (10.15.7). Connecting from Apache Directory Studio on macOS >>>> connected to Apache Directory Server (*apacheds-2.0.0.AM26*) running on >>>> Linux (Raspbian GNU/Linux 10 (Debian Buster). >>>> >>>> Yes, I wait for the prior batch deletion to complete to 100% before >>>> attempting to delete another batch. >>>> >>>> I usually try to wait about 1 second for every entry deleted, which is >>>> frequently insufficien (e.g., if I delete 30 entries, I wait 30 seconds >>>> before attempting to delete the next batch, AFTER the prior batch reaches >>>> 100%). >>>> >>>>> Server: Do you use the ApacheDS embedded in Studio? Otherwise which type >>>>> of directory server do you use? >>>>> >>>>>> To Reproduce: >>>>>> >>>>>> Load a large number (200) of entire into the directory via JNDI API. >>>>>> >>>>>> After the load completes, select 8 - 10 entries at a time in Studio, hit >>>>>> the [Delete] key, and confirm deleting those entries. >>>>>> >>>>>> If done successively, without waiting a sufficient amount of time, will >>>>>> generate the above error, and not all of the selected entries will have >>>>>> been deleted (but usually one or a few have been deleted). Hit F5 to >>>>>> refresh to determine which entires still remain. >>>>>> >>>>>> Expected Result: >>>>>> >>>>>> All entires should be successfully deleted without any error or warning >>>>>> message. >>>>>> >>>>>> Work Around: >>>>>> >>>>>> Wait 30 - 60 scones between each batch (8 - 10 entires) of deletions. >>>>>> Or select a smaller set of entries (5 - 6) and wait 15 - 20 seconds >>>>>> between each batch. >>>>>> >>>>>> Has anyone else experienced this? Is this a known limitation? >>>>> >>>>> I the previous batch completed before you run the next batch? If a batch >>>>> is still running it should show up in the "Progress" view [1], and after >>>>> it finished the "LDAP Browser" view usually is refreshed. In you run a >>>>> 2nd delete batch while the first one is still running it may be related >>>>> to the fact that the LDAP API is not really thread safe [2], so if the >>>>> same connection is used for multiple operations unexpected things may >>>>> happen. >>>>> >>>>> Kind Regards, >>>>> Stefan >>>>> >>>>> >>>>> [1] >>>>> https://nightlies.apache.org/directory/studio/2.0.0.v20210213-M16/userguide/ldap_browser/tools_progress_view.html >>>>> >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_directory_studio_2.0.0.v20210213-2DM16_userguide_ldap-5Fbrowser_tools-5Fprogress-5Fview.html&d=DwMFaQ&c=eLbWYnpnzycBCgmb7vCI4uqNEB9RSjOdn_5nBEmmeq0&r=Pa2DB88IW_s2TyLfktHtWA&m=bMxIopAUY5g_re6P4d62Jp3WqTOQ51tSs-LHaoqvlaI&s=vPj2Xg22FGEhXmAoVTXlXOmq2bQrq4Tc8zLom6a0FfI&e=> >>>>> >>>>> [2]https://issues.apache.org/jira/browse/DIRAPI-237 >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DIRAPI-2D237&d=DwMFaQ&c=eLbWYnpnzycBCgmb7vCI4uqNEB9RSjOdn_5nBEmmeq0&r=Pa2DB88IW_s2TyLfktHtWA&m=bMxIopAUY5g_re6P4d62Jp3WqTOQ51tSs-LHaoqvlaI&s=0xw1MxmCAJq3d8sGd13q_utPG1gLpY6D7MhhXsf3wIs&e=> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail:users-unsubscr...@directory.apache.org >>>>> <mailto:users-unsubscr...@directory.apache.org> >>>>> For additional commands, e-mail:users-h...@directory.apache.org >>>>> <mailto:users-h...@directory.apache.org> >>>> >>> > > -- > *Emmanuel Lécharny - CTO* 205 Promenade des Anglais – 06200 NICE > T. +33 (0)4 89 97 36 50 > P. +33 (0)6 08 33 32 61 > emmanuel.lecha...@busit.com <mailto:emmanuel.lecha...@busit.com> > https://www.busit.com/ <https://www.busit.com/> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@directory.apache.org > <mailto:users-unsubscr...@directory.apache.org> > For additional commands, e-mail: users-h...@directory.apache.org > <mailto:users-h...@directory.apache.org>
