I've got a pipeline that reads a stream consisting of names of groups-of-objects, feeds them to a stage that does an LDAP lookup to get a list of the object ids that belong to each group and then uses those object ids to retrieve the email-address field from each of the objects. It then outputs a record for each group, consisting of a comma-delimited list of email addresses from the group.
Some of these groups can be very large (e.g. one group I am testing this with contains 458 objects). If I feed all 458 object ids to the LDAP lookup stage one at a time, it all works, but takes about a minute to finish. Since the LDAP lookup stage has the capability to lookup more than one object per invocation, I am trying to group the object ids together in to reasonably-sized groups to reduce the number of invocations. So, I threw in a "join 100" stage to group 100 object ids together. As soon as I did that, the pipeline started abending with a 2006 error trying to allocate a socket. However, this socket error only happened for the first record on join's primary output (i.e. the first group of 100 objects). The lookup of the remaining groups worked fine (I think there were 4 or 5 records coming out of join). So clearly join is messing with the timing. Being somewhat naiive with respect to timing issues, I did what I usually do, I started throwing in buffer stages in various places to see if that helped :-) Interestingly, adding a buffer stage caused ALL the records from join to elicit the socket error. For fun, I replaced the buffer with a copy stage. That caused the first two records to elicit the error, but the remaining ones worked. The LDAP lookup stage is a black box. I have no easy access to the source (although I do know the author, so if I did have to get ahold of the source, it might be possible). My question is, does the behavior I observed suggest that something is wrong inside the LDAP lookup stage (i.e. could it be doing anything differently such that me throwing in the join stage does NOT cause this error), or is there anything I can do myself to get around this? Cheers, bc
