I see what you mean: if postprocessing the CSV is required, that sounds like a pain. Ideally there’s a way to unencode the column values as they’re streamed into pg_bulkupload somehow, even if it requires an additional programmatic shim.
> On Sep 15, 2022, at 12:47 PM, Brooks Bridges <[email protected]> wrote: > > I'm more concerned about the time and load involved in having to loop through > a massive csv and run that against specific columns of each line. There may > be an option to avoid having to do that though, I'll experiment with it a > bit. Thanks for the suggestion! > > -----Original Message----- > From: sr-users <[email protected]> On Behalf Of Alex > Balashov > Sent: Thursday, September 15, 2022 09:39 > To: Kamailio (SER) - Users Mailing List <[email protected]> > Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces > > base64 is a pretty trivial algorithm. You’d be surprised. > >> On Sep 15, 2022, at 12:37 PM, Brooks Bridges <[email protected]> wrote: >> >> Unfortunately due to the volume of records, I don't think that's going to be >> really feasible as I'm dumping these out directly to a csv to be archived >> and imported into a database using pg_bulkload, and I don't think the >> additional overhead of having to process every record (of which there will >> be at least 1M per minute, likely more in the future) is feasible without >> impacting the performance of the system. >> >> -----Original Message----- >> From: sr-users <[email protected]> On Behalf Of Alex >> Balashov >> Sent: Thursday, September 15, 2022 09:11 >> To: Kamailio (SER) - Users Mailing List <[email protected]> >> Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces >> >> A common approach to avoid delimiter issues in general is to store >> base64-encoded strings in Redis rather than the original strings. If >> whatever is reading from Redis can un-encode them, that pretty well solves >> the problem. >> >>> On Sep 15, 2022, at 12:09 PM, Brooks Bridges <[email protected]> wrote: >>> >>> I am aware of, and have used sucessfully, the %s substitution option in >>> redis_cmd, however when trying to do 4 values I'm getting a parse error >>> about too many arguments. A review of the module's source appears to >>> indicate that there is a hard limit of 3 substitution values when using >>> this method. >>> >>> Is there a suitable workaround anyone has for this to enable insertion of >>> values with spaces in them that won't get interpreted by Kamailio itself? >>> Escaping the value, even by using $_s to eval a dynamic string, ends up >>> with Kamailio itself apparently trying to parse the escape and breaking >>> things further. >>> >>> Thanks! >>> Confidentiality Notice: This e-mail, and any attachment to it, contains >>> privileged and confidential information intended only for the use of the >>> individual(s) or entity named on the e-mail. If the reader of this e-mail >>> is not the intended recipient, or the employee or agent responsible for >>> delivering it to the intended recipient, you are hereby notified that >>> reading this e-mail is strictly prohibited. If you have received this >>> e-mail in error, please immediately return it to the sender and delete it >>> from your system. >>> >>> __________________________________________________________ >>> Kamailio - Users Mailing List - Non Commercial Discussions >>> * [email protected] >>> Important: keep the mailing list in the recipients, do not reply only to >>> the sender! >>> Edit mailing list options or unsubscribe: >>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Alex Balashov | Principal | Evariste Systems LLC >> >> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) >> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ >> >> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> * [email protected] >> Important: keep the mailing list in the recipients, do not reply only to the >> sender! >> Edit mailing list options or unsubscribe: >> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> Confidentiality Notice: This e-mail, and any attachment to it, contains >> privileged and confidential information intended only for the use of the >> individual(s) or entity named on the e-mail. If the reader of this e-mail is >> not the intended recipient, or the employee or agent responsible for >> delivering it to the intended recipient, you are hereby notified that >> reading this e-mail is strictly prohibited. If you have received this e-mail >> in error, please immediately return it to the sender and delete it from your >> system. >> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> * [email protected] >> Important: keep the mailing list in the recipients, do not reply only to the >> sender! >> Edit mailing list options or unsubscribe: >> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > -- > Alex Balashov | Principal | Evariste Systems LLC > > Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) > Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ > > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > * [email protected] > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > Confidentiality Notice: This e-mail, and any attachment to it, contains > privileged and confidential information intended only for the use of the > individual(s) or entity named on the e-mail. If the reader of this e-mail is > not the intended recipient, or the employee or agent responsible for > delivering it to the intended recipient, you are hereby notified that reading > this e-mail is strictly prohibited. If you have received this e-mail in > error, please immediately return it to the sender and delete it from your > system. > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > * [email protected] > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
