On Friday, March 23, 2018 at 2:18:45 PM UTC-7, [email protected] wrote: > > Hi, > > At Logstash we have a enrichment filter that allows a user to query a > remote db (using Sequel) and populate a local Derby db with the results. > > The existing code is: > > > @db[loader.temp_table].multi_insert(records) > > > With large results we hit a Derby bug > <https://issues.apache.org/jira/browse/DERBY-1735>. > > I have tried paginating the insert in a transaction and its is slow but it > works. > > In researching the bug I saw that some advice was given to use the data > import facility SYSCS_UTIL.SYSCS_IMPORT_TABLE claiming that is more > robust and performant. > > Initial tests bear this out however I have to persist the file before I > can import it. To do this I am using: > > > ::File.open(import_file, "w") do |fd| > dataset = @db[loader.table] > records.each do |hash| > array = hash.values.map {|val| dataset.literal(val) } > fd.puts(array.join(", ")) > end > fd.fsync > end > > > This is OK and works really well. > > I was wondering if Sequel offers a more elegant/concise way of generating > the CSV literal string? >
I don't believe Sequel does. There is the csv_serializer model plugin, but it's for models, not datasets. I'm not sure that the way you have of generating the CSV will work correctly for all data types. For example, using the literal SQL for a string will quote the string using single quotes, with any single quotes inside the string escaped per the database's escaping rules, without escaping any commas. I'm not sure if SYSCS_UTIL.SYSCS_IMPORT_TABLE expects that syntax. You should make very sure that your method correctly handles strings and other data types in the dataset correctly, and considering using a real CSV library to generate the CSV. Thanks, Jeremy -- You received this message because you are subscribed to the Google Groups "sequel-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sequel-talk. For more options, visit https://groups.google.com/d/optout.
