On Thu, May 16, 2013 at 11:25 PM, Jack Royal-Gordon <[email protected]> wrote: > I'm using LogEntries to store my sys logs, but I'm apparently generating a > very large log volume (~87GB/~270M events per day). My app has relatively > few users (100) who use it very lightly (16,665 HTTP Requests per day). The > main cause seems to be that my inserts can be very large (including web pages > that I have scraped), and when they are logged it includes the entire > contents of the web page (over 1000 lines per insert, 25,000 inserts into the > web page table per day). I have asked Heroku about any of the following: > > 1) Shorten the logging entry from Postgres inserts to truncate the field > listings > 2) Disable the logging of long-running Heroku commands > > The response I got is that none of these is possible, and yet LogEntries > Support tells me that they had another user in the past who was able to get > Heroku to stop logging from their Postgres database.
The problem of log verbosity has been without satisfactory resolution for some time. Sorry about that. As for disabling logs: depending on the phase of the moon, a member of the engineering staff servicing a support request might opt to tweak logging settings on an experimental basis, with a slew of caveats. However, those cases are generally left to individual discretion: it's not a feature of the service. > Does anyone know how to accomplish one of these tweaks? On 9.2, truncation or drops of logs do happen at 10KB, which may or may not prove to truncate in your case. It also supports multi-line queries without log record fragmentation, so one will receive one log records with literal newlines in it (in this aspect, logplex is more expressive than syslog), so it will not produce one log record per line, rather, it will produce one log record per query. I'm going to guess this will make some logging support software work a lot better (since they typically can't deal with Postgres' fragmentation) or worse (if they choke on literal newlines in the output). I haven't heard any bug reports either way, so if you feel inclined to relate your experiences, I'd like to read about them. For large INSERTs, you can also use COPY, which is recommended for bulk loading. It does not consider the contents being loaded as query text to be logged, unlike INSERT; only the COPY statement itself will be logged, so you'd likely get 25,000 log records a day in the above case. COPY's documentation is here: http://www.postgresql.org/docs/9.2/static/sql-copy.html Many drivers implement support for it: (libpq-based) Ruby pg: http://deveiate.org/code/pg/PG/Connection.html#method-i-put_copy_data Python psycopg2: http://initd.org/psycopg/docs/cursor.html#cursor.copy_from (Java) JDBC: http://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyManager.html -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
