Hi All, Is there a best practice for keeping logs for more than 30 days yet in an easily readable format?
Recently we had an issue and wanted to check the logs of the previous month. Since the logs are only kept for 30 days in *Log Viewer *we couldn't find it there. We have previously setup a sink to BigQuery, but although the logs seems to be there, we couldn't find the logs we're looking for! Not sure if this is lack of experience with BigQuery or is it that only the request logs coming from actual users are the ones kept. We are looking for a particular monthly CRON job logs, which we couldn't find in the backed up logs on BigQuery for some reason. We thought we would setup another sink to Cloud storage for the future, but it seems to be saved in JSON format, which includes too much information that makes it hard to read in critical cases where we want to see normal log flow as it appears in *Log Viewer. * We used to use instructions in the Download logs page <https://cloud.google.com/appengine/docs/standard/python/tools/downloading-logs> using the command *appcfg.py request_logs *to download logs periodically, but we stopped doing that a while ago. The problem with this method is that instead of showing the date and time as they appear on the *Log Viewer *page*, *they appear in what I assume to be an Epoch format which still makes it hard to read, example: *1:1565112559.564261** log line...* *3:1565112559.564692 log line...* *...* furthermore this method seems to be on its way to be deprecated as mentioned on the download logs page and the new *gcloud* tool does not have a download option *Warning:* The following describes using the appcfg tool to download logs. > This tool is now deprecated > <https://cloud.google.com/appengine/docs/standard/python/sdk-gcloud-migration>. > > Currently the replacement, Cloud SDK <https://cloud.google.com/sdk/docs/>, > does not support this download capability; however, you can view your logs > using the gcloud app log commands > <https://cloud.google.com/sdk/gcloud/reference/app/logs/>. I am not sure why it is not easy to export log viewer as it is, it's really very convenient as is, we just a way to download it straight forwardly. Resources*:* 1. Export logs documentation <https://cloud.google.com/logging/docs/export/?hl=en_US> 2. Download logs documentation <https://cloud.google.com/appengine/docs/standard/python/tools/downloading-logs> *Questions:* 1. Knowing that we have a sink setup to BigQuery, is there a way to fetch the logs of the previous month CORN job? Was it exported as part of the sink? How can we filter through BigQuery to fetch those logs? 2. Is there a way to export logs to keep for more than 30 days, without changing the log viewer format? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0f5aad1c-dba6-4c88-a2f7-e5da6e4975b2%40googlegroups.com.
