Re: ORM for structured Logs

[guettli]

Thank you for sharing. 

Yes, the model is not difficult, that's why everybody helps himself up to 
now. Maybe this is the best solution and there is not much in common.

The things which are in common (ORM, Admin-Interface) are already provided 
by django.

It's just a bit of glue-code to get it working.

Again, thank you for sharing your solution.

Regards,
  Thomas

Am Samstag, 8. April 2017 18:58:59 UTC+2 schrieb Scot Hacker:
>
> On Tuesday, April 4, 2017 at 5:18:42 AM UTC-7, guettli wrote:
>>
>> In the past I was told: Don't store logs in the database.
>>
>
> For general purposes, I agree with this. Logging is a python standard, 
> logs can be verbose, logrolling solutions are well established (and built 
> in), etc. However, there are certain situations or activities where 
> database logging makes sense, most likely *in addition to* standard logging 
> rather than instead of. In one of my projects, half a dozen non-technical 
> managers need the ability to track certain types of actions (related to 
> account activations at a school). For this, I developed a simple ORM 
> logging solution that lets those managers search and filter these special 
> logs in the Django admin. 
>
> It's not something that really deserves to be its own project, IMO - just 
> a typical thing a dev might do in Django to satisfy an institutional need. 
> But I've put my solution in this gist, in case its helpful:
>
> https://gist.github.com/shacker/05bc1de527a2d7412de361ac659aecde
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/e20e56bd-e82c-4027-afa2-bc08ca35c00b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[Scot Hacker]

On Tuesday, April 4, 2017 at 5:18:42 AM UTC-7, guettli wrote:
>
> In the past I was told: Don't store logs in the database.
>

For general purposes, I agree with this. Logging is a python standard, logs 
can be verbose, logrolling solutions are well established (and built in), 
etc. However, there are certain situations or activities where database 
logging makes sense, most likely *in addition to* standard logging rather 
than instead of. In one of my projects, half a dozen non-technical managers 
need the ability to track certain types of actions (related to account 
activations at a school). For this, I developed a simple ORM logging 
solution that lets those managers search and filter these special logs in 
the Django admin. 

It's not something that really deserves to be its own project, IMO - just a 
typical thing a dev might do in Django to satisfy an institutional need. 
But I've put my solution in this gist, in case its helpful:

https://gist.github.com/shacker/05bc1de527a2d7412de361ac659aecde
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c9486b92-010d-445b-b137-1b18f7ca5efb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[guettli]

Hi Marten,

Am Donnerstag, 6. April 2017 14:10:58 UTC+2 schrieb knbk:
>
> Hi Thomas,
>
> The primary purpose of logging is to catch and examine errors. If 
> something went wrong, you want to know *when *and *why*. Logging to a 
> database increases the complexity and greatly increases the number of 
> things that can go wrong. The last thing you want to find out when 
> retracing an error is that you don't have any logs because the logging 
> system failed. You may also need to log errors that happened during 
> startup, before a database connection can be established. Logging to file 
> is the simplest method, and has the least chance of failure. That's why you 
> should always log to file. 
>
> The two options are not mutually exclusive. Like you said, times have 
> changed, and the overhead to store logs both in a file and in a database 
> are nowadays acceptable. If you have a good reason to store the logs in a 
> database, then go ahead. Just remember that it should be *in addition to 
> *file-based 
> logging. 
>
>
Yes, you are right. During the initialization of processes no db 
connections exists yet. I don't like redundancy but here its needed for a 
higher availability. 

Thank you for your reply.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a9a0fe4e-4b65-4ceb-8336-cc4ceeb82f2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[guettli]

Am Donnerstag, 6. April 2017 10:42:17 UTC+2 schrieb Christian Ledermann:
>
> On 6 April 2017 at 09:15, guettli  
> wrote: 
> > Hello Brick Wall, how are you doing? 
>
> Hello Stonemason. 
>
> What is your question? 
>
>

It was more an idea than a question.

The question could be: What do you think about this idea?

I am lazy and like postgresql. Up to now we store are logs in files and we 
search for an alternative.

I am unfamiliar with ElasticSearch (DB of ELK-Stack).

ELK-Stack is very popular, but maybe is overrated. I don't know. Why not 
use PostgreSQL?



 

> I do not have a strong opinion on your approach - i don't even know 
> the problem you are trying to solve. 
> or how big your logs are. a couple of KB per day or some GB per hour? 
>
>
Traffic does not play any role here. This question is only about the 
structure (tables and columns), not about the rows.

 

> [the brickwall shrugs its shoulders] 
>
>

Thank you for your reply.


 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/cc315693-bd7b-47a0-bb54-c83a822bf4a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[knbk]

Hi Thomas,

The primary purpose of logging is to catch and examine errors. If something 
went wrong, you want to know *when *and *why*. Logging to a database 
increases the complexity and greatly increases the number of things that 
can go wrong. The last thing you want to find out when retracing an error 
is that you don't have any logs because the logging system failed. You may 
also need to log errors that happened during startup, before a database 
connection can be established. Logging to file is the simplest method, and 
has the least chance of failure. That's why you should always log to file. 

The two options are not mutually exclusive. Like you said, times have 
changed, and the overhead to store logs both in a file and in a database 
are nowadays acceptable. If you have a good reason to store the logs in a 
database, then go ahead. Just remember that it should be *in addition to 
*file-based 
logging. 

Kind regards,

Marten

On Tuesday, April 4, 2017 at 2:18:42 PM UTC+2, guettli wrote:
>
> In the past I was told: Don't store logs in the database.
>
> Time (and hardware) has changed.
>
> I think it is time to store logs where I have great tools for analyzing 
> logs.
>
> Storing in a model which I can access via django orm is my current 
> strategy.
>
> It seems no one has done this before. I could not find such a project up 
> to now.
>
> My first draft looks like this:
>
>
> class Log(models.Model):
> datetime=models.DateTimeField(default=datetime.datetime.now, 
> db_index=True)
> data=jsonfield.JSONField()
> host=models.CharField(max_length=256, default='localhost', db_index=True)
> system=models.CharField(max_length=256, default='', db_index=True)
>
> I am missing two things:
>
>  Missing1: Log level: INFO, WARN, ...
>
>  Missing2: A way to store exceptions.
>
>
> What do you think?
>
> What's wrong with this, what could be improved?
>
> Regards,
>   Thomas Güttler
>
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/21746446-93c7-40d7-b2fa-6adaad9f4aee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[Christian Ledermann]

On 6 April 2017 at 09:15, guettli  wrote:
> Hello Brick Wall, how are you doing?

Hello Stonemason.

What is your question?

I do not have a strong opinion on your approach - i don't even know
the problem you are trying to solve.
or how big your logs are. a couple of KB per day or some GB per hour?

[the brickwall shrugs its shoulders]

>
> Am Dienstag, 4. April 2017 14:18:42 UTC+2 schrieb guettli:
>>
>> In the past I was told: Don't store logs in the database.
>>
>> Time (and hardware) has changed.
>>
>> I think it is time to store logs where I have great tools for analyzing
>> logs.
>>
>> Storing in a model which I can access via django orm is my current
>> strategy.
>>
>> It seems no one has done this before. I could not find such a project up
>> to now.
>>
>> My first draft looks like this:
>>
>>
>> class Log(models.Model):
>> datetime=models.DateTimeField(default=datetime.datetime.now,
>> db_index=True)
>> data=jsonfield.JSONField()
>> host=models.CharField(max_length=256, default='localhost',
>> db_index=True)
>> system=models.CharField(max_length=256, default='', db_index=True)
>>
>> I am missing two things:
>>
>>  Missing1: Log level: INFO, WARN, ...
>>
>>  Missing2: A way to store exceptions.
>>
>>
>> What do you think?
>>
>> What's wrong with this, what could be improved?
>>
>> Regards,
>>   Thomas Güttler
>>
>>
>>
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/c8742484-a252-4c05-939a-f7c3f2b4f505%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



-- 
Best Regards,

Christian Ledermann

Newark-on-Trent - UK
Mobile : +44 7474997517

https://uk.linkedin.com/in/christianledermann
https://github.com/cleder/


<*)))>{

If you save the living environment, the biodiversity that we have left,
you will also automatically save the physical environment, too. But If
you only save the physical environment, you will ultimately lose both.

1) Don’t drive species to extinction

2) Don’t destroy a habitat that species rely on.

3) Don’t change the climate in ways that will result in the above.

}<(((*>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CABCjzWr3G%3DNnqkx75CEEXZ3bMXsV9Wd31A2vhMAxytjJREEBEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM for structured Logs

[guettli]

Hello Brick Wall, how are you doing?

Am Dienstag, 4. April 2017 14:18:42 UTC+2 schrieb guettli:
>
> In the past I was told: Don't store logs in the database.
>
> Time (and hardware) has changed.
>
> I think it is time to store logs where I have great tools for analyzing 
> logs.
>
> Storing in a model which I can access via django orm is my current 
> strategy.
>
> It seems no one has done this before. I could not find such a project up 
> to now.
>
> My first draft looks like this:
>
>
> class Log(models.Model):
> datetime=models.DateTimeField(default=datetime.datetime.now, 
> db_index=True)
> data=jsonfield.JSONField()
> host=models.CharField(max_length=256, default='localhost', db_index=True)
> system=models.CharField(max_length=256, default='', db_index=True)
>
> I am missing two things:
>
>  Missing1: Log level: INFO, WARN, ...
>
>  Missing2: A way to store exceptions.
>
>
> What do you think?
>
> What's wrong with this, what could be improved?
>
> Regards,
>   Thomas Güttler
>
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c8742484-a252-4c05-939a-f7c3f2b4f505%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.