Re: django.contrib.auth CLI

[Jamesie Pic]

Hi smart nerds,

I'm sorry if my sense of humour was too much to handle for you.

But to compensate I offer the unfrustrating python CLI package, and at the
same time most awesome CLI CRUD for Django to my knowledge, on
yourlabs.io/oss/clilabs

One last thing, verrry important for the record, in case anybody reading
wants to learn eXtreme DevOps: making a django app is an horrible
suggestion, because automation tool should be automatically made available,
pip install is automatable so it works, we can have an automatic pre-task
that installs it, but adding to INSTALLED_APPS is a manual step, which
makes django-apps and management commands that require in-project
installation a horrible mistake, typical of "going one step forward and
then one step back" in automation.

Exactly why this is a python package, with support for django, from the
beginning. Maybe if django supports a system like pluggy this can change,
but for now making this a django app would be counter productive.

Have a great day

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1-sPCvhS_cqzZ8L4QqjdT83CnKhgqwZ2516tiiV8LobUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: QuerySet.iterator together with prefetch_related because of chunk_size

[Curtis Maloney]

On 10/12/18 10:51 AM, charettes wrote:

Hello Tobias,

 From my understanding the introduction of chunk_size doest't help here.

The fundamental reason why iterator() cannot be used with 
prefetch_related() is that the latter requires a set of model instance 
to be materialized to work appropriately which chunk_size doesn't 
control at all.


In other words chunk_size only controls how many rows should be fetched 
from the database cursor and kept into memory at a time. Even when this 
parameter is used, iterator() will only materialize a single model 
instance per yield.


I inferred from the original post they were suggesting to do a set of 
prefetch_related "filling" queries _per_ _chunk_


It wouldn't be as efficient as the 1+P (where P is number of prefetched 
relations) of a normal prefetch_related, but at 1+(P x Chunks) it would 
_probably_ be more efficient than the 1+(P x N-rows) it would otherwise 
be [or worse!]


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fad10ba6-783d-bfbe-6b78-2500e475983f%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: QuerySet.iterator together with prefetch_related because of chunk_size

[charettes]

Hello Tobias,

>From my understanding the introduction of chunk_size doest't help here.

The fundamental reason why iterator() cannot be used with 
prefetch_related() is that the latter requires a set of model instance to 
be materialized to work appropriately which chunk_size doesn't control at 
all.

In other words chunk_size only controls how many rows should be fetched 
from the database cursor and kept into memory at a time. Even when this 
parameter is used, iterator() will only materialize a single model instance 
per yield.

Now, something you could do if you really want to be using prefetch_related 
with iterator() is to materialize chunks and use the now public 
prefetch_related_objects[0] helper to prefetch relationships.

A somewhat complete implementation would look like

def prefetch_related_iterator(queryset, *related_lookups, chunk_size=100):
iterator = queryset.iterator(chunk_size=chunk_size)
while True:
chunk = list(itertools.islice(iterator, chunk_size))
if not chunk:
return
prefetch_related_objects(chunk, *related_lookups)
yield from chunk

Given that iterator() always ignored prefetch related lookups instead of 
erroring out when they were specified make me feel like turning such a 
feature on by default could be problematic as it could balloon the memory 
usage which is the main reason why iterator is useful anyway.

Cheers,
Simon

[0] 
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#django.db.models.prefetch_related_objects

Le jeudi 11 octobre 2018 15:44:06 UTC-4, tobias@truffls.com a écrit :
>
> Hi everyone!
>
> The docs (
> https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator) 
> state that the
>
>  use of iterator() causes previous prefetch_related() calls to be ignored 
>> since these two optimizations do not make sense together.
>
>
> I am wondering, if this is still true with the introduction of the 
> *chunk_size* parameter. Having only one additional query per chunk could 
> heavily reduce the amount of queries per batch.
>
> Best regards
> Tobi
>
>
>
> *Truffls GmbH*
> Chausseestraße 86, 10115 Berlin 
> 
> https://truffls.de
>
> Vertretungsberechtigte Geschäftsführer: Clemens Dittrich, Matthes Dohmeyer 
> Amtsgericht Charlottenburg, HRB 160036B
>
> Diese Nachricht ist vertraulich. Sie darf ausschließlich durch den 
> vorgesehenen Empfänger und Adressaten gelesen, kopiert oder genutzt werden. 
> Sollten Sie diese Nachricht versehentlich erhalten haben, bitten wir, den 
> Absender (durch Antwort-E-Mail) hiervon unverzüglich zu informieren und die 
> Nachricht zu löschen. Jede Nutzung oder Weitergabe des Inhalts dieser 
> Nachricht ist unzulässig.
> This message (including any attachments) is confidential and may 
> be privileged. It may be read, copied and used only by the intended 
> recipient. If you have received it in error please contact the sender (by 
> return e-mail) immediately and delete this message. Any unauthorized use or 
> dissemination of this message in whole or in part is strictly prohibited.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/afe9c247-45ac-4f63-8673-2b8b4c337d54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

QuerySet.iterator together with prefetch_related because of chunk_size

[tobias . kroenke]

Hi everyone!

The docs 
(https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator) 
state that the

 use of iterator() causes previous prefetch_related() calls to be ignored 
> since these two optimizations do not make sense together.


I am wondering, if this is still true with the introduction of the 
*chunk_size* parameter. Having only one additional query per chunk could 
heavily reduce the amount of queries per batch.

Best regards
Tobi

-- 





Truffls GmbH
Chausseestraße 86, 10115 Berlin 


https://truffls.de 



Vertretungsberechtigte 
Geschäftsführer: Clemens Dittrich, Matthes Dohmeyer Amtsgericht 
Charlottenburg, HRB 160036B


Diese Nachricht ist vertraulich. Sie darf 
ausschließlich durch den vorgesehenen Empfänger und Adressaten gelesen, 
kopiert oder genutzt werden. Sollten Sie diese Nachricht versehentlich 
erhalten haben, bitten wir, den Absender (durch Antwort-E-Mail) hiervon 
unverzüglich zu informieren und die Nachricht zu löschen. Jede Nutzung oder 
Weitergabe des Inhalts dieser Nachricht ist unzulässig.
This message 
(including any attachments) is confidential and may be privileged. It may 
be read, copied and used only by the intended recipient. If you have 
received it in error please contact the sender (by return e-mail) 
immediately and delete this message. Any unauthorized use or dissemination 
of this message in whole or in part is strictly prohibited.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/985d983f-09e1-401c-beb8-cd9b6b60e484%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.contrib.auth CLI

[Jamesie Pic]

Forgot a note about extreme devops:

It's when you have an automatic deployment per feature branch (example
).

GitLab even has a feature to help it called dynamic environments
,
currently they only support Kubernetes out of the box, but it's also
possible with playlabs, the result of open sourcing the playbooks we've had
in production this year at the government.

Have a great day, and most importantly: have FUN :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1_%3D5zR1Q%3DQ-XvqaC9sUdo-22Bk%3DTXwUEUJA4qe68%3DFa5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.contrib.auth CLI

[Jamesie Pic]

Hello Ludovic,

I'm glad this little joke tickled you, sorry for being that childish but
sometimes I can't help it (I don't see how anybody could feel rationnaly
offended since this joke talking about me). In reality, I've just been
automating Django deployments for 10 year. So, if it takes 50 years to be
an expert in something, I'm definitely too young to be an expert in
anything ;)

Yes I have package deployment automation, because I maintain more than 30
open source packages, most of them for Django, on PyPi alone, some on NPM.
The reason it's not the point, is that my CM aims at automating deployment
of django projects, not of "django projects that have added my poor django
app that provides a fixed createsuperuser and a removeuser CLI".

So, if i'm doing this automation again (and I am), it will be in the CM,
**not** in an app, because in this case, it **doesn't make sense to do it
in an app, it would just make my CM more incompatible with OOTB Django**.
It's the third time I'm explaining this, please excuse me for giving up as
from where i'm standing it seems nobody is able to counter that objection,
but I really feel like it's being purposely ignored.

We never have two deploys "one local one production", we always have "one
local one staging one production" so that's three.

Ludovic, nowadays in the DevOps world we don't "automate deployment of
requirements.txt" anymore, we don't "compile a virtualenv on the target"
anymore, because we have separated "build-time" from "test-time" and
"deploy-time", thanks to containers.

This doesn't need to "react to changes or user feedback", the needs have
been the same for 10 years, add/remove users in an automated way.
Currently, we're injecting code in django-admin shell.

Oh what's that final line of your email you're accusing me of being passive
aggressive :D

I do not have any more energy for this contribution, I'll just add it to my
little monster crudlfap because that's my layer for any new project, as I
have exigence for Controllers to refactor view code, and working
ForeignKey/FileField forms out of the box (that means JS), now is the time
crudlfap also compensate for Django's lack of OOTB automation feature,
despite how basic they can be.

Case closed as far as I'm concerned.

My apologies to all the community for being unable to defend this proposal.
Note to self: if I couldn't defend something as basic and obvious as this,
learn that I will never be able to defend any proposal.

Keep up the great work overall, sincerely

Have a beautiful day,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1_61ysHz%2BSR-kvHuw%2Bm8eN8r_b2FGx2LHuhDLpC5t37CA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.contrib.auth CLI

[ludovic coues]

I'm sure your extreme DevOps expertise can find a way to automate the
deployment of a pip package.

I mean, you are argument about manual step for an automatic step is a straw
man.

Let's say you make two deployment of your app. One on your system, one on
premises on a customer system. You want both system to make use of a
different set of user. But you want that idenpotent function to manage user
on both system. Because that's part of your solution. You want to automate
deploying your requirements.txt more than your set of user.

Beside, these days, starting your pet Django feature as a third party
project is the recommended way to go. You will be able to react more
quickly to user demand while your feature mature and the Django project
will be able to judge how useful is your project.

Look at whitenoise or Django rest framework.

And please, read the code of conduct one more time. No subscriber of this
mailing list should have to read Neither this email nor your because we are
acting like children, being all snarky and passive aggressive.

On Oct 11, 2018 12:27, "Jamesie Pic"  wrote:

Sorry, I forgot to answer about the loaddata proposal.

So, loaddata will not do user removal for one thing.

About user creation, the process we have is:

- add the user to the users list,
- set the password in `ansible-vault create passwords/username`
- CM will execute createsuperuser for users,

>From that, it should maintain the userbase, on an ootb django project.

The workflow you are suggesting is:

- add the user to the users list,
- set the password in `ansible-vault create passwords/username`,
- somehow parse the settings to know the user model,
- generate a json file with the users, and hope that it will work with
their model,
- use a script that django doesn't have to generate a salted password,
- upload the script and apply it.

I don't think I want my automation steps to rely on hope.

What about the debugging process of such a complex implementation for such
a simple task ?
The cost is not worth the benefit. Injecting code in django-admin shell
still does the same with
a much cheaper complexity cost.

Have a great day.

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAC6Op19o1wUCd4CSvxHLKHAdBduPOO4H6PuhtuyZc_LYBO9ooA%40mail.gmail.com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAEuG%2BTYjpMJna-QScxooaLkonR9ixdBr7oJSuHpa9EyM2gi5bQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.contrib.auth CLI

[Jamesie Pic]

Sorry, I forgot to answer about the loaddata proposal.

So, loaddata will not do user removal for one thing.

About user creation, the process we have is:

- add the user to the users list,
- set the password in `ansible-vault create passwords/username`
- CM will execute createsuperuser for users,

>From that, it should maintain the userbase, on an ootb django project.

The workflow you are suggesting is:

- add the user to the users list,
- set the password in `ansible-vault create passwords/username`,
- somehow parse the settings to know the user model,
- generate a json file with the users, and hope that it will work with
their model,
- use a script that django doesn't have to generate a salted password,
- upload the script and apply it.

I don't think I want my automation steps to rely on hope.

What about the debugging process of such a complex implementation for such
a simple task ?
The cost is not worth the benefit. Injecting code in django-admin shell
still does the same with
a much cheaper complexity cost.

Have a great day.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op19o1wUCd4CSvxHLKHAdBduPOO4H6PuhtuyZc_LYBO9ooA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.contrib.auth CLI

[Jamesie Pic]

Hi Adam,

Yeah it would be funny, maybe I'm even the only Django user that automates
deployments with a CM, or to take "ootb automation experience" up to that
point (maybe because I have extreme DevOps expertise ?).

Implementing this in a third party app is pointless: it will be more useful
as scripts in the CM role that will then work with any django project, as
I've been doing so far.

Because then my generic CM recipe can deploy any django project, without
"please install my app that will provide a working CLI for impotence and
user removal" (great security feature to have by the way). Yeah, it's
almost 2019, can Django learn about post-deployment task idempotence ?

Doing this in an external app is pointless, we'd be adding a manual step to
automate another one, i'm sorry but i can't do that, it doesn't make any
sense to me.

Also, that was explained in my initial email, sorry if it wasn't clear.

Best

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op18N%3D37xwNRvB0dLJ2LDUbS2fPi_6jaYW7h_Q77%2B9ijF8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.