Re: How to run an external command that must run with the server
Yes, it runs constantly, whenever the server is up. Ok, I'll stick with the management command as a supervisor process, thank you! On Tuesday, 3 April 2018 14:09:40 UTC+1, Ken Whitesell wrote: > > There's an aspect of your situation which isn't entirely clear to me - is > this management command one that remains running all the time, like a > Celery task would be; or is it one that starts, runs a process for a period > of time, and then ends - only to be restarted at a later time? If the > former, then yes, running the management command from supervisor makes > sense to me. If the latter, then I'd probably be looking at doing something > different. > > > Ken > > > On April 3, 2018 at 6:15 AM 'Alex' via Django users < > django...@googlegroups.com > wrote: > > That makes sense. So leave the source as a management command (as it is > now), and just run python manage.py source through supervisor? > > > On Sunday, 1 April 2018 13:16:38 UTC+1, Ken Whitesell wrote: > > We set up all our Django-related processes as a group under supervisor. > This includes our celery-based processes. (We have some long-running tasks > that are kicked off by the web site.) By setting it up as a group, we can > manage all the different processes as a set. > > Whether or not that's the "best" way is possibly debatable. But it works > for us and doesn't give us any problems. > > Ken > > On 4/1/2018 7:06 AM, 'Alex' via Django users wrote: > > I have a daphne server running a django channels application. I also have > a python script that aggregates data from various sources, and sticks it > into the channel layer (called source.py). At the moment, I run it as a > management command (python manage.py source). It is nearly time for > deployment(!), so I'm moving towards production solutions. Daphne itself > currently runs under supervisor. > > My question is, what is the best way to run source.py? As a management > command, also under supervisor? Using celery? In some other way? Since > source.py feeds into the channel layer, it needs access to settings.py in > order to identify the details of the channel layer etc. > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users...@googlegroups.com. > To post to this group, send email to django...@googlegroups.com. > Visit this group at https://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com > > <https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users...@googlegroups.com . > To post to this group, send email to django...@googlegroups.com > . > Visit this group at https://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/432d449c-d089-43cd-a739-702847729b6b%40googlegroups.com > > <https://groups.google.com/d/msgid/django-users/432d449c-d089-43cd-a739-702847729b6b%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/bfa13b9b-8a7a-41e4-8d40-cf21cf6448b7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How to run an external command that must run with the server
That makes sense. So leave the source as a management command (as it is now), and just run python manage.py source through supervisor? On Sunday, 1 April 2018 13:16:38 UTC+1, Ken Whitesell wrote: > > We set up all our Django-related processes as a group under supervisor. > This includes our celery-based processes. (We have some long-running tasks > that are kicked off by the web site.) By setting it up as a group, we can > manage all the different processes as a set. > > Whether or not that's the "best" way is possibly debatable. But it works > for us and doesn't give us any problems. > > Ken > > On 4/1/2018 7:06 AM, 'Alex' via Django users wrote: > > I have a daphne server running a django channels application. I also have > a python script that aggregates data from various sources, and sticks it > into the channel layer (called source.py). At the moment, I run it as a > management command (python manage.py source). It is nearly time for > deployment(!), so I'm moving towards production solutions. Daphne itself > currently runs under supervisor. > > My question is, what is the best way to run source.py? As a management > command, also under supervisor? Using celery? In some other way? Since > source.py feeds into the channel layer, it needs access to settings.py in > order to identify the details of the channel layer etc. > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users...@googlegroups.com . > To post to this group, send email to django...@googlegroups.com > . > Visit this group at https://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com > > <https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/432d449c-d089-43cd-a739-702847729b6b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to run an external command that must run with the server
I have a daphne server running a django channels application. I also have a python script that aggregates data from various sources, and sticks it into the channel layer (called source.py). At the moment, I run it as a management command (python manage.py source). It is nearly time for deployment(!), so I'm moving towards production solutions. Daphne itself currently runs under supervisor. My question is, what is the best way to run source.py? As a management command, also under supervisor? Using celery? In some other way? Since source.py feeds into the channel layer, it needs access to settings.py in order to identify the details of the channel layer etc. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Overriding channel-layer's group_send and group_add to add persistence
I have a partial solution to this, if anybody is interested. For some
reason, creating a new version of the core.py file in which
RedisChannelLayer is defined, and editing it directly, gave the desired
results. It was something about overriding only part of it, but I have no
idea what!
On Friday, 23 March 2018 09:56:13 UTC, Alex wrote:
>
> I've been trying to add persistence to channel layers, such that each new
> consumer joining a group is sent the most recent message from that group,
> on connect. Below are my attempts. For some reason, the message in the
> highlighted line always seems to be of type 'None'. Am I going about this
> completely incorrectly? I'd be really grateful for any help.
>
>
> from channels_redis.core import RedisChannelLayer
> from channels.exceptions import ChannelFull
> import time
>
>
> class RedisChannelLayerGroupPersistence(RedisChannelLayer):
>
>
>
>
> async def group_send(self, group, message):
> """
> Sends a message to the entire group.
> """
> assert self.valid_group_name(group), "Group name not valid"
> # Retrieve list of all channel names
> key = self._group_key(group)
> pers_key = str(key) + "_PERS"
> async with self.connection(self.consistent_hash(group)) as
> connection:
> # Discard old channels based on group_expiry
> await connection.zremrangebyscore(key, min=0, max=int(time.
> time()) - self.group_expiry)
> # Return current lot
> channel_names = [
> x.decode("utf8") for x in
> await connection.zrange(key, 0, -1)
> ]
> # TODO: More efficient implementation (lua script per shard?)
> try:
> await connection.persist(pers_key)
> await connection.set(pers_key, str(message))
> print("TYPE =
> {}".format(type(str(
> message
>
>
> for channel in channel_names:
> try:
> await self.send(channel, message)
> except ChannelFull:
> pass
>
>
> async def group_add(self, group, channel):
>
>
> """
> Adds the channel name to a group.
> """
> # Check the inputs
> assert self.valid_group_name(group), "Group name not valid"
> assert self.valid_channel_name(channel), "Channel name not valid"
> # Get a connection to the right shard
> group_key = self._group_key(group)
> pers_key = str(group_key) + "_PERS"
> async with self.connection(self.consistent_hash(group)) as
> connection:
> message = await connection.get(pers_key) #ISSUE HERE
> -- MESSAGE IS NONE
> # Add to group sorted set with creation time as timestamp
> await connection.zadd(
> group_key,
> time.time(),
> channel,
> )
> # Set expiration to be group_expiry, since everything in
> # it at this point is guaranteed to expire before that
> try:
> await self.send(channel, str(message))
> except ChannelFull:
> pass
>
>
>
>
> await connection.expire(group_key, self.group_expiry)
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/5fa1bdd4-07e1-42d8-8bc0-2662e754eb77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Overriding channel-layer's group_send and group_add to add persistence
I've just checked, and self.consistent_hash(group) returns '0' in both
functions, so it should be connecting to the same db...
On Friday, 23 March 2018 16:05:16 UTC, Andrew Godwin wrote:
>
> I would check the connection is going to the right server/database as
> well? But past that, I can't help you - I'd try doing some things with
> plain aioredis to see if you can replicate it there.
>
> Andrew
>
> On Fri, Mar 23, 2018 at 9:01 AM, 'Alex' via Django users <
> django...@googlegroups.com > wrote:
>
>> Hi,
>>
>> I've used the redis-cli to get the contents of the key, and it has filled
>> it properly, so the information is definitely in redis under that key. The
>> issue seems to be that message = await connection.get(pers_key) always
>> returns none. One thing I'm certain of is that it's in redis!
>>
>> Alex
>>
>> On Friday, 23 March 2018 15:58:37 UTC, Andrew Godwin wrote:
>>>
>>> It looks correct at first glance - I would insert a debugger there and
>>> see what the Redis database contained manually at that point.
>>>
>>> Andrew
>>>
>>> On Fri, Mar 23, 2018 at 2:56 AM, 'Alex' via Django users <
>>> django...@googlegroups.com> wrote:
>>>
>>>> I've been trying to add persistence to channel layers, such that each
>>>> new consumer joining a group is sent the most recent message from that
>>>> group, on connect. Below are my attempts. For some reason, the message in
>>>> the highlighted line always seems to be of type 'None'. Am I going about
>>>> this completely incorrectly? I'd be really grateful for any help.
>>>>
>>>>
>>>> from channels_redis.core import RedisChannelLayer
>>>> from channels.exceptions import ChannelFull
>>>> import time
>>>>
>>>>
>>>> class RedisChannelLayerGroupPersistence(RedisChannelLayer):
>>>>
>>>>
>>>>
>>>>
>>>> async def group_send(self, group, message):
>>>> """
>>>> Sends a message to the entire group.
>>>> """
>>>> assert self.valid_group_name(group), "Group name not valid"
>>>> # Retrieve list of all channel names
>>>> key = self._group_key(group)
>>>> pers_key = str(key) + "_PERS"
>>>> async with self.connection(self.consistent_hash(group)) as
>>>> connection:
>>>> # Discard old channels based on group_expiry
>>>> await connection.zremrangebyscore(key, min=0, max=int(time.
>>>> time()) - self.group_expiry)
>>>> # Return current lot
>>>> channel_names = [
>>>> x.decode("utf8") for x in
>>>> await connection.zrange(key, 0, -1)
>>>> ]
>>>> # TODO: More efficient implementation (lua script per shard?)
>>>> try:
>>>> await connection.persist(pers_key)
>>>> await connection.set(pers_key, str(message))
>>>> print("TYPE =
>>>> {}".format(type(str(
>>>> message
>>>>
>>>>
>>>> for channel in channel_names:
>>>> try:
>>>> await self.send(channel, message)
>>>> except ChannelFull:
>>>> pass
>>>>
>>>>
>>>> async def group_add(self, group, channel):
>>>>
>>>>
>>>> """
>>>> Adds the channel name to a group.
>>>> """
>>>> # Check the inputs
>>>> assert self.valid_group_name(group), "Group name not valid"
>>>> assert self.valid_channel_name(channel), "Channel name not
>>>> valid"
>>>> # Get a connection to the right shard
>>>> group_key = self._group_key(group)
>>>> pers_key = str(group_key) + "_PERS"
>>>> async with self.connection(self.consistent_hash(group)) as
>>>> connection:
>>>> message = await connection.get(pers_key) #ISSUE HERE
>>>> -- MESSAGE IS NONE
>>>> # Add to group sorted set with creation time as times
Re: Overriding channel-layer's group_send and group_add to add persistence
That did occur to me, but both connect using the function below, so unless
they're somehow referring to different groups, I'm not sure how they could
be going to different servers/databases...
async with self.connection(self.consistent_hash(group)) as connection:
On Friday, 23 March 2018 16:05:16 UTC, Andrew Godwin wrote:
>
> I would check the connection is going to the right server/database as
> well? But past that, I can't help you - I'd try doing some things with
> plain aioredis to see if you can replicate it there.
>
> Andrew
>
> On Fri, Mar 23, 2018 at 9:01 AM, 'Alex' via Django users <
> django...@googlegroups.com > wrote:
>
>> Hi,
>>
>> I've used the redis-cli to get the contents of the key, and it has filled
>> it properly, so the information is definitely in redis under that key. The
>> issue seems to be that message = await connection.get(pers_key) always
>> returns none. One thing I'm certain of is that it's in redis!
>>
>> Alex
>>
>> On Friday, 23 March 2018 15:58:37 UTC, Andrew Godwin wrote:
>>>
>>> It looks correct at first glance - I would insert a debugger there and
>>> see what the Redis database contained manually at that point.
>>>
>>> Andrew
>>>
>>> On Fri, Mar 23, 2018 at 2:56 AM, 'Alex' via Django users <
>>> django...@googlegroups.com> wrote:
>>>
>>>> I've been trying to add persistence to channel layers, such that each
>>>> new consumer joining a group is sent the most recent message from that
>>>> group, on connect. Below are my attempts. For some reason, the message in
>>>> the highlighted line always seems to be of type 'None'. Am I going about
>>>> this completely incorrectly? I'd be really grateful for any help.
>>>>
>>>>
>>>> from channels_redis.core import RedisChannelLayer
>>>> from channels.exceptions import ChannelFull
>>>> import time
>>>>
>>>>
>>>> class RedisChannelLayerGroupPersistence(RedisChannelLayer):
>>>>
>>>>
>>>>
>>>>
>>>> async def group_send(self, group, message):
>>>> """
>>>> Sends a message to the entire group.
>>>> """
>>>> assert self.valid_group_name(group), "Group name not valid"
>>>> # Retrieve list of all channel names
>>>> key = self._group_key(group)
>>>> pers_key = str(key) + "_PERS"
>>>> async with self.connection(self.consistent_hash(group)) as
>>>> connection:
>>>> # Discard old channels based on group_expiry
>>>> await connection.zremrangebyscore(key, min=0, max=int(time.
>>>> time()) - self.group_expiry)
>>>> # Return current lot
>>>> channel_names = [
>>>> x.decode("utf8") for x in
>>>> await connection.zrange(key, 0, -1)
>>>> ]
>>>> # TODO: More efficient implementation (lua script per shard?)
>>>> try:
>>>> await connection.persist(pers_key)
>>>> await connection.set(pers_key, str(message))
>>>> print("TYPE =
>>>> {}".format(type(str(
>>>> message
>>>>
>>>>
>>>> for channel in channel_names:
>>>> try:
>>>> await self.send(channel, message)
>>>> except ChannelFull:
>>>> pass
>>>>
>>>>
>>>> async def group_add(self, group, channel):
>>>>
>>>>
>>>> """
>>>> Adds the channel name to a group.
>>>> """
>>>> # Check the inputs
>>>> assert self.valid_group_name(group), "Group name not valid"
>>>> assert self.valid_channel_name(channel), "Channel name not
>>>> valid"
>>>> # Get a connection to the right shard
>>>> group_key = self._group_key(group)
>>>> pers_key = str(group_key) + "_PERS"
>>>> async with self.connection(self.consistent_hash(group)) as
>>>> connection:
>>>> message = await connection.get(pers_k
Re: Overriding channel-layer's group_send and group_add to add persistence
Hi,
I've used the redis-cli to get the contents of the key, and it has filled
it properly, so the information is definitely in redis under that key. The
issue seems to be that message = await connection.get(pers_key) always
returns none. One thing I'm certain of is that it's in redis!
Alex
On Friday, 23 March 2018 15:58:37 UTC, Andrew Godwin wrote:
>
> It looks correct at first glance - I would insert a debugger there and see
> what the Redis database contained manually at that point.
>
> Andrew
>
> On Fri, Mar 23, 2018 at 2:56 AM, 'Alex' via Django users <
> django...@googlegroups.com > wrote:
>
>> I've been trying to add persistence to channel layers, such that each new
>> consumer joining a group is sent the most recent message from that group,
>> on connect. Below are my attempts. For some reason, the message in the
>> highlighted line always seems to be of type 'None'. Am I going about this
>> completely incorrectly? I'd be really grateful for any help.
>>
>>
>> from channels_redis.core import RedisChannelLayer
>> from channels.exceptions import ChannelFull
>> import time
>>
>>
>> class RedisChannelLayerGroupPersistence(RedisChannelLayer):
>>
>>
>>
>>
>> async def group_send(self, group, message):
>> """
>> Sends a message to the entire group.
>> """
>> assert self.valid_group_name(group), "Group name not valid"
>> # Retrieve list of all channel names
>> key = self._group_key(group)
>> pers_key = str(key) + "_PERS"
>> async with self.connection(self.consistent_hash(group)) as
>> connection:
>> # Discard old channels based on group_expiry
>> await connection.zremrangebyscore(key, min=0, max=int(time.
>> time()) - self.group_expiry)
>> # Return current lot
>> channel_names = [
>> x.decode("utf8") for x in
>> await connection.zrange(key, 0, -1)
>> ]
>> # TODO: More efficient implementation (lua script per shard?)
>> try:
>> await connection.persist(pers_key)
>> await connection.set(pers_key, str(message))
>> print("TYPE =
>> {}".format(type(str(
>> message
>>
>>
>> for channel in channel_names:
>> try:
>> await self.send(channel, message)
>> except ChannelFull:
>> pass
>>
>>
>> async def group_add(self, group, channel):
>>
>>
>> """
>> Adds the channel name to a group.
>> """
>> # Check the inputs
>> assert self.valid_group_name(group), "Group name not valid"
>> assert self.valid_channel_name(channel), "Channel name not valid"
>> # Get a connection to the right shard
>> group_key = self._group_key(group)
>> pers_key = str(group_key) + "_PERS"
>> async with self.connection(self.consistent_hash(group)) as
>> connection:
>> message = await connection.get(pers_key) #ISSUE HERE
>> -- MESSAGE IS NONE
>> # Add to group sorted set with creation time as timestamp
>> await connection.zadd(
>> group_key,
>> time.time(),
>> channel,
>> )
>> # Set expiration to be group_expiry, since everything in
>> # it at this point is guaranteed to expire before that
>> try:
>> await self.send(channel, str(message))
>> except ChannelFull:
>> pass
>>
>>
>>
>>
>> await connection.expire(group_key, self.group_expiry)
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/abc8747d-8d80-4ec4-a2ca-d5e91c161c08%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/django-users/abc8747d-8d80-4ec4-a2ca-d5e91c161c08%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/24613c4d-c566-4631-97cd-e858d286aadf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Overriding channel-layer's group_send and group_add to add persistence
I've been trying to add persistence to channel layers, such that each new
consumer joining a group is sent the most recent message from that group,
on connect. Below are my attempts. For some reason, the message in the
highlighted line always seems to be of type 'None'. Am I going about this
completely incorrectly? I'd be really grateful for any help.
from channels_redis.core import RedisChannelLayer
from channels.exceptions import ChannelFull
import time
class RedisChannelLayerGroupPersistence(RedisChannelLayer):
async def group_send(self, group, message):
"""
Sends a message to the entire group.
"""
assert self.valid_group_name(group), "Group name not valid"
# Retrieve list of all channel names
key = self._group_key(group)
pers_key = str(key) + "_PERS"
async with self.connection(self.consistent_hash(group)) as
connection:
# Discard old channels based on group_expiry
await connection.zremrangebyscore(key, min=0, max=int(time.time
()) - self.group_expiry)
# Return current lot
channel_names = [
x.decode("utf8") for x in
await connection.zrange(key, 0, -1)
]
# TODO: More efficient implementation (lua script per shard?) try:
await connection.persist(pers_key)
await connection.set(pers_key, str(message))
print("TYPE =
{}".format(type(str(message
for channel in channel_names:
try:
await self.send(channel, message)
except ChannelFull:
pass
async def group_add(self, group, channel):
"""
Adds the channel name to a group.
"""
# Check the inputs
assert self.valid_group_name(group), "Group name not valid"
assert self.valid_channel_name(channel), "Channel name not valid"
# Get a connection to the right shard
group_key = self._group_key(group)
pers_key = str(group_key) + "_PERS"
async with self.connection(self.consistent_hash(group)) as
connection:
message = await connection.get(pers_key) #ISSUE HERE
-- MESSAGE IS NONE
# Add to group sorted set with creation time as timestamp
await connection.zadd(
group_key,
time.time(),
channel,
)
# Set expiration to be group_expiry, since everything in
# it at this point is guaranteed to expire before that
try:
await self.send(channel, str(message))
except ChannelFull:
pass
await connection.expire(group_key, self.group_expiry)
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/abc8747d-8d80-4ec4-a2ca-d5e91c161c08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
