If the objects cannot be reclaimed due to Python objects cycles where objects
have __del__() methods, they simply are not able to be reclaimed.
You can try:
import gc
gc.collect()
but will likely have little effect.
I would suggest you also read:
http://christian.hofstaedtler.name/blog/2013/01/gc-garbage.html
Graham
On 15/02/2015, at 10:00 PM, Paul Royik <[email protected]> wrote:
> Can I force cleaning memory after algorithm has finished?
>
> On Sunday, February 15, 2015 at 12:53:30 PM UTC+2, Graham Dumpleton wrote:
>
> On 15/02/2015, at 9:41 PM, Paul Royik <[email protected]> wrote:
>
>> I reduced possibility by restricting very long and complex expressions.
>>
>> Code, that I use is simple.
>>
>> class TimeLimit(object):
>> def __init__(self, timeout=60):
>> self.timeout = timeout
>> self.end = None
>>
>> def check_timeout(self):
>> if self.end and time.time() > self.end:
>> raise TimeoutException
>> else:
>> self.start()
>>
>> def start(self):
>> if not self.end:
>> self.end = time.time() + self.timeout
>>
>> class MainClass(object):
>> def __init__(self):
>> self.timer = TimeLimit()
>> algo(timer)
>>
>>
>> def algo(timer):
>> do_something()
>> external()
>> timer.check_timeout()
>> algo(timer)
>
> Which ultimately is no different to what was being done with the decorator.
>
> If external() can run a lot longer than the timeout, either way is not going
> to be much use. It is only going to be of much use if external() is always a
> short call and on subsequent calls you are simply calling it with different
> arguments.
>
>> I debugged code thoroughly and it works as expected. And as shown memory is
>> not drastically increased during execution, but later. Is that possible that
>> reasong in apache conf? Or I need to look for problem in code? I have no
>> clue where to search as everything works. Is it possible to see memory usage
>> somehow on localhost (I'm under Windows)?
>
> It is unlikely to be the Apache configuration. It is more likely that after
> finishing the execution of the algorithm, the algorithm isn't clearing out
> any working data structures it creates when it is doing calculations and they
> are persisting in memory. Subsequent calls simply add more memory each time
> and so it grows in that way.
>
> There are various reasons that Python objects cannot be reclaimed and will
> keep using memory.
>
> There aren't really great tools for tracking that sort of stuff down.
>
> Graham
>
>> On Sunday, February 15, 2015 at 12:31:36 PM UTC+2, Graham Dumpleton wrote:
>>
>> On 15/02/2015, at 9:26 PM, Paul Royik <[email protected]> wrote:
>>
>>> Yes. External library could run long (although I reduced this possibility).
>>
>> How have you reduced possibility?
>>
>>> So, what should I do with locking threads? Is it resolvable?
>>
>> No idea. It isn't my code. I can only highlight what looks suspicious in the
>> log output.
>>
>>> I'm just now playing with code on stackoverflow. People suggested to create
>>> class instead of decorator and pass its instance in all functions. I did as
>>> they proposed and it worked.
>>
>> Since you don't provide the code for that, I can't comment.
>>
>>> However, with the flow of time I see increase in memory.
>>> So, now nobody access site.
>>>
>>> I ran command ps -u simamura -o pid,rss,command | awk '{print $0}{sum+=$2}
>>> END {print "Total", sum/1024, "MB"}' and see 200 MB.
>>> Then I invoke integral calculator, while it works above command shows 200
>>> MB.
>>> Calculator finished, I check memory - still 200 MB. In 10-15 minutes
>>> (remember nobody access the site) I see 240 MB.
>>>
>>> In result, memory increased from 130 MB (memory immediately after server
>>> restart) to 435 MB in 10 hours.
>>>
>>> How can I resolve this issue? I feel that in another 12 hours webfaction
>>> will kill the process, because I memomy will be over limit.
>>
>> Have you tried adding print() debug statements to your code to track through
>> what is being called and to determine whether your code is doing what is
>> expected?
>>
>> Graham
>>
>>> On Sunday, February 15, 2015 at 12:12:39 PM UTC+2, Graham Dumpleton wrote:
>>>
>>> On 15/02/2015, at 9:03 PM, Paul Royik <[email protected]> wrote:
>>>
>>>> How do you know that it runs too long?
>>>
>>> In explained the log output previously and what was happening.
>>>
>>> If you go back far enough in the logs even before what you provided you
>>> will see some messages like:
>>>
>>> [Fri Feb 06 12:39:06.254056 2015] [wsgi:info] [pid 9731:tid
>>> 140178879313664] mod_wsgi (pid=9731): Daemon process request time limit
>>> exceeded, stopping process 'localhost:20241'.
>>> [Fri Feb 06 12:39:06.254108 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): Shutdown requested 'localhost:20241'.
>>> [Fri Feb 06 12:39:06.286339 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): Dumping stack trace for active Python
>>> threads.
>>> [Fri Feb 06 12:39:06.286359 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): Thread 140178316318464 executing file
>>> "/usr/local/lib/python2.7/threading.py", line 147, in acquire
>>> [Fri Feb 06 12:39:06.286363 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/compatibility.py", line 851, in
>>> wrapper,
>>> [Fri Feb 06 12:39:06.286366 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/cache.py", line 89, in wrapper,
>>> [Fri Feb 06 12:39:06.286369 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/operations.py", line 127, in
>>> _matches_commutative,
>>> [Fri Feb 06 12:39:06.286372 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 808, in matches,
>>> [Fri Feb 06 12:39:06.286375 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/add.py", line 358, in
>>> _matches_simple,
>>> [Fri Feb 06 12:39:06.286378 2015] [wsgi:info] [pid 9731:tid
>>> 140179136616192] mod_wsgi (pid=9731): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/operations.py", line 127, in
>>> _matches_commutative,
>>>
>>> So it says that request time limit was exceeded.
>>>
>>> This is followed by stack dumps for what all the request threads were doing.
>>>
>>> In the part you did show, they both are stuck in:
>>>
>>> [Sat Feb 14 19:58:00.261118 2015] [wsgi:info] [pid 12528:tid
>>> 139862016030464] mod_wsgi (pid=12528): Thread 139861715523328 executing
>>> file "/usr/local/lib/python2.7/threading.py", line 147, in acquire
>>> [Sat Feb 14 19:58:00.261120 2015] [wsgi:info] [pid 12528:tid
>>> 139862016030464] mod_wsgi (pid=12528): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/compatibility.py", line 851, in
>>> wrapper,
>>> [Sat Feb 14 19:58:00.261122 2015] [wsgi:info] [pid 12528:tid
>>> 139862016030464] mod_wsgi (pid=12528): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/cache.py", line 89, in wrapper,
>>> [Sat Feb 14 19:58:00.261125 2015] [wsgi:info] [pid 12528:tid
>>> 139862016030464] mod_wsgi (pid=12528): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 373, in _gather,
>>> [Sat Feb 14 19:58:00.261127 2015] [wsgi:info] [pid 12528:tid
>>> 139862016030464] mod_wsgi (pid=12528): called from file
>>> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 98, in flatten,
>>>
>>> That is, they are both waiting on a thread lock.
>>>
>>> This suggests to me that the code isn't possibly dealing with locking
>>> properly, or a lock is held for a very long time while in some certain
>>> section and blocking other threads from running and so why they are delayed
>>> and take so long.
>>>
>>>> No more than 300 seconds as I wrote in conf file (socket-timeout and
>>>> request-timeout).
>>>>
>>>> I can not control it thoroughly.
>>>> def algo():
>>>> do_something()
>>>> invoke_external_library() # this could run slighlty longer, so timeout
>>>> is not precise
>>>> check_timeout()
>>>> algo()
>>>
>>> So the thing that takes the unpredictable amount of time and you need to
>>> interrupt is not even your code?
>>>
>>> Graham
>>>
>>>> On Sunday, February 15, 2015 at 11:36:05 AM UTC+2, Graham Dumpleton wrote:
>>>> So we are back the fact that your requests are still running too long and
>>>> however you have set up the timeout check isn't working.
>>>>
>>>> Have you added print() debug statements in your code to validate that the
>>>> die on timeout check is even being run regularly?
>>>>
>>>> To be able to distinguish each request thread, you can include the output
>>>> of threading.currentThread() on the print statements.
>>>>
>>>> Graham
>>>>
>>>> On 15/02/2015, at 8:19 PM, Paul Royik <[email protected]> wrote:
>>>>
>>>>> [Sat Feb 14 19:58:00.261309 2015] [wsgi:info] [pid 12528:tid
>>>>> 139862016030464] mod_wsgi (pid=12528): called from file
>>>>> "/home/simamura/lib/python2.7/mod_wsgi/server/__init__.py", line 1137, in
>>>>> handle_request.
>>>>> [Sat Feb 14 19:58:05.245850 2015] [wsgi:info] [pid 12528:tid
>>>>> 139861541500672] mod_wsgi (pid=12528): Aborting process 'localhost:20241'.
>>>>> [Sat Feb 14 19:58:05.245889 2015] [wsgi:info] [pid 12528:tid
>>>>> 139861541500672] mod_wsgi (pid=12528): Exiting process 'localhost:20241'.
>>>>> [Sun Feb 15 01:58:05.352861 2015] [wsgi:error] [pid 11272:tid
>>>>> 139862014424832] [client 127.0.0.1:48193] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.352890 2015] [wsgi:error] [pid 11272:tid
>>>>> 139862015223552] [client 127.0.0.1:48563] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.352958 2015] [wsgi:error] [pid 11272:tid
>>>>> 139861948524288] [client 127.0.0.1:48240] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.352992 2015] [wsgi:error] [pid 11272:tid
>>>>> 139862015489792] [client 127.0.0.1:48440] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.353008 2015] [wsgi:error] [pid 11666:tid
>>>>> 139862014957312] [client 127.0.0.1:48329] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.353015 2015] [wsgi:error] [pid 11272:tid
>>>>> 139862014691072] [client 127.0.0.1:48620] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.353081 2015] [wsgi:error] [pid 11272:tid
>>>>> 139861949323008] [client 127.0.0.1:48778] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.353081 2015] [wsgi:error] [pid 11666:tid
>>>>> 139861948790528] [client 127.0.0.1:48735] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.353096 2015] [wsgi:error] [pid 11666:tid
>>>>> 139862015223552] [client 127.0.0.1:48792] Truncated or oversized response
>>>>> headers received from daemon process 'localhost:20241':
>>>>> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
>>>>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
>>>>> [Sun Feb 15 01:58:05.972993 2015] [wsgi:info] [pid 24792:tid
>>>>> 139862016030464] mod_wsgi (pid=12528): Process 'localhost:20241' has
>>>>> died, deregister and restart it.
>>>>> [Sun Feb 15 01:58:06.081051 2015] [wsgi:info] [pid 24792:tid
>>>>> 139862016030464] mod_wsgi (pid=12528): Process 'localhost:20241' has been
>>>>> deregistered and will no longer be monitored.
>>>>> [Sun Feb 15 01:58:06.081333 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028): Starting process 'localhost:20241'
>>>>> with threads=10.
>>>>> [Sun Feb 15 01:58:06.163910 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028): Python home /usr/local.
>>>>> [Sun Feb 15 01:58:06.163937 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028): Initializing Python.
>>>>> [Sun Feb 15 01:58:06.534740 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028): Attach interpreter ''.
>>>>> [Sun Feb 15 01:58:06.549849 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028): Imported 'mod_wsgi'.
>>>>> [Sun Feb 15 01:58:06.549942 2015] [wsgi:info] [pid 23028:tid
>>>>> 139862016030464] mod_wsgi (pid=23028, process='localhost:20241',
>>>>> application=''): Loading WSGI script
>>>>> '/home/simamura/webapps/django_math/express/handler.wsgi'.
>>>>> [Sun Feb 15 02:06:36.476939 2015] [core:info] [pid 11272:tid
>>>>> 139861948524288] [client 127.0.0.1:56156] AH00128: File does not exist:
>>>>> /home/simamura/webapps/math_root/browserconfig.xml
>>>>>
>>>>>
>>>>> And that's all.
>>>>>
>>>>> On Sunday, February 15, 2015 at 10:11:25 AM UTC+2, Graham Dumpleton wrote:
>>>>> Provide me with the logging from a period before the restart until a
>>>>> period after.
>>>>>
>>>>> The log can be very informative if you know what to look for.
>>>>>
>>>>> Graham
>>>>>
>>>>> On 15/02/2015, at 6:43 PM, Paul Royik <[email protected]> wrote:
>>>>>
>>>>>> I actually have custom error 500 page. This means, that when Django
>>>>>> throws error, custom page should be shown.
>>>>>> But it is not the case. Standard white-background apache page is shown.
>>>>>> So, I think that problem is not in Django exception. Otherwise Django
>>>>>> would show custom page, which is not the case.
>>>>>> In logs I saw only that server was restarted.
>>>>>> Can I somehow tune logging, so that real error is written?
>>>>>>
>>>>>> On Sunday, February 15, 2015 at 2:44:50 AM UTC+2, Graham Dumpleton wrote:
>>>>>>
>>>>>> On 15/02/2015, at 10:53 AM, Paul Royik <[email protected]> wrote:
>>>>>>
>>>>>> > I already tried this approach.
>>>>>> >
>>>>>> > It seems good.
>>>>>> > But when I tested it on server, by simultaneously executing page on
>>>>>> > two different computers, it gave me 500 error.
>>>>>> >
>>>>>> > How this can be explained? Again something with apache? Logs didn't
>>>>>> > show anything. But I noticed again serious memory usage. And this
>>>>>> > happens only when I use code for limiting time.
>>>>>>
>>>>>>
>>>>>> Some tips for you about ensuring you get the best help.
>>>>>>
>>>>>> Stop discarding the message content for the message you are replying to.
>>>>>> It can be frustrating to have to keep going back to old messages to see
>>>>>> what was said originally and what you may be following up to, especially
>>>>>> when on a phone. When asking questions on mailing list, let the person
>>>>>> you are asking questions of decide what can be discarded from the
>>>>>> message chain as they will know better what should be kept in the
>>>>>> message to provide easy access to important information for context.
>>>>>>
>>>>>> When you reply to say you are have a new error, provide the actual code
>>>>>> you were using exactly at the time you had the error. Saves the person
>>>>>> who is helping you having to go back and ask for it if it is unclear
>>>>>> what code you were using.
>>>>>>
>>>>>> Your messages here and on StackOverflow show that you play around with
>>>>>> the example code I am giving you and I have reduced confidence you were
>>>>>> running with the code I suggested at the time of the problem you are now
>>>>>> relating to.
>>>>>>
>>>>>> If you are getting 500 errors and nothing is being logged, it is because
>>>>>> Django is capturing the exception and converting it to a generic 500
>>>>>> error response page. Configure Django to send you emails with the
>>>>>> details of the exceptions. If you are on a local system, then set
>>>>>> DEBUG=True in the Django settings so the details of the error are shown
>>>>>> in response that goes back to the browser.
>>>>>>
>>>>>> As to the code, I did test it this time, including under mod_wsgi with
>>>>>> 20 request threads and hitting it with concurrent and successive
>>>>>> requests with a benchmarking tool. I saw no issues and it appeared to
>>>>>> behave as I would expect.
>>>>>>
>>>>>> So in a test2.py file I had:
>>>>>>
>>>>>> import time
>>>>>> import functools
>>>>>> import threading
>>>>>>
>>>>>> def time_limit(seconds):
>>>>>> def decorator(func):
>>>>>> func.info = threading.local()
>>>>>> def check_timeout():
>>>>>> if time.time() > func.info.end_time:
>>>>>> raise RuntimeError('timeout')
>>>>>>
>>>>>> func.check_timeout = check_timeout
>>>>>>
>>>>>> @functools.wraps(func)
>>>>>> def wrapper(*args, **kwargs):
>>>>>> print 'hasattr counter', hasattr(func.info, 'counter')
>>>>>> if not hasattr(func.info, 'counter'):
>>>>>> print 'init counter to 0'
>>>>>> func.info.counter = 0
>>>>>> if func.info.counter == 0:
>>>>>> func.info.end_time = time.time() + seconds
>>>>>> print 'counter', func.info.counter
>>>>>> func.info.counter += 1
>>>>>> try:
>>>>>> return func(*args, **kwargs)
>>>>>> finally:
>>>>>> func.info.counter -= 1
>>>>>>
>>>>>> return wrapper
>>>>>>
>>>>>> return decorator
>>>>>>
>>>>>> @time_limit(5)
>>>>>> def algorithm(limit, nest=0):
>>>>>> algorithm.check_timeout()
>>>>>> print 'sleep'
>>>>>> time.sleep(1.0)
>>>>>> if nest == limit:
>>>>>> print 'return'
>>>>>> return
>>>>>> algorithm(limit, nest+1)
>>>>>>
>>>>>> In serial2.py I had a straight serialised test:
>>>>>>
>>>>>> from test2 import algorithm
>>>>>>
>>>>>> try:
>>>>>> algorithm(3)
>>>>>> except RuntimeError:
>>>>>> print 'timeout'
>>>>>>
>>>>>> try:
>>>>>> algorithm(10)
>>>>>> except RuntimeError:
>>>>>> print 'timeout'
>>>>>>
>>>>>> try:
>>>>>> algorithm(10)
>>>>>> except RuntimeError:
>>>>>> print 'timeout'
>>>>>>
>>>>>> And then in hello2.wsgi I had it used by a WSGI application.
>>>>>>
>>>>>> from test2 import algorithm
>>>>>>
>>>>>> def application(environ, start_response):
>>>>>> status = '200 OK'
>>>>>> output = b'Hello World!'
>>>>>>
>>>>>> algorithm(1)
>>>>>>
>>>>>> response_headers = [('Content-type', 'text/plain'),
>>>>>> ('Content-Length', str(len(output)))]
>>>>>> start_response(status, response_headers)
>>>>>>
>>>>>> return [output]
>>>>>>
>>>>>> In the latter case I ran mod_wsgi-express against it as:
>>>>>>
>>>>>> mod_wsgi-express start-server hello2.wsgi --port 8002 --threads=20
>>>>>>
>>>>>> and then hit it with ab as:
>>>>>>
>>>>>> ab -n 100 -c 15 http://localhost:8002/
>>>>>>
>>>>>> In this case it wasn't designed to timeout anything, but that should not
>>>>>> be a concern as the counter initialisation is still being tested.
>>>>>>
>>>>>> Do note that if you did cut and paste that last code, I did change the
>>>>>> exception type.
>>>>>>
>>>>>> Anyway, the best thing to do is setup Django so that it provides the
>>>>>> details of the exception it captured but then effectively discarded
>>>>>> because it converted it to a 500 page.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/modwsgi.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/modwsgi.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
