Re: [galaxy-dev] How to rotate Galaxy log file
On Oct 1, 2012, at 2:57 PM, Lance Parsons wrote:
> I also use a logrotate script: '/etc/logrotate.d/galaxy'
>
> /path/to/galaxy/install/*.log {
> weekly
> rotate 52
> copytruncate
> }
>
>
> The 'compress' options seems like it might be nice, and I'm not sure of the
> effect of size 1. I don't think the 'create 640 galaxy-user galaxy-group' has
> an effect when the copytruncate option is used. For the same reason, I don't
> think the missingok option is needed either.
>
> This might be a nice thing to add to the wiki at:
> "http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server";.
>
> Lance
That's right. I agree that 'size 1', 'missingok' and 'create galaxy-user
galaxy-group' options are practically not needed and can be excluded. As
mentioned earlier, it's the copytruncate option that should be useful for
rotating galaxy log files.
Also, I would like to mention that you can run logrotate program as a non-root
user (galaxy-user) if your galaxy application is served from a root squash
enabled NFS mount. In this case, the logrotate configuration can be put in a
non-default location (/etc/logrotate.d) and can be called through non-root
user's cron jobs.
--
Shantanu
>> On Sep 25, 2012, at 6:57 AM, Lukasz Lacinski wrote:
>>
>>> On 9/24/12 12:40 PM, Nate Coraor wrote:
On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote:
> repost to galaxy-dev
>
> On 9/7/12 6:39 PM, Lukasz Lacinski wrote:
>> Dear All,
>>
>> I use an init script that comes with Galaxy in the contrib/ subdirectory
>> to start Galaxy. The log file
>>
>> --log-file /home/galaxy/galaxy.log
>>
>> specified in the script grows really quickly. How to logrotate the file?
Hi Lukasz,
I'd suggest using whatever log rotation utility is provided by your OS.
You'll need to restart the Galaxy process to begin writing to the new log
once the old one has been rotated.
>>> Hi Nate,
>>>
>>> When Galaxy is started again, it fails because it cannot bind a socket to
>>> port 8080, that is already bound by child Galaxy processes orphaned by the
>>> former Galaxy process.
>>>
>>> When Galaxy forks to run tools, a child process does not close open
>>> files/sockets that the child process does not need.
>>>
>>> Thanks,
>>> Lukasz
>>
>>
>> I am not sure about multiple galaxy process install, but I think a logrotate
>> configuration with copytruncate option should work fine. For example,
>>
>> {{{
>> /apps/galaxy/galaxy-latest/paster.log {
>> missingok
>> rotate 10
>> copytruncate
>> # logrotate script should be called by a cron job and hence daily/weekly
>> settings are not added to conf file.
>> # 'size 1' setting ensures that log file will be rotated whenevr it is
>> called by the cron unless size< 1 byte
>> size 1
>> compress
>> create 640 galaxy-user galaxy-group
>> }}}
>>
>> --
>> Shantanu
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>> http://lists.bx.psu.edu/
>
> --
> Lance Parsons - Scientific Programmer
> 134 Carl C. Icahn Laboratory
> Lewis-Sigler Institute for Integrative Genomics
> Princeton University
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
I also use a logrotate script: '/etc/logrotate.d/galaxy'
/path/to/galaxy/install/*.log {
weekly
rotate 52
copytruncate
}
The 'compress' options seems like it might be nice, and I'm not sure of
the effect of size 1. I don't think the 'create 640 galaxy-user
galaxy-group' has an effect when the copytruncate option is used. For
the same reason, I don't think the missingok option is needed either.
This might be a nice thing to add to the wiki at:
"http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server";.
Lance
On Sep 25, 2012, at 6:57 AM, Lukasz Lacinski wrote:
On 9/24/12 12:40 PM, Nate Coraor wrote:
On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote:
repost to galaxy-dev
On 9/7/12 6:39 PM, Lukasz Lacinski wrote:
Dear All,
I use an init script that comes with Galaxy in the contrib/ subdirectory
to start Galaxy. The log file
--log-file /home/galaxy/galaxy.log
specified in the script grows really quickly. How to logrotate the file?
Hi Lukasz,
I'd suggest using whatever log rotation utility is provided by your OS. You'll
need to restart the Galaxy process to begin writing to the new log once the old
one has been rotated.
Hi Nate,
When Galaxy is started again, it fails because it cannot bind a socket to port
8080, that is already bound by child Galaxy processes orphaned by the former
Galaxy process.
When Galaxy forks to run tools, a child process does not close open
files/sockets that the child process does not need.
Thanks,
Lukasz
I am not sure about multiple galaxy process install, but I think a logrotate
configuration with copytruncate option should work fine. For example,
{{{
/apps/galaxy/galaxy-latest/paster.log {
missingok
rotate 10
copytruncate
# logrotate script should be called by a cron job and hence daily/weekly
settings are not added to conf file.
# 'size 1' setting ensures that log file will be rotated whenevr it is called by
the cron unless size< 1 byte
size 1
compress
create 640 galaxy-user galaxy-group
}}}
--
Shantanu
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
--
Lance Parsons - Scientific Programmer
134 Carl C. Icahn Laboratory
Lewis-Sigler Institute for Integrative Genomics
Princeton University
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
On Sep 25, 2012, at 12:35 PM, Lukasz Lacinski wrote: > I use start-stop-daemon that sends SIGTERM (which is a default signal sent to > a service that is being asked to stop) to Galaxy to let Galaxy store all > information in a database, close files, etc., and terminate itself. When > Galaxy process is terminated, both shell and python scripts (Galaxy tools) > are still running. > > Effects of using SIGKILL/SIGSTOP are not predictable. I do not want to risk > that Galaxy does not insert entries to the database that should be inserted > to start Galaxy correctly again, or does not flush buffers to write all data > to files and close the files correctly. Hi Lukasz, Galaxy does not terminate jobs running in the local runner when the Galaxy server stops. If you need to be able to restart the server without losing running jobs, you'll need to set up a cluster scheduler, even if it just submits jobs to the localhost. --nate > > Thanks, > Lukasz > > On 9/25/12 10:40 AM, Scott McManus wrote: >> Lukasz- >> >> How are you stopping the process? It's possible that the python >> task for galaxy is still running. I would recommend trying to >> send a "kill" signal to that process. Something like >> "kill -9 PID", where PID is the process id for galaxy, should >> work. >> >> -Scott >> >> - Original Message - >>> On 9/24/12 12:40 PM, Nate Coraor wrote: On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: > repost to galaxy-dev > > On 9/7/12 6:39 PM, Lukasz Lacinski wrote: >> Dear All, >> >> I use an init script that comes with Galaxy in the contrib/ >> subdirectory >> to start Galaxy. The log file >> >> --log-file /home/galaxy/galaxy.log >> >> specified in the script grows really quickly. How to logrotate >> the file? Hi Lukasz, I'd suggest using whatever log rotation utility is provided by your OS. You'll need to restart the Galaxy process to begin writing to the new log once the old one has been rotated. >>> Hi Nate, >>> >>> When Galaxy is started again, it fails because it cannot bind a >>> socket to port 8080, that is already bound by child Galaxy processes >>> orphaned by the former Galaxy process. >>> >>> When Galaxy forks to run tools, a child process does not close open >>> files/sockets that the child process does not need. >>> >>> Thanks, >>> Lukasz >>> --nate >> Thanks, >> Lukasz >> ___ >> The Galaxy User list should be used for the discussion of >> Galaxy analysis and other features on the public server >> at usegalaxy.org. Please keep all replies on the list by >> using "reply all" in your mail client. For discussion of >> local Galaxy instances and the Galaxy source code, please >> use the Galaxy Development list: >> >> http://lists.bx.psu.edu/listinfo/galaxy-dev >> >> To manage your subscriptions to this and other Galaxy lists, >> please use the interface at: >> >> http://lists.bx.psu.edu/ > -- > Jennifer Jackson > http://galaxyproject.org > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >>> > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
On Sep 25, 2012, at 6:57 AM, Lukasz Lacinski wrote:
> On 9/24/12 12:40 PM, Nate Coraor wrote:
>> On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote:
>>
>>> repost to galaxy-dev
>>>
>>> On 9/7/12 6:39 PM, Lukasz Lacinski wrote:
Dear All,
I use an init script that comes with Galaxy in the contrib/ subdirectory
to start Galaxy. The log file
--log-file /home/galaxy/galaxy.log
specified in the script grows really quickly. How to logrotate the file?
>> Hi Lukasz,
>>
>> I'd suggest using whatever log rotation utility is provided by your OS.
>> You'll need to restart the Galaxy process to begin writing to the new log
>> once the old one has been rotated.
>
> Hi Nate,
>
> When Galaxy is started again, it fails because it cannot bind a socket to
> port 8080, that is already bound by child Galaxy processes orphaned by the
> former Galaxy process.
>
> When Galaxy forks to run tools, a child process does not close open
> files/sockets that the child process does not need.
>
> Thanks,
> Lukasz
I am not sure about multiple galaxy process install, but I think a logrotate
configuration with copytruncate option should work fine. For example,
{{{
/apps/galaxy/galaxy-latest/paster.log {
missingok
rotate 10
copytruncate
# logrotate script should be called by a cron job and hence daily/weekly
settings are not added to conf file.
# 'size 1' setting ensures that log file will be rotated whenevr it is called
by the cron unless size < 1 byte
size 1
compress
create 640 galaxy-user galaxy-group
}}}
--
Shantanu
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
I use start-stop-daemon that sends SIGTERM (which is a default signal sent to a service that is being asked to stop) to Galaxy to let Galaxy store all information in a database, close files, etc., and terminate itself. When Galaxy process is terminated, both shell and python scripts (Galaxy tools) are still running. Effects of using SIGKILL/SIGSTOP are not predictable. I do not want to risk that Galaxy does not insert entries to the database that should be inserted to start Galaxy correctly again, or does not flush buffers to write all data to files and close the files correctly. Thanks, Lukasz On 9/25/12 10:40 AM, Scott McManus wrote: Lukasz- How are you stopping the process? It's possible that the python task for galaxy is still running. I would recommend trying to send a "kill" signal to that process. Something like "kill -9 PID", where PID is the process id for galaxy, should work. -Scott - Original Message - On 9/24/12 12:40 PM, Nate Coraor wrote: On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: repost to galaxy-dev On 9/7/12 6:39 PM, Lukasz Lacinski wrote: Dear All, I use an init script that comes with Galaxy in the contrib/ subdirectory to start Galaxy. The log file --log-file /home/galaxy/galaxy.log specified in the script grows really quickly. How to logrotate the file? Hi Lukasz, I'd suggest using whatever log rotation utility is provided by your OS. You'll need to restart the Galaxy process to begin writing to the new log once the old one has been rotated. Hi Nate, When Galaxy is started again, it fails because it cannot bind a socket to port 8080, that is already bound by child Galaxy processes orphaned by the former Galaxy process. When Galaxy forks to run tools, a child process does not close open files/sockets that the child process does not need. Thanks, Lukasz --nate Thanks, Lukasz ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
Galaxy should probably be shutdown with --stop-daemon (depending on how you are running it). In the future we can look at making the logging handle the HUP signal. -- jt On Tue, Sep 25, 2012 at 11:40 AM, Scott McManus wrote: > > Lukasz- > > How are you stopping the process? It's possible that the python > task for galaxy is still running. I would recommend trying to > send a "kill" signal to that process. Something like > "kill -9 PID", where PID is the process id for galaxy, should > work. > > -Scott > > - Original Message - >> On 9/24/12 12:40 PM, Nate Coraor wrote: >> > On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: >> > >> >> repost to galaxy-dev >> >> >> >> On 9/7/12 6:39 PM, Lukasz Lacinski wrote: >> >>> Dear All, >> >>> >> >>> I use an init script that comes with Galaxy in the contrib/ >> >>> subdirectory >> >>> to start Galaxy. The log file >> >>> >> >>> --log-file /home/galaxy/galaxy.log >> >>> >> >>> specified in the script grows really quickly. How to logrotate >> >>> the file? >> > Hi Lukasz, >> > >> > I'd suggest using whatever log rotation utility is provided by your >> > OS. You'll need to restart the Galaxy process to begin writing to >> > the new log once the old one has been rotated. >> >> Hi Nate, >> >> When Galaxy is started again, it fails because it cannot bind a >> socket to port 8080, that is already bound by child Galaxy processes >> orphaned by the former Galaxy process. >> >> When Galaxy forks to run tools, a child process does not close open >> files/sockets that the child process does not need. >> >> Thanks, >> Lukasz >> >> > >> > --nate >> > >> >>> Thanks, >> >>> Lukasz >> >>> ___ >> >>> The Galaxy User list should be used for the discussion of >> >>> Galaxy analysis and other features on the public server >> >>> at usegalaxy.org. Please keep all replies on the list by >> >>> using "reply all" in your mail client. For discussion of >> >>> local Galaxy instances and the Galaxy source code, please >> >>> use the Galaxy Development list: >> >>> >> >>> http://lists.bx.psu.edu/listinfo/galaxy-dev >> >>> >> >>> To manage your subscriptions to this and other Galaxy lists, >> >>> please use the interface at: >> >>> >> >>> http://lists.bx.psu.edu/ >> >> -- >> >> Jennifer Jackson >> >> http://galaxyproject.org >> >> ___ >> >> Please keep all replies on the list by using "reply all" >> >> in your mail client. To manage your subscriptions to this >> >> and other Galaxy lists, please use the interface at: >> >> >> >> http://lists.bx.psu.edu/ >> > >> > ___ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > >> >http://lists.bx.psu.edu/ >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ >> > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
Lukasz- How are you stopping the process? It's possible that the python task for galaxy is still running. I would recommend trying to send a "kill" signal to that process. Something like "kill -9 PID", where PID is the process id for galaxy, should work. -Scott - Original Message - > On 9/24/12 12:40 PM, Nate Coraor wrote: > > On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: > > > >> repost to galaxy-dev > >> > >> On 9/7/12 6:39 PM, Lukasz Lacinski wrote: > >>> Dear All, > >>> > >>> I use an init script that comes with Galaxy in the contrib/ > >>> subdirectory > >>> to start Galaxy. The log file > >>> > >>> --log-file /home/galaxy/galaxy.log > >>> > >>> specified in the script grows really quickly. How to logrotate > >>> the file? > > Hi Lukasz, > > > > I'd suggest using whatever log rotation utility is provided by your > > OS. You'll need to restart the Galaxy process to begin writing to > > the new log once the old one has been rotated. > > Hi Nate, > > When Galaxy is started again, it fails because it cannot bind a > socket to port 8080, that is already bound by child Galaxy processes > orphaned by the former Galaxy process. > > When Galaxy forks to run tools, a child process does not close open > files/sockets that the child process does not need. > > Thanks, > Lukasz > > > > > --nate > > > >>> Thanks, > >>> Lukasz > >>> ___ > >>> The Galaxy User list should be used for the discussion of > >>> Galaxy analysis and other features on the public server > >>> at usegalaxy.org. Please keep all replies on the list by > >>> using "reply all" in your mail client. For discussion of > >>> local Galaxy instances and the Galaxy source code, please > >>> use the Galaxy Development list: > >>> > >>> http://lists.bx.psu.edu/listinfo/galaxy-dev > >>> > >>> To manage your subscriptions to this and other Galaxy lists, > >>> please use the interface at: > >>> > >>> http://lists.bx.psu.edu/ > >> -- > >> Jennifer Jackson > >> http://galaxyproject.org > >> ___ > >> Please keep all replies on the list by using "reply all" > >> in your mail client. To manage your subscriptions to this > >> and other Galaxy lists, please use the interface at: > >> > >> http://lists.bx.psu.edu/ > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > >http://lists.bx.psu.edu/ > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
On 9/24/12 12:40 PM, Nate Coraor wrote: On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: repost to galaxy-dev On 9/7/12 6:39 PM, Lukasz Lacinski wrote: Dear All, I use an init script that comes with Galaxy in the contrib/ subdirectory to start Galaxy. The log file --log-file /home/galaxy/galaxy.log specified in the script grows really quickly. How to logrotate the file? Hi Lukasz, I'd suggest using whatever log rotation utility is provided by your OS. You'll need to restart the Galaxy process to begin writing to the new log once the old one has been rotated. Hi Nate, When Galaxy is started again, it fails because it cannot bind a socket to port 8080, that is already bound by child Galaxy processes orphaned by the former Galaxy process. When Galaxy forks to run tools, a child process does not close open files/sockets that the child process does not need. Thanks, Lukasz --nate Thanks, Lukasz ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to rotate Galaxy log file
On Sep 19, 2012, at 9:50 AM, Jennifer Jackson wrote: > repost to galaxy-dev > > On 9/7/12 6:39 PM, Lukasz Lacinski wrote: >> Dear All, >> >> I use an init script that comes with Galaxy in the contrib/ subdirectory >> to start Galaxy. The log file >> >> --log-file /home/galaxy/galaxy.log >> >> specified in the script grows really quickly. How to logrotate the file? Hi Lukasz, I'd suggest using whatever log rotation utility is provided by your OS. You'll need to restart the Galaxy process to begin writing to the new log once the old one has been rotated. --nate >> >> Thanks, >> Lukasz >> ___ >> The Galaxy User list should be used for the discussion of >> Galaxy analysis and other features on the public server >> at usegalaxy.org. Please keep all replies on the list by >> using "reply all" in your mail client. For discussion of >> local Galaxy instances and the Galaxy source code, please >> use the Galaxy Development list: >> >> http://lists.bx.psu.edu/listinfo/galaxy-dev >> >> To manage your subscriptions to this and other Galaxy lists, >> please use the interface at: >> >> http://lists.bx.psu.edu/ > > -- > Jennifer Jackson > http://galaxyproject.org > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
