Re: what if native systemd service is slower than old sysvinit script?
On 09/17/2011 01:09 AM, Adam Williamson wrote: On Fri, 2011-09-16 at 23:22 -0400, Steve Clark wrote: Oh, I must have misunderstood - Gene's Mailist comment: . Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. Looked like talk about session management to me. That was the comment I replied to and said 'we're not really talking about systemd any more'. =) Hi Adam, I guess I am confused by your comment based on what is stated at http://fedoraproject.org/wiki/Features/systemd Which explicitly states systemd System and Session Manager. So it appears to me that Session Management is the next thing systemd wants to do. In fact Lennart has stated that. -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 2011-09-15 at 17:06 +, Jóhann B. Guðmundsson wrote: On 09/15/2011 04:11 PM, Michal Schmidt wrote: On 09/15/2011 05:54 PM, Ralf Corsepius wrote: On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plotbootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg From the long delay before swap.target is reached it seems that your defined swap partition never comes up and systemd times out waiting on it. If you dont use any lvm, raid and encrypted devices, you can safely turn off all fedora-* services ( Anaconda (F16) finally offers desktop users an easy way to opt out from lvm without being partitioning experts ) #cd /lib/systemd/system #for i in fedora-*; do ln -s /dev/null /etc/systemd/system/$i;done We throw in udev-settle since it gets pulled in by the storage setup #ln -s /dev/null /etc/systemd/system/udev-settle.service If you want speed not eye candy you can disable plymouth #cd /lib/systemd/system #for i in plymouth-*; do ln -s /dev/null /etc/systemd/system/$i;done Then proceeding disabling all the service you dont use. #for i in service1 service2 service3 etc... ; do systemctl disable $i ;done Harald Hoyer made a nice guide on this topic: http://www.harald-hoyer.de/personal/blog/fedora-15-boot-optimization -- Vratislav Podzimek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 2011-09-15 at 14:32 -0400, Genes MailLists wrote: -- (i) Server. -- These run all the time - reboots are most often in maintenance window (or evenings / weekends for home servers) primarily if not soely for kernel updates. *** boot time pain more occasional fsck costs and not service startup Pain caused by O/S updates - rolling release model would be ideal for these. When I was dabbling with HA clustering (which was admittedly a long time ago), there were environments with cold standby nodes (cold as in off). If the hot node went down, the cluster management software switched these standby nodes on, which booted normally then. In that case, a small boot time would have been very appreciated. I'm not sure if such scenarios are used nowadays. Nils -- Nils Philippsen Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety. -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 05:17:43PM -0700, Adam Williamson wrote: True. As far as GNOME goes, though, whenever you suggest 'bulletproof session management', they say 'that's what suspend is for'... I'd like to see proper session management. However, the existing X protocol is terrible (a KDE'er talked about the horrors @ Desktop Summit), and session management itself is really difficult. -- Regards, Olav (a GNOMEr:) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/16/2011 05:01 AM, Olav Vitters wrote: On Thu, Sep 15, 2011 at 05:17:43PM -0700, Adam Williamson wrote: True. As far as GNOME goes, though, whenever you suggest 'bulletproof session management', they say 'that's what suspend is for'... I'd like to see proper session management. However, the existing X protocol is terrible (a KDE'er talked about the horrors @ Desktop Summit), and session management itself is really difficult. Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. :-) Let the Wayland, KDE, LXDE, Gnome etc daemons each deal with their own things. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/16/2011 05:05 AM, Nils Philippsen wrote: On Thu, 2011-09-15 at 14:32 -0400, Genes MailLists wrote: -- (i) Server. -- These run all the time - reboots are most often in maintenance window (or evenings / weekends for home servers) primarily if not soely for kernel updates. *** boot time pain more occasional fsck costs and not service startup Pain caused by O/S updates - rolling release model would be ideal for these. When I was dabbling with HA clustering (which was admittedly a long time ago), there were environments with cold standby nodes (cold as in off). If the hot node went down, the cluster management software switched these standby nodes on, which booted normally then. In that case, a small boot time would have been very appreciated. I'm not sure if such scenarios are used nowadays. Nils Tho for that use case, would not a sleep/awake, as Adam suggested, have been far superior to a cold boot in that case? It could even work to quiesce a system and restore to running state for those processes doing real work without requiring the apps to manage their own restarts ... I assume it was power consumption being lowered here. Fast boot is of course desirable - I don't think anyone is against it obviously, I am merely suggesting it importance is somewhat limited in scope .. I also forgot marketing .. as in .. Fedora boots faster than WingleBuntuDos/X running on same hardware in the MoronIX speed test :-) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Fri, 2011-09-16 at 08:48 -0400, Genes MailLists wrote: On 09/16/2011 05:01 AM, Olav Vitters wrote: On Thu, Sep 15, 2011 at 05:17:43PM -0700, Adam Williamson wrote: True. As far as GNOME goes, though, whenever you suggest 'bulletproof session management', they say 'that's what suspend is for'... I'd like to see proper session management. However, the existing X protocol is terrible (a KDE'er talked about the horrors @ Desktop Summit), and session management itself is really difficult. Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. We aren't really talking about systemd any more, in this branch of the thread. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/16/2011 08:08 PM, Adam Williamson wrote: On Fri, 2011-09-16 at 08:48 -0400, Genes MailLists wrote: On 09/16/2011 05:01 AM, Olav Vitters wrote: On Thu, Sep 15, 2011 at 05:17:43PM -0700, Adam Williamson wrote: True. As far as GNOME goes, though, whenever you suggest 'bulletproof session management', they say 'that's what suspend is for'... I'd like to see proper session management. However, the existing X protocol is terrible (a KDE'er talked about the horrors @ Desktop Summit), and session management itself is really difficult. Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. We aren't really talking about systemd any more, in this branch of the thread. Were not? From: http://fedoraproject.org/wiki/Features/systemd systemd System and Session Manager -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/17/2011 06:33 AM, Steve Clark wrote: Were not? From: http://fedoraproject.org/wiki/Features/systemd systemd System and Session Manager That page does answer your question. systemd can work as a session manager but it isn't part of Fedora yet and this particular discussion wasn't about systemd but difficulties in session management in general. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/16/2011 11:03 PM, Rahul Sundaram wrote: On 09/17/2011 06:33 AM, Steve Clark wrote: Were not? From: http://fedoraproject.org/wiki/Features/systemd systemd System and Session Manager That page does answer your question. systemd can work as a session manager but it isn't part of Fedora yet and this particular discussion wasn't about systemd but difficulties in session management in general. Rahul Oh, I must have misunderstood - Gene's Mailist comment: . Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. Looked like talk about session management to me. -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/17/2011 08:52 AM, Steve Clark wrote: On 09/16/2011 11:03 PM, Rahul Sundaram wrote: On 09/17/2011 06:33 AM, Steve Clark wrote: Were not? From: http://fedoraproject.org/wiki/Features/systemd systemd System and Session Manager That page does answer your question. systemd can work as a session manager but it isn't part of Fedora yet and this particular discussion wasn't about systemd but difficulties in session management in general. Rahul Oh, I must have misunderstood - Gene's Mailist comment: . Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. Looked like talk about session management to me. Yes, Gene's comment merely added to the confusion. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Fri, 2011-09-16 at 23:22 -0400, Steve Clark wrote: Oh, I must have misunderstood - Gene's Mailist comment: . Temptinh as it might be, just please keep session management away from the init daemon and let it do its one important job properly, robustly and well and not suffer the path to sure death of trying to be all things - just coz it can coz its PID 1,2, 3 etc. Looked like talk about session management to me. That was the comment I replied to and said 'we're not really talking about systemd any more'. =) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 7:25 AM, Ralf Corsepius rc040...@freenet.de wrote: On 09/14/2011 06:23 PM, drago01 wrote: On Wed, Sep 14, 2011 at 5:34 PM, Ralf Corsepiusrc040...@freenet.de wrote: My netbook boots up F14 in ca. 60 secs, while F15 boots up in 62 secs. I'd call this below measurement accuracy. What kind of disk is that? It's ca. 3 years old WD Scorpio Blue 160 GB ( WD1600BEVT) in a first generation Atom N270 (32bit only) based netbook w/ 2GB RAM. For a mechanical drive any gain from parallel startup would get killed by disk seeks. Sure, slow disks certainly are a factor contributing to slow bootup times. In general, there are other factors coming into play, such as parallel startup using more memory, parallelization not providing many advantages on systems with a small number of CPU cores, hard synchronisation points in the bootup process, poorly configured services, ... and finally ... bugs. The number of CPUs shouldn't really matter as it is mostly I/O bound. Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Do they both start the same services? Unless you tweaked your fedora installation where we start a bunch of stuff that pretty much nobody would use in a typical desktop system that is to be expected. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Tom Lane t...@redhat.com:
=?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes:
Ok, I made four series of tests:
- start/stop an old init script
- start/stop an old init script with dropping caches - should simulate
system booting
- start/stop service file
- start/stop service file with dropping caches
In each series of tests were repeated five times.
series 1 - start - 2.2+ sec
series 1 - stop - 1.2+ sec
series 2 - start - 2.4+ sec
series 2 - stop - 1.3+ sec
series 3 - start - 3.1+ sec
series 3 - stop - 1.1+ sec
series 4 - start - 4.2+ sec
series 4 - stop - 1.1+ sec
Results are reproducible.
I tried to replicate these results on my own F15 laptop, and could not
--- the service file method doesn't really seem significantly faster
than the init script, but it's not slower either.
Here's what I did:
1. Install the postgresql-9.1.0 RPMs (rebuilt for F15 of course)
and do postgresql-setup initdb.
I did the same thing, just with PGSQL 9.0
2. Set log_line_prefix = '%m %p ' and log_connections = on in
postgresql.conf, so that log messages will be timestamped. Also set
timezone and log_timezone to desired values (I use 'US/Eastern');
if you don't do that, the server startup time is increased significantly
while Postgres tries to figure out the system timezone setting.
I also changed these parameters
3. As root, do
date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns
with dropped cache
date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns
2011-09-15 08:44:40.348239703+02:00
2011-09-15 08:44:44.651134587+02:00
without dropped cache
date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns
2011-09-15 08:45:38.388010217+02:00
2011-09-15 08:45:42.633229665+02:00
4. Note the time from the first date output to the database system is
ready to accept connections message getting logged (in the appropriate
file under /var/lib/pgsql/data/pg_log, if you haven't changed any other
logging settings). Stop and restart a few times to get a good average.
5. Install the F15 version of postgresql.init (be sure to adjust the
PGVERSION setting near the top of the file to be 9.1.0).
6. Start it that way a few times, note the same elapsed time.
I'm seeing numbers consistently around 0.3 second for the unit file,
and a bit less consistent but maybe 0.35 - 0.5 second for the script.
Ok, this is good news, because it seems that this behavior only occurs
on my system. Also Jef Spaleta can not repeat this behavior of
service.
Note that the time for the service to report itself ready after the
database has started is likely to be quite a bit different between the
two methods, but that is not systemd's fault. The init script just
launches the postmaster, sleeps for 2 seconds, and then reports OK
if it sees the postmaster has created a PID file. The unit file uses
pg_ctl, which actually waits till it can make a successful connection
to the postmaster, sleeping 1 second between tries. So it's a bit of a
crapshoot which will be longer, though if you are starting from a clean
database shutdown I'd expect pg_ctl to usually come back after the first
sleep.
So I'm not sure what's happening on Michal's machine, but from here I
don't see anything egregiously wrong with systemd's performance on
this test.
I tried to figureout what is happening with strace. I don't see any
significant difference between start with an old init script and
service file. I wonder just what's going on with Resource temporarily
unavailable - it seems to me that if I use service, it wait's a bit
longer (but I'm not able to measure time, so it can only be an
impression).
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS, {B38400 opost isig icanon echo ...}) = 0
clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f1ef78cea90) = 3934
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{l\1\0\1$\0\0\0\1\0\0\0\240\0\0\0\1\1o\0\31\0\0\0/org/fre...,
176}, {\22\0\0\0postgresql.service\0\0\7\0\0\0repl..., 36}],
msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 212
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL,
msg_iov(1)=[{l\2\1\1'\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0...,
2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC},
MSG_CMSG_CLOEXEC) = 71
recvmsg(3, 0x7fffe59f6a10, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource
temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{l\1\0\1\27\0\0\0\2\0\0\0\227\0\0\0\1\1o\0\31\0\0\0/org/fre...,
168}, {\22\0\0\0postgresql.service\0, 23}], msg_controllen=0,
msg_flags=0}, MSG_NOSIGNAL) = 191
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL,
msg_iov(1)=[{l\4\1\1+\0\0\0\2\0\0\0\203\0\0\0\1\1o\0\31\0\0\0/org/fre...,
2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC},
MSG_CMSG_CLOEXEC) = 283
recvmsg(3,
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Jef Spaleta jspal...@gmail.com: 2011/9/14 Michał Piotrowski mkkp...@gmail.com Exactly. F15, PostgreSQL 9.0 and just service file from PostgreSQL 9.1. Root filesystem and database are on SSD and Ext4. Okay... brace yourself. I just ran this test on my non-SSD ext4 based F15 system and I get the opposite result on start...repeatably. On my system the systemd based start up in the admittedly simple service postgresql start testing is consistently faster by about a second. And just as interesting... my sysinit times are slower than yours, that's a real head scratcher for me. If my db test was less overall work to init for some reason I'd expect my numbers to be faster than yourse in both cases. I have no off the top of my head theory that could explain that except that your low level disk i/o is significantly different than mine. Both variants of service postgresql stop is within a 1/10 of a second in my testing I can give you the screen log captures for my tests if you desire to review my actions but here here is the summary for my simple timing test: And admittedly I'm using a simply initiated postgres db. It could be that your real world database initialization workload is more intensive than my test and the disk i/o is now a limiting factor in a different way. And of course my numbers do not discount what your seeing as my test is as anecdotal as your is. Your numbers may still be indicative of a more nuanced problem that can be resolved and its good to have your numbers as a starting point for a discussion around understanding optimization issues (like disk i/o). We just can't jump to conclusions about why you are seeing what your are seeing. I hope my numbers serve as a cautionary reminder to those others on this list that benchmarking disk io intensive tasks can be very complex and very system dependent. Certain other people in this discussion are being overly bombastic and seem to have forgotten this fact. I hope for all our sakes they can find a way to ratchet down the hyperbole and look at these sort of issues with a more clinical approach. Okay so here's my summary of the quick tests. Please if you have an updated methodology for me to test, let me know. I'm willing to reuse a specially crafted pgsql database if you feel that will help you. Though I'd think we wan't to do that sort of deep dive comparison off list until we are ready to publish some summaries after we both feel comfortable with the test runs. It seems to me that the database settings or database content has nothing to do with my test results. Yesterday I removed all databases (it's only development system) and I did the whole setup again. Results are the same. So I came to the conclusion that it must be relevant to some other factor. Thanks for sending your test results. Later I'll try to work out this problem. -jef == SysVinit: service postgresql status postgresql.service - LSB: start and stop PostgreSQL server Loaded: loaded (/etc/rc.d/init.d/postgresql) Active: inactive (dead) CGroup: name=systemd:/system/postgresql.service time service postgresql start real 0m2.329s user 0m0.028s sys 0m0.046s time service postgresql stop real 0m1.281s user 0m0.035s sys 0m0.096s time service postgresql start real 0m2.242s user 0m0.031s sys 0m0.038s time service postgresql stop real 0m1.235s user 0m0.031s sys 0m0.036s = Systemd based: status postgresql.service postgresql.service - PostgreSQL database server Loaded: loaded (/lib/systemd/system/postgresql.service) Active: inactive (dead) CGroup: name=systemd:/system/postgresql.service time service postgresql start real 0m1.141s user 0m0.019s sys 0m0.019s time service postgresql stop real 0m1.146s user 0m0.017s sys 0m0.017s time service postgresql start real 0m1.153s user 0m0.016s sys 0m0.019s time service postgresql stop real 0m1.144s user 0m0.026s sys 0m0.014s -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 05:25 AM, Ralf Corsepius wrote: In general, there are other factors coming into play, such as parallel startup using more memory, parallelization not providing many advantages on systems with a small number of CPU cores, hard synchronisation points in the bootup process, poorly configured services, ... and finally ... bugs. Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plot bootup.svg and post it somewhere online Thanks JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Sep 15, 2011, at 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. better is 1 working and 9 sitting than 10 injured :) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Jan F. Chadima jchad...@redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 02:07 AM, drago01 wrote: On Thu, Sep 15, 2011 at 7:25 AM, Ralf Corsepiusrc040...@freenet.de wrote: On 09/14/2011 06:23 PM, drago01 wrote: On Wed, Sep 14, 2011 at 5:34 PM, Ralf Corsepiusrc040...@freenet.dewrote: snip Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Do they both start the same services? Unless you tweaked your fedora installation where we start a bunch of stuff that pretty much nobody would use in a typical desktop system that is to be expected. Is there a reason Fedora starts a bunch of stuff that pretty much nobody would use in a typical desktop system ? Fedora is certainly not targeted at the server/enterprise target, being on the bleeding edge and all. -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 12:45 PM, Steve Clark scl...@netwolves.com wrote: On 09/15/2011 02:07 AM, drago01 wrote: On Thu, Sep 15, 2011 at 7:25 AM, Ralf Corsepius rc040...@freenet.de wrote: On 09/14/2011 06:23 PM, drago01 wrote: On Wed, Sep 14, 2011 at 5:34 PM, Ralf Corsepiusrc040...@freenet.de wrote: snip Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Do they both start the same services? Unless you tweaked your fedora installation where we start a bunch of stuff that pretty much nobody would use in a typical desktop system that is to be expected. Is there a reason Fedora starts a bunch of stuff that pretty much nobody would use in a typical desktop system ? Well one of the reasons is every time someone tries to remove a service ends up in a flamefest with no outcome ;) (ex. search the list archives for MTA, which is the most prominent one). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 12:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. better is 1 working and 9 sitting than 10 injured :) Well why not fire the other 9 and save money then? ;) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Sep 15, 2011, at 1:02 PM, drago01 wrote: On Thu, Sep 15, 2011 at 12:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. better is 1 working and 9 sitting than 10 injured :) Well why not fire the other 9 and save money then? ;) so cut off the remanding cores from the CPU, save only one, good luck :D -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Jan F. Chadima jchad...@redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 1:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 1:02 PM, drago01 wrote: On Thu, Sep 15, 2011 at 12:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. better is 1 working and 9 sitting than 10 injured :) Well why not fire the other 9 and save money then? ;) so cut off the remanding cores from the CPU, save only one, good luck :D No I mean either write software to take advantage of them (i.e use them) or save money and buy slower / cheaper CPUs because you prefer software that does not use the available resources when needed. I prefer the former you seem to prefer the later ;) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadimajchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. Not quite. One normally wants to get a job done _reasonably_ and _reliable_ within the possibilities of _available resources_. Wrt. system bootup, this means, I don't care much on whether booting takes 40 or 80 seconds and don't care whether booting is parallelised, nor do I actually care about which init system is being used. However, resource requirements, usability and reliabilty are a concern to me. That said, /me feels similarly to Gnome3, systemd still has a long way ahead to mature. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Sep 15, 2011, at 1:13 PM, drago01 wrote: On Thu, Sep 15, 2011 at 1:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 1:02 PM, drago01 wrote: On Thu, Sep 15, 2011 at 12:10 PM, Jan F. Chadima jchad...@redhat.com wrote: On Sep 15, 2011, at 11:03 AM, drago01 wrote: On Thu, Sep 15, 2011 at 10:16 AM, Jan F. Chadima jchad...@redhat.com wrote: [...] . When watching the load of the virtual machine that starts with systemd it is clear to me that the total CPU consumption is significantly greater than in the case of upstart one. That's the whole point of doing things in parallel ... the CPU is actually being *used* hence the higher CPU consumption. When you have work to do you want to throw all resources at it. To get back to your digger analogy ... when you employ 10 workers you'd rather want all of them to work not one doing all the work and the other 9 just sitting around. better is 1 working and 9 sitting than 10 injured :) Well why not fire the other 9 and save money then? ;) so cut off the remanding cores from the CPU, save only one, good luck :D No I mean either write software to take advantage of them (i.e use them) or save money and buy slower / cheaper CPUs because you prefer software that does not use the available resources when needed. I prefer the former you seem to prefer the later ;) If your only goal is to boot up and the to switch off or to crash … then you have to parallelize boot. I prefer to load the system by the applications ….. And I have no care on 10 second longer boot Monitor the duration of the boot is ridiculous if uptime is more than a few hours. Time spent discussing about the absurdity of systemd is fair enough for more than a year everyday classical boot sequence -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Jan F. Chadima jchad...@redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/15 Michał Piotrowski mkkp...@gmail.com: 2011/9/14 Tom Lane t...@redhat.com: 3. As root, do date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns with dropped cache date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns 2011-09-15 08:44:40.348239703+02:00 2011-09-15 08:44:44.651134587+02:00 without dropped cache date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns 2011-09-15 08:45:38.388010217+02:00 2011-09-15 08:45:42.633229665+02:00 Note that those timings are not interesting -- you are comparing apples to watermelons. Got to take #4 into consideration for a useful reading. (Read Tom's original email to the end for an explanation -- the init script and service units exit on very peculiar conditions. So timing their wallclock is a waste of time.) 4. Note the time from the first date output to the database system is ready to accept connections message getting logged (in the appropriate file under /var/lib/pgsql/data/pg_log, if you haven't changed any other logging settings). Stop and restart a few times to get a good average. So, what are your timings following step 4? cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
Steve Clark (scl...@netwolves.com) said: Is there a reason Fedora starts a bunch of stuff that pretty much nobody would use in a typical desktop system ? One example of a synchronization point is the LVM/dmraid/etc. startup. If we were able to conclusively determine that there weren't any of these devices to start, we wouldn't have to have that synchronization point and tool execution. We're working on a way to properly assemble this stuff as it comes in if needed, but we're not there yet. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: In general, there are other factors coming into play, such as parallel startup using more memory, parallelization not providing many advantages on systems with a small number of CPU cores, hard synchronisation points in the bootup process, poorly configured services, ... and finally ... bugs. Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plot bootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg FWIW: Though the log says 110s, the wrist watch measured bootup time is ca. 120-140 secs. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 05:54 PM, Ralf Corsepius wrote: On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plot bootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg From the long delay before swap.target is reached it seems that your defined swap partition never comes up and systemd times out waiting on it. Michal -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 04:11 PM, Michal Schmidt wrote: On 09/15/2011 05:54 PM, Ralf Corsepius wrote: On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plotbootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg From the long delay before swap.target is reached it seems that your defined swap partition never comes up and systemd times out waiting on it. If you dont use any lvm, raid and encrypted devices, you can safely turn off all fedora-* services ( Anaconda (F16) finally offers desktop users an easy way to opt out from lvm without being partitioning experts ) #cd /lib/systemd/system #for i in fedora-*; do ln -s /dev/null /etc/systemd/system/$i;done We throw in udev-settle since it gets pulled in by the storage setup #ln -s /dev/null /etc/systemd/system/udev-settle.service If you want speed not eye candy you can disable plymouth #cd /lib/systemd/system #for i in plymouth-*; do ln -s /dev/null /etc/systemd/system/$i;done Then proceeding disabling all the service you dont use. #for i in service1 service2 service3 etc... ; do systemctl disable $i ;done JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, Sep 15, 2011 at 10:16:29AM +0200, Jan F. Chadima wrote: It amazes me how much energy people spend on it to save 3 seconds a day. Of course, if the computer being turned daily. In the case of a server time saving is even more amazing. Virtualization changes things. The way _I_ use Fedora involves booting it dozens of times a day (of course, I'm talking about a VM or a very small Fedora appliance). In any case, why *not* make things faster? The way things are now, I have computers that boot a hundred of times more *slowly* than the first computer I owned. Yes they do hundreds of times more things, but they also have a processor that is orders of magnitude faster. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 15 Sep 2011 18:27:29 +0100 Richard W.M. Jones rjo...@redhat.com wrote: On Thu, Sep 15, 2011 at 10:16:29AM +0200, Jan F. Chadima wrote: It amazes me how much energy people spend on it to save 3 seconds a day. Of course, if the computer being turned daily. In the case of a server time saving is even more amazing. Virtualization changes things. The way _I_ use Fedora involves booting it dozens of times a day (of course, I'm talking about a VM or a very small Fedora appliance). In any case, why *not* make things faster? The way things are now, I have computers that boot a hundred of times more *slowly* than the first computer I owned. Yes they do hundreds of times more things, but they also have a processor that is orders of magnitude faster. Rich. Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. -- Bernd Stramm bernd.str...@gmail.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 02:14 PM, Bernd Stramm wrote: On Thu, 15 Sep 2011 18:27:29 +0100 Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. Possibly for some. I think we need to divide things up into 4 categories (maybe there are more). In my view, for most scenarios startup time is not terribly important at all - for testers and developers it probably can be far more significant. Any speedups are great to have for most, but if the choice was speedup boot/start times or speedup the GUI, or fix bugs or just about anything else ... probably best to spend resources elsewhere ... (assuming resources are fungible, which of course they aren't :-) ) -- (i) Server. -- These run all the time - reboots are most often in maintenance window (or evenings / weekends for home servers) primarily if not soely for kernel updates. *** boot time pain more occasional fsck costs and not service startup Pain caused by O/S updates - rolling release model would be ideal for these. -- (ii) Desktop. -- Often left on, but reboots may happen a little sooner on kernel update. Some turn them off for power consumption or other reasons - not sure what fraction. *** Startup time not too important except possibly for developers - esp kernel devs. -- (iii) laptop. -- typically put into sleep mode for transportation (i.e. close lid). Restart time is extremely fast. Reboots as for (ii). *** Similar to (ii) -- (iv) Virtual Machines -- (a) Server *** Rarely rebooted - otherwise same as (ii) (b) specific needs (e.g. program a remote which needs windows ** boot time not too significant. (c) Testing and development on VM's - *** boot time probably could be important -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 2011-09-15 at 17:54 +0200, Ralf Corsepius wrote: On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: In general, there are other factors coming into play, such as parallel startup using more memory, parallelization not providing many advantages on systems with a small number of CPU cores, hard synchronisation points in the bootup process, poorly configured services, ... and finally ... bugs. Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plot bootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg FWIW: Though the log says 110s, the wrist watch measured bootup time is ca. 120-140 secs. OS-side boot measurement tools cannot know how long the BIOS/EFI and bootloader stages of boot took. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 2011-09-15 at 14:14 -0400, Bernd Stramm wrote: Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. What you really want there, though, is efficient and reliable suspend, not full power cycle. This is what everyone does with cellphones and tablets. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 15 Sep 2011 15:27:49 -0700 Adam Williamson awill...@redhat.com wrote: On Thu, 2011-09-15 at 14:14 -0400, Bernd Stramm wrote: Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. What you really want there, though, is efficient and reliable suspend, not full power cycle. This is what everyone does with cellphones and tablets. Yes, usually a user would want to preserve the state of their work across the periods of them they are not using the system, as opposed to a cold boot that clears everything. That is also what is normally wanted from overnight shutdowns of systems that run during the business day. -- Bernd Stramm bernd.str...@gmail.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 05:27 PM, Adam Williamson wrote: On Thu, 2011-09-15 at 14:14 -0400, Bernd Stramm wrote: Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. What you really want there, though, is efficient and reliable suspend, not full power cycle. This is what everyone does with cellphones and tablets. Or bulletproof session management. With an encrypted disk, putting the data at rest is beneficial. Alternatively, hibernate could work if it were much faster, but my recent experience with hibernate is that a full boot is notably faster. Once everything is built on systemd units and it be parallelized... - Michael -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Thu, 2011-09-15 at 19:10 -0500, Michael Ekstrand wrote: On 09/15/2011 05:27 PM, Adam Williamson wrote: On Thu, 2011-09-15 at 14:14 -0400, Bernd Stramm wrote: Many computers are booted very rarely, once a day or so, and then sit idle for very long periods of time. This is very wasteful. The reason people do this is because booting takes a long time compared to starting the set of applications they use. If you could boot and start applications in say, 1/2 second, usage patterns would be completely different. What you really want there, though, is efficient and reliable suspend, not full power cycle. This is what everyone does with cellphones and tablets. Or bulletproof session management. True. As far as GNOME goes, though, whenever you suggest 'bulletproof session management', they say 'that's what suspend is for'... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/15/2011 06:11 PM, Michal Schmidt wrote: On 09/15/2011 05:54 PM, Ralf Corsepius wrote: On 09/15/2011 09:42 AM, Jóhann B. Guðmundsson wrote: On 09/15/2011 05:25 AM, Ralf Corsepius wrote: Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Could you run systemd-analyze plotbootup.svg and post it somewhere online See: http://corsepiu.fedorapeople.org/scratch/bootup-20110915.1.svg From the long delay before swap.target is reached it seems that your defined swap partition never comes up and systemd times out waiting on it. Good catch! A bogus UUID for swap in /etc/fstab seems to have been the cause of this excessive bootup time. After having fixed the UUID, bootup times of this configuration are now are at ca. 70 secs, i.e. at par with booting from internal HD. Interestingly the Ubuntu setup had a similar UUID issue. After having fixed it there, booting Ubuntu now is at ca. 50 secs. However, this questions the earlier claims related to the impact of disk-io speed on bootup times - If disk-io was the limiting factor on this HW, I'd expect booting from internal HD to be significantly faster than booting from an external USB-HD. Apparently this does not apply [1]. Ralf [1] Both setups' configurations are similar, but I haven't checked the details, yet. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Tom Lane t...@redhat.com: =?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/14 Tom Lane t...@redhat.com: Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? I copied the service to /etc/systemd/system and changed PGDATA variable, then I enabled the service and rebooted. After boot I checked system boot time with systemd-analyze - I saw that it starts slow, so I disabled it and deleted from /etc/systemd/system. After another reboot again checked boot time with systemd-analyze. I'll check tomorrow how repeatable is native service boot time. I'd suggest first timing some rounds of manual service postgresql start, service postgresql stop to see what things look like without all the other noise involved in a system boot. Ok, I made four series of tests: - start/stop an old init script - start/stop an old init script with dropping caches - should simulate system booting - start/stop service file - start/stop service file with dropping caches In each series of tests were repeated five times. series 1 - start - 2.2+ sec series 1 - stop - 1.2+ sec series 2 - start - 2.4+ sec series 2 - stop - 1.3+ sec series 3 - start - 3.1+ sec series 3 - stop - 1.1+ sec series 4 - start - 4.2+ sec series 4 - stop - 1.1+ sec Results are reproducible. === old init script === time sudo systemctl start postgresql.service real0m2.248s user0m0.012s sys 0m0.022s time sudo systemctl stop postgresql.service real0m1.288s user0m0.007s sys 0m0.027s time sudo systemctl start postgresql.service real0m2.252s user0m0.014s sys 0m0.020s time sudo systemctl stop postgresql.service real0m1.282s user0m0.012s sys 0m0.021s time sudo systemctl start postgresql.service real0m2.230s user0m0.006s sys 0m0.028s time sudo systemctl stop postgresql.service real0m1.273s user0m0.012s sys 0m0.021s time sudo systemctl start postgresql.service real0m2.232s user0m0.007s sys 0m0.028s time sudo systemctl stop postgresql.service real0m1.266s user0m0.010s sys 0m0.023s time sudo systemctl start postgresql.service real0m2.246s user0m0.011s sys 0m0.023s time sudo systemctl stop postgresql.service real0m1.277s user0m0.007s sys 0m0.026s === old init script + echo 1 drop_caches === time sudo systemctl start postgresql.service real0m2.586s user0m0.013s sys 0m0.034s time sudo systemctl stop postgresql.service real0m1.393s user0m0.009s sys 0m0.034s time sudo systemctl start postgresql.service real0m2.492s user0m0.014s sys 0m0.032s time sudo systemctl stop postgresql.service real0m1.391s user0m0.009s sys 0m0.036s time sudo systemctl start postgresql.service real0m2.598s user0m0.009s sys 0m0.037s time sudo systemctl stop postgresql.service real0m1.385s user0m0.011s sys 0m0.031s time sudo systemctl start postgresql.service real0m2.563s user0m0.015s sys 0m0.031s time sudo systemctl stop postgresql.service real0m1.384s user0m0.015s sys 0m0.029s time sudo systemctl start postgresql.service real0m2.581s user0m0.016s sys 0m0.030s time sudo systemctl stop postgresql.service real0m1.391s user0m0.010s sys 0m0.035s === systemd service === time sudo systemctl start postgresql.service real0m3.167s user0m0.008s sys 0m0.025s time sudo systemctl stop postgresql.service real0m1.124s user0m0.014s sys 0m0.020s time sudo systemctl start postgresql.service real0m3.180s user0m0.009s sys 0m0.024s time sudo systemctl stop postgresql.service real0m1.121s user0m0.008s sys 0m0.025s time sudo systemctl start postgresql.service real0m3.164s user0m0.012s sys 0m0.022s time sudo systemctl stop postgresql.service real0m1.112s user0m0.006s sys 0m0.027s time sudo systemctl start postgresql.service real0m3.161s user0m0.014s sys 0m0.019s time sudo systemctl stop postgresql.service real0m1.130s user0m0.010s sys 0m0.024s time sudo systemctl start postgresql.service real0m3.174s user0m0.011s sys 0m0.022s time sudo systemctl stop postgresql.service real0m1.123s user0m0.008s sys 0m0.026s === systemd service + echo 1 drop_caches === time sudo systemctl start postgresql.service real0m4.320s user0m0.014s sys 0m0.030s time sudo systemctl stop postgresql.service real0m1.196s user0m0.012s sys 0m0.033s time sudo systemctl start postgresql.service real0m4.289s user0m0.008s sys 0m0.037s time sudo systemctl stop postgresql.service real0m1.204s user0m0.011s sys 0m0.033s time sudo systemctl start postgresql.service real0m4.284s
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Adam Williamson awill...@redhat.com: On Tue, 2011-09-13 at 20:37 -0400, Tom Lane wrote: =?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/14 Tom Lane t...@redhat.com: Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? I copied the service to /etc/systemd/system and changed PGDATA variable, then I enabled the service and rebooted. After boot I checked system boot time with systemd-analyze - I saw that it starts slow, so I disabled it and deleted from /etc/systemd/system. After another reboot again checked boot time with systemd-analyze. I'll check tomorrow how repeatable is native service boot time. I'd suggest first timing some rounds of manual service postgresql start, service postgresql stop to see what things look like without all the other noise involved in a system boot. yeah; it may well be starting at a different point in the process now it's being ordered as a systemd-native service rather than via lsb deps. did the *overall* startup time increase by a corresponding amount? I sent the results of tests for simple daemon start/stop. I see no point in testing boot speed in this case, but of course I can do some tests later if you think that the results may be useful. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/13/2011 11:03 PM, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lanet...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service First of all you cant reliably measure startup performance between legacy sysv init script and a native systemd unit since either one of those might be doing less/more. So I wonder if it makes sense to convert in such case? Yes systemd is bringing more to the table then just startup speed which by the way is completely irrelevant in server environment. I personally look at the boot decrease as an side effect not an feature. We are still a long way from actually deliver that degreased boot time out of the box into the hands of the desktop end users or perhaps I should rather say there is room for plenty of improvements in that regard. Once we have done that it willl highlight other issues such as the log into the desktop time which currently is taking longer time for me ( Gnome it might be faster on other *DE ) than it takes to booting the operating system. JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 10:56 AM, Steve Clark wrote: Thats right! Just wave your hands and say it is all ok that systemd is slower now but it is doing so much more and we will make it better in the future...! I never said that what I said was it's irrelivent the startup time of a service on a service platform which you should actually know yourself if are used to run servers in a proper production deployment not in your garage at back home. This was a simple test to start postgresql - what else needs to be done! An simple test to measure this reliably is to strip down the legacy sysv init script to the start up command only and have a strip down unit file to the startup command only. Then time the startup of either. I would be surprised if there was any measurable difference by all means since this seems so important to you test it. JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 04:35 AM, Jóhann B. Guðmundsson wrote: On 09/13/2011 11:03 PM, Micha? Piotrowski wrote: Hi 2011/9/13 Tom Lanet...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service First of all you cant reliably measure startup performance between legacy sysv init script and a native systemd unit since either one of those might be doing less/more. So I wonder if it makes sense to convert in such case? Yes systemd is bringing more to the table then just startup speed which by the way is completely irrelevant in server environment. I personally look at the boot decrease as an side effect not an feature. We are still a long way from actually deliver that degreased boot time out of the box into the hands of the desktop end users or perhaps I should rather say there is room for plenty of improvements in that regard. Once we have done that it willl highlight other issues such as the log into the desktop time which currently is taking longer time for me ( Gnome it might be faster on other *DE ) than it takes to booting the operating system. JBG Thats right! Just wave your hands and say it is all ok that systemd is slower now but it is doing so much more and we will make it better in the future...! This was a simple test to start postgresql - what else needs to be done! -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 14 Sep 2011 08:35:47 + Jóhann B. Guðmundsson johan...@gmail.com wrote: 2011/9/13 Tom Lanet...@redhat.com: ... I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service First of all you cant reliably measure startup performance between legacy sysv init script and a native systemd unit since either one of those might be doing less/more. As far as I understand it, systemd follows a dependendy dag, starting every node (service) as soon as possible. There is no guarantee that this will _always_ finish everything sooner than some other, arbitrary ordering. In this kind of environment, usually it will go faster, but not always. -- Bernd Stramm bernd.str...@gmail.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 01:03:04AM +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 4:22 PM, Richard W.M. Jones rjo...@redhat.com wrote: On Wed, Sep 14, 2011 at 01:03:04AM +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. My laptop (using a Samsung ssd) booted up in 14-16 seconds on F14. Now (F15) it boots up in 7-8 seconds. Which is a rather huge boost. (Not using lvm or any fancy stuff). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Jóhann B. Guðmundsson johan...@gmail.com: On 09/14/2011 10:56 AM, Steve Clark wrote: This was a simple test to start postgresql - what else needs to be done! An simple test to measure this reliably is to strip down the legacy sysv init script to the start up command only and have a strip down unit file to the startup command only. Then time the startup of either. Why? The current numbers show that the service file is _slower_ even when the old init script is supposedly doing much more work in shell. If anything, stripping the unessential parts should make the service file _even slower_ in relative terms. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 02:22 PM, Richard W.M. Jones wrote: Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. We arent optimising the default desktop install live or otherwise for an actual desktop install. It's currently aimed at Generic or Corporate installs. This is my HP Pavilion DM1 laptop bootup time with rooms for improvement after minor tweaking on rotating media with no loss of functionality what so ever. # systemd-analyze Startup finished in 5761ms (kernel) + 11938ms (userspace) = 17700ms I'm assuming the kernel time improves once we turn off debugging in it. Afaik none of the *DE groups are properly tweaking things for their target audience. Anything above 4 seconds on ssd's would I consider unacceptable ( I think Lennart and Kay are booting around s14s second on those ) JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
=?ISO-8859-2?Q?Miloslav_Trma=E8?= m...@volny.cz writes: 2011/9/14 Jóhann B. Guðmundsson johan...@gmail.com: An simple test to measure this reliably is to strip down the legacy sysv init script to the start up command only and have a strip down unit file to the startup command only. Then time the startup of either. Why? The current numbers show that the service file is _slower_ even when the old init script is supposedly doing much more work in shell. If anything, stripping the unessential parts should make the service file _even slower_ in relative terms. Yes. The unit file is already stripped down: it does nothing except pg_ctl start. The init script had accumulated a whole lot of perhaps-unnecessary sanity-checking, which frankly I'd rather have kept but the systemd mantra seems to be no shell scripting so I didn't. Michal's numbers look pretty damning, and I find it remarkable that the systemd advocates seem to have managed not to read them, let alone admit that they suggest something's seriously wrong. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 7:28 AM, Tom Lane t...@redhat.com wrote: Michal's numbers look pretty damning, and I find it remarkable that the systemd advocates seem to have managed not to read them, let alone admit that they suggest something's seriously wrong. Michal's numbers look intriguing. I look forward to watching the discussion that unfolds in the upstream mailinglist and see if someone can trackdown an optimization based on those numbers. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 04:31 PM, drago01 wrote: On Wed, Sep 14, 2011 at 4:22 PM, Richard W.M. Jonesrjo...@redhat.com wrote: On Wed, Sep 14, 2011 at 01:03:04AM +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lanet...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) Is systemd boot actually any faster? Not from my personal experience. There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. My laptop (using a Samsung ssd) booted up in 14-16 seconds on F14. Now (F15) it boots up in 7-8 seconds. Which is a rather huge boost. (Not using lvm or any fancy stuff). My netbook boots up F14 in ca. 60 secs, while F15 boots up in 62 secs. I'd call this below measurement accuracy. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Jóhann B. Guðmundsson johan...@gmail.com: On 09/14/2011 02:22 PM, Richard W.M. Jones wrote: Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. We arent optimising the default desktop install live or otherwise for an actual desktop install. It's currently aimed at Generic or Corporate installs. This is my HP Pavilion DM1 laptop bootup time with rooms for improvement after minor tweaking on rotating media with no loss of functionality what so ever. # systemd-analyze Startup finished in 5761ms (kernel) + 11938ms (userspace) = 17700ms I'm assuming the kernel time improves once we turn off debugging in it. Afaik none of the *DE groups are properly tweaking things for their target audience. Anything above 4 seconds on ssd's would I consider unacceptable ( I think Lennart and Kay are booting around s14s second on those ) I use ssd. Startup finished in 1819ms (kernel) + 2495ms (initrd) + 8013ms (userspace) = 12328ms The slowest part is 3633ms mysqld.service 2288ms postgresql.service old versions that are better than native services. JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
Michał Piotrowski wrote: Ok, I made four series of tests: - start/stop an old init script - start/stop an old init script with dropping caches - should simulate system booting - start/stop service file - start/stop service file with dropping caches Just to be clear. This is done on an F15 system using the postgresql systemd based service file from the F16 branch? I'd like to do some followup testing on my systems and I want to make sure I'm doing as faithful comparison of the init configs you tested. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 11:28 AM, Tom Lane wrote: =?ISO-8859-2?Q?Miloslav_Trma=E8?=m...@volny.cz writes: 2011/9/14 Jóhann B. Guðmundssonjohan...@gmail.com: An simple test to measure this reliably is to strip down the legacy sysv init script to the start up command only and have a strip down unit file to the startup command only. Then time the startup of either. Why? The current numbers show that the service file is _slower_ even when the old init script is supposedly doing much more work in shell. If anything, stripping the unessential parts should make the service file _even slower_ in relative terms. Yes. The unit file is already stripped down: it does nothing except pg_ctl start. The init script had accumulated a whole lot of perhaps-unnecessary sanity-checking, which frankly I'd rather have kept but the systemd mantra seems to be no shell scripting so I didn't. Michal's numbers look pretty damning, and I find it remarkable that the systemd advocates seem to have managed not to read them, let alone admit Don't confuse we with facts! I've already made up my mind! ;-) that they suggest something's seriously wrong. regards, tom lane -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.cl...@netwolves.com http://www.netwolves.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 5:34 PM, Ralf Corsepius rc040...@freenet.de wrote: On 09/14/2011 04:31 PM, drago01 wrote: On Wed, Sep 14, 2011 at 4:22 PM, Richard W.M. Jonesrjo...@redhat.com wrote: On Wed, Sep 14, 2011 at 01:03:04AM +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lanet...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) Is systemd boot actually any faster? Not from my personal experience. There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. My laptop (using a Samsung ssd) booted up in 14-16 seconds on F14. Now (F15) it boots up in 7-8 seconds. Which is a rather huge boost. (Not using lvm or any fancy stuff). My netbook boots up F14 in ca. 60 secs, while F15 boots up in 62 secs. I'd call this below measurement accuracy. What kind of disk is that? For a mechanical drive any gain from parallel startup would get killed by disk seeks. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Jef Spaleta jspal...@gmail.com: Michał Piotrowski wrote: Ok, I made four series of tests: - start/stop an old init script - start/stop an old init script with dropping caches - should simulate system booting - start/stop service file - start/stop service file with dropping caches Just to be clear. This is done on an F15 system using the postgresql systemd based service file from the F16 branch? Exactly. F15, PostgreSQL 9.0 and just service file from PostgreSQL 9.1. Root filesystem and database are on SSD and Ext4. I'd like to do some followup testing on my systems and I want to make sure I'm doing as faithful comparison of the init configs you tested. Please post your results I am curious if this is repeatable on other systems. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 2011-09-14 at 15:22 +0100, Richard W.M. Jones wrote: On Wed, Sep 14, 2011 at 01:03:04AM +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. At present I'd say on average not hugely. AFAIK no-one's really worked on boot speed optimization since right after I came on board at RH: https://fedoraproject.org/wiki/Test_Day:2009-02-19 (man, it's so much harder to find test days from back when we didn't put the subject in the page name...) overall boot speed is a pretty chaotic equation, with a lot more than just the init system involved. I think it's fair to say that, all other things being equal in an ideal world, it may be possible to make things faster with systemd than with upstart, but it's by no means a given, and there are likely other things we can do that will have a more significant impact on boot speeds. We could do another such test event for F17, but to be really effective it needs buy-in from a developer in the position to understand the results and do something about it, like Lennart and Harald. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 2011-09-14 at 11:28 -0400, Tom Lane wrote: Michal's numbers look pretty damning, and I find it remarkable that the systemd advocates seem to have managed not to read them, let alone admit that they suggest something's seriously wrong. Tests of a single service on a single system in an unknown configuration? Why, that's certainly a great reason to panic! It's an interesting result that's worth looking into, but I don't think calling it 'damning' is helping the tone of the debate. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 2011-09-14 at 18:23 +0200, drago01 wrote: What kind of disk is that? For a mechanical drive any gain from parallel startup would get killed by disk seeks. There is no real 'gain' from parallel startup when switching from upstart to systemd, especially in the F15 case where almost all the initscripts used are still the same. It seems to often be forgotten that upstart already did parallelization, this is not something new we're getting with systemd. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 2011-09-14 at 17:47 +0200, Michał Piotrowski wrote: 2011/9/14 Jóhann B. Guðmundsson johan...@gmail.com: On 09/14/2011 02:22 PM, Richard W.M. Jones wrote: Is systemd boot actually any faster? There seems to be no noticable difference in boot times for me over whatever we were using in F14. ie. both methods still takes ages, far longer than should be necessary. We arent optimising the default desktop install live or otherwise for an actual desktop install. It's currently aimed at Generic or Corporate installs. ... Anything above 4 seconds on ssd's would I consider unacceptable ( I think Lennart and Kay are booting around s14s second on those ) I use ssd. Startup finished in 1819ms (kernel) + 2495ms (initrd) + 8013ms (userspace) = 12328ms The slowest part is 3633ms mysqld.service 2288ms postgresql.service old versions that are better than native services. I think Johann's 1-4s scenario assumes a basic desktop configuration and a tweaked startup config aimed at such a system. Such a system would not have two big database engines starting up, but zero. =) The lowest I've seen on my system, which is quick and has a third-gen SSD in it, is 8s, but I haven't made any tweaks to startup config at all. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Michał Piotrowski mkkp...@gmail.com Exactly. F15, PostgreSQL 9.0 and just service file from PostgreSQL 9.1. Root filesystem and database are on SSD and Ext4. Okay... brace yourself. I just ran this test on my non-SSD ext4 based F15 system and I get the opposite result on start...repeatably. On my system the systemd based start up in the admittedly simple service postgresql start testing is consistently faster by about a second. And just as interesting... my sysinit times are slower than yours, that's a real head scratcher for me. If my db test was less overall work to init for some reason I'd expect my numbers to be faster than yourse in both cases. I have no off the top of my head theory that could explain that except that your low level disk i/o is significantly different than mine. Both variants of service postgresql stop is within a 1/10 of a second in my testing I can give you the screen log captures for my tests if you desire to review my actions but here here is the summary for my simple timing test: And admittedly I'm using a simply initiated postgres db. It could be that your real world database initialization workload is more intensive than my test and the disk i/o is now a limiting factor in a different way. And of course my numbers do not discount what your seeing as my test is as anecdotal as your is. Your numbers may still be indicative of a more nuanced problem that can be resolved and its good to have your numbers as a starting point for a discussion around understanding optimization issues (like disk i/o). We just can't jump to conclusions about why you are seeing what your are seeing. I hope my numbers serve as a cautionary reminder to those others on this list that benchmarking disk io intensive tasks can be very complex and very system dependent. Certain other people in this discussion are being overly bombastic and seem to have forgotten this fact. I hope for all our sakes they can find a way to ratchet down the hyperbole and look at these sort of issues with a more clinical approach. Okay so here's my summary of the quick tests. Please if you have an updated methodology for me to test, let me know. I'm willing to reuse a specially crafted pgsql database if you feel that will help you. Though I'd think we wan't to do that sort of deep dive comparison off list until we are ready to publish some summaries after we both feel comfortable with the test runs. -jef == SysVinit: service postgresql status postgresql.service - LSB: start and stop PostgreSQL server Loaded: loaded (/etc/rc.d/init.d/postgresql) Active: inactive (dead) CGroup: name=systemd:/system/postgresql.service time service postgresql start real0m2.329s user0m0.028s sys 0m0.046s time service postgresql stop real0m1.281s user0m0.035s sys 0m0.096s time service postgresql start real0m2.242s user0m0.031s sys 0m0.038s time service postgresql stop real0m1.235s user0m0.031s sys 0m0.036s = Systemd based: status postgresql.service postgresql.service - PostgreSQL database server Loaded: loaded (/lib/systemd/system/postgresql.service) Active: inactive (dead) CGroup: name=systemd:/system/postgresql.service time service postgresql start real0m1.141s user0m0.019s sys 0m0.019s time service postgresql stop real0m1.146s user0m0.017s sys 0m0.017s time service postgresql start real0m1.153s user0m0.016s sys 0m0.019s time service postgresql stop real0m1.144s user0m0.026s sys 0m0.014s -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
=?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: Ok, I made four series of tests: - start/stop an old init script - start/stop an old init script with dropping caches - should simulate system booting - start/stop service file - start/stop service file with dropping caches In each series of tests were repeated five times. series 1 - start - 2.2+ sec series 1 - stop - 1.2+ sec series 2 - start - 2.4+ sec series 2 - stop - 1.3+ sec series 3 - start - 3.1+ sec series 3 - stop - 1.1+ sec series 4 - start - 4.2+ sec series 4 - stop - 1.1+ sec Results are reproducible. I tried to replicate these results on my own F15 laptop, and could not --- the service file method doesn't really seem significantly faster than the init script, but it's not slower either. Here's what I did: 1. Install the postgresql-9.1.0 RPMs (rebuilt for F15 of course) and do postgresql-setup initdb. 2. Set log_line_prefix = '%m %p ' and log_connections = on in postgresql.conf, so that log messages will be timestamped. Also set timezone and log_timezone to desired values (I use 'US/Eastern'); if you don't do that, the server startup time is increased significantly while Postgres tries to figure out the system timezone setting. 3. As root, do date --rfc-3339=ns ; systemctl start postgresql.service ; date --rfc-3339=ns 4. Note the time from the first date output to the database system is ready to accept connections message getting logged (in the appropriate file under /var/lib/pgsql/data/pg_log, if you haven't changed any other logging settings). Stop and restart a few times to get a good average. 5. Install the F15 version of postgresql.init (be sure to adjust the PGVERSION setting near the top of the file to be 9.1.0). 6. Start it that way a few times, note the same elapsed time. I'm seeing numbers consistently around 0.3 second for the unit file, and a bit less consistent but maybe 0.35 - 0.5 second for the script. Note that the time for the service to report itself ready after the database has started is likely to be quite a bit different between the two methods, but that is not systemd's fault. The init script just launches the postmaster, sleeps for 2 seconds, and then reports OK if it sees the postmaster has created a PID file. The unit file uses pg_ctl, which actually waits till it can make a successful connection to the postmaster, sleeping 1 second between tries. So it's a bit of a crapshoot which will be longer, though if you are starting from a clean database shutdown I'd expect pg_ctl to usually come back after the first sleep. So I'm not sure what's happening on Michal's machine, but from here I don't see anything egregiously wrong with systemd's performance on this test. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 6:54 PM, Adam Williamson awill...@redhat.com wrote: On Wed, 2011-09-14 at 18:23 +0200, drago01 wrote: What kind of disk is that? For a mechanical drive any gain from parallel startup would get killed by disk seeks. There is no real 'gain' from parallel startup when switching from upstart to systemd, Well we never had upstart (it was just a renamed sysvinit i.e we didn't use any of its parallel startup stuff). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 2011-09-14 at 19:43 +0200, drago01 wrote: On Wed, Sep 14, 2011 at 6:54 PM, Adam Williamson awill...@redhat.com wrote: On Wed, 2011-09-14 at 18:23 +0200, drago01 wrote: What kind of disk is that? For a mechanical drive any gain from parallel startup would get killed by disk seeks. There is no real 'gain' from parallel startup when switching from upstart to systemd, Well we never had upstart (it was just a renamed sysvinit i.e we didn't use any of its parallel startup stuff). hum, I thought we were using its ability to parallel start things based on LSB deps. oh well. it's certainly *capable* of that, anyway. so parallelization is not a systemd win. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, Sep 14, 2011 at 1:06 PM, Tom Lane t...@redhat.com wrote: Here's what I did: ... 4. Note the time from the first date output to the database system is ready to accept connections message getting logged Tom, your methodology is sound. You're too kind to say but given the completion criteria of both init V script and systemd unit file, comparing their run times is a complete waste of time. The same is likely to be true with any long-lived network service. Thanks for steering the discussion towards something sane and productive. I'm seeing numbers consistently around 0.3 second for the unit file, and a bit less consistent but maybe 0.35 - 0.5 second for the script. Makes sense. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
Adam Williamson (awill...@redhat.com) said: Well we never had upstart (it was just a renamed sysvinit i.e we didn't use any of its parallel startup stuff). hum, I thought we were using its ability to parallel start things based on LSB deps. oh well. it's certainly *capable* of that, anyway. so parallelization is not a systemd win. upstart can parallelize native upstart services; it doesn't do anything with sysv/LSB scripts. In upstart, they were launched from /etc/rc.d/rc, just as before. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Wed, 14.09.11 01:03, Michał Piotrowski (mkkp...@gmail.com) wrote: Hi 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? (I know that it is not about boot speed, it can start slower if needed.) I am currently travelling, so I don't have the peace to fully read and reply to this thread, but let me clear up a few things: a) don't misunderstand systemd-analyze, the times reported by it are wallclock times, and hence parallelization increases these values since the jobs are influenced by others (while decreasing the overall time). the values are interesting in comparison to other values from the same boot, but even then need to be read with a grain of salt. b) systemd doesn't really do anything that was computation intensive. so there's no real reason for the slowdown, and definitely fixable. I am not sure what pgsql is doing there on startup... might be something on those scripts, not necessarily systemd at fault. As long as this is a problem for pg only and nothing else this is a strong indication for this. Either way, I don't want to point fingers, and I don't really know what's going on, but what I actually do know is that the currently available measurement data is not useful, we need to investigate this further and then figure out what's really going on and where there's something to fix. I'll look into this in detail next week. Thanks, Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On 09/14/2011 06:23 PM, drago01 wrote: On Wed, Sep 14, 2011 at 5:34 PM, Ralf Corsepiusrc040...@freenet.de wrote: My netbook boots up F14 in ca. 60 secs, while F15 boots up in 62 secs. I'd call this below measurement accuracy. What kind of disk is that? It's ca. 3 years old WD Scorpio Blue 160 GB ( WD1600BEVT) in a first generation Atom N270 (32bit only) based netbook w/ 2GB RAM. For a mechanical drive any gain from parallel startup would get killed by disk seeks. Sure, slow disks certainly are a factor contributing to slow bootup times. In general, there are other factors coming into play, such as parallel startup using more memory, parallelization not providing many advantages on systems with a small number of CPU cores, hard synchronisation points in the bootup process, poorly configured services, ... and finally ... bugs. Anyway, some more figures: On the same machine, bootup times when booting from a (slow) external (IDE) USB2 HD: - Fedora 15/i386: ca. 135 secs. - Ubuntu 11.04/i386: ca. 70 secs. [Here bootup time: Wirst watch measured time from grub prompt to login screen] It shows the effect of slow disks (60secs w/ internal HD vs. 2.15 minutes w/ USB HD), but raises questions on why Ubuntu appears to be so much faster in this configuration. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
Hi, On Wed, 2011-09-14 at 01:03 +0200, Michał Piotrowski wrote: Hi 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? Shouldn't better ask this on systemd-devel ML (systemd-de...@lists.freedesktop.org) ? (I know that it is not about boot speed, it can start slower if needed.) regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- Sérgio M. B. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
=?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? The reason it makes sense to convert is that sysv init scripts are second-class citizens in the eyes of systemd, and the systemd developers exhibit no interest in making such scripts actually usable. In particular the handling of error reports is several steps south of unacceptable --- cf bug #622663, which is more than a year old and has been steadfastly ignored. I don't think this is accidental; the systemd developers want to force all packages to migrate to native systemd scripts eventually, and one of the best ways to do that is to make sure that the old scripts are as unfriendly to use as possible. Minor performance differences aren't going to outweigh complaints like my database didn't start and there is no useful error message anywhere, especially not where systemd told me to look. Still, given that we were told that eliminating the use of shell scripting ought to make things faster, your report surprises me. Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
2011/9/14 Tom Lane t...@redhat.com: =?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/13 Tom Lane t...@redhat.com: (This isn't new with 9.1, btw --- the last version or so of 9.0 for F16 was the same, since we switched over to native systemd files.) I used this service file on F15 and it starts slower 4214ms postgresql.service if we compare with an old SysVinit script 2469ms postgresql.service So I wonder if it makes sense to convert in such case? The reason it makes sense to convert is that sysv init scripts are second-class citizens in the eyes of systemd, and the systemd developers exhibit no interest in making such scripts actually usable. In particular the handling of error reports is several steps south of unacceptable --- cf bug #622663, which is more than a year old and has been steadfastly ignored. I don't think this is accidental; the systemd developers want to force all packages to migrate to native systemd scripts eventually, and one of the best ways to do that is to make sure that the old scripts are as unfriendly to use as possible. Minor performance differences aren't going to outweigh complaints like my database didn't start and there is no useful error message anywhere, especially not where systemd told me to look. Still, given that we were told that eliminating the use of shell scripting ought to make things faster, your report surprises me. I'm surprised too - it's the first such case that native service boots slower that I saw. Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? I copied the service to /etc/systemd/system and changed PGDATA variable, then I enabled the service and rebooted. After boot I checked system boot time with systemd-analyze - I saw that it starts slow, so I disabled it and deleted from /etc/systemd/system. After another reboot again checked boot time with systemd-analyze. I'll check tomorrow how repeatable is native service boot time. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Best regards, Michal http://eventhorizon.pl/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
=?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/14 Tom Lane t...@redhat.com: Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? I copied the service to /etc/systemd/system and changed PGDATA variable, then I enabled the service and rebooted. After boot I checked system boot time with systemd-analyze - I saw that it starts slow, so I disabled it and deleted from /etc/systemd/system. After another reboot again checked boot time with systemd-analyze. I'll check tomorrow how repeatable is native service boot time. I'd suggest first timing some rounds of manual service postgresql start, service postgresql stop to see what things look like without all the other noise involved in a system boot. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: what if native systemd service is slower than old sysvinit script?
On Tue, 2011-09-13 at 20:37 -0400, Tom Lane wrote: =?ISO-8859-2?Q?Micha=B3_Piotrowski?= mkkp...@gmail.com writes: 2011/9/14 Tom Lane t...@redhat.com: Certainly postgresql.init was never exactly lean-and-mean, so it seems like it ought to have been doing more work than the unit file requires. Are you sure you were comparing apples to apples as far as the state of the database, kernel disk cache, etc goes? I copied the service to /etc/systemd/system and changed PGDATA variable, then I enabled the service and rebooted. After boot I checked system boot time with systemd-analyze - I saw that it starts slow, so I disabled it and deleted from /etc/systemd/system. After another reboot again checked boot time with systemd-analyze. I'll check tomorrow how repeatable is native service boot time. I'd suggest first timing some rounds of manual service postgresql start, service postgresql stop to see what things look like without all the other noise involved in a system boot. yeah; it may well be starting at a different point in the process now it's being ordered as a systemd-native service rather than via lsb deps. did the *overall* startup time increase by a corresponding amount? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
