Re: running poudriere with 8 builders

[Matthias Apitz]

On Saturday, 4 March 2017 17:19:12 CET, Michael Gmelin  
wrote:





I now have made tar-backups of /usr/local/poudiere und /usr/local
(without) poudriere; I will umount the 2nd disk (da1) and restore
/usr/local, which will end up in the 1st disk;


Make sure to initialize poudriere properly on zfs (it will 
create various filesystems for jails/ports etc). Simply 
untarring won't cut it.


I moved /usr/local to the 1st disk, created a ZFS pool.on the other and 
there the jail and ports tree with the poudriere command...


Now it's really starting fast, within a minute, and is building around 100 
ports in 10 minutes, all CPUs run busy.


thanks

matthias



--
Sent from my Ubuntu phone
http://www.unixarea.de/
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Dewayne Geraghty]

Set maxvnodes to 50 and jump 100k until you're happy.  I have Xeon e3
that uses 800,000 and thats reasonable; but i keep it at load 8 for a two
days. Keep an eye on your swap, I set mine to swap out idle stuff early.

Thanks for sharing your experience
-- 
*Disclaimer:*



*As implied by email protocols, the information in this message is not
confidential. Any intermediary or recipient may inspect, modify (add),
copy, forward, reply to, delete, or filter email for any purpose unless
said parties are otherwise obligated.  Nothing in this message may be
legally binding without cryptographic evidence of its integrity and/or
confidentiality.*
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Michael Gmelin]

> On 4 Mar 2017, at 16:32, Matthias Apitz  wrote:
> 
>> El día Saturday, March 04, 2017 a las 12:57:40PM +0100, Kurt Jaeger escribió:
>> 
>> Hi!
>> 
 Did you use ZFS as file system ?
>>> 
>>> When I did the disk setup, I was also thinking in ZFS, but this would have 
>>> reduced the netto disk space from 2* 280 GB to the half.
>> 
>> It does not have to be that way. Use only one disk for the system
>> and the second disk for a poudriere ZFS pool ?
>> 
>>> I have / for 
>>> all.the system and /usr/local for poudriere on the 2nd disk.
>> 
>> Use the same setup, but with ZFS.
> 
> I now have made tar-backups of /usr/local/poudiere und /usr/local
> (without) poudriere; I will umount the 2nd disk (da1) and restore
> /usr/local, which will end up in the 1st disk;
> 

Make sure to initialize poudriere properly on zfs (it will create various 
filesystems for jails/ports etc). Simply untarring won't cut it.


> about the ZFS I'm unsure; the poudriere.conf says:
> 
> ...
> #ZPOOL=tank
> ...
> # root of the poudriere zfs filesystem, by default /poudriere
> # ZROOTFS=/poudriere
> 
> Does this mean I should set:
> 
> ZPOOL=poudriere
> ZROOTFS=/poudriere
> 
> and do:
> 
> # zpool create poudriere /dev/da1
> 
> and restore the poudriere.tgz into this having
> 
> /poudriere/poudriere/data/
> /poudriere/poudriere/jails/
> /poudriere/poudriere/ports/
> 
> or only
> 
> /poudriere/data/
> /poudriere/jails/
> /poudriere/ports/
> 
> Thanks for clarification.
> 
>matthias
> 
> -- 
> Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎ 
> +49-176-38902045
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Kurt Jaeger]

Hi!

> > Use the same setup, but with ZFS.
> 
> I now have made tar-backups of /usr/local/poudiere und /usr/local
> (without) poudriere; I will umount the 2nd disk (da1) and restore
> /usr/local, which will end up in the 1st disk;
> 
> about the ZFS I'm unsure; the poudriere.conf says:
> 
> ...
> #ZPOOL=tank
> ...
> # root of the poudriere zfs filesystem, by default /poudriere
> # ZROOTFS=/poudriere
> 
> Does this mean I should set:
> 
> ZPOOL=poudriere
> ZROOTFS=/poudriere

I have

ZPOOL=pou
ZROOTFS=/pou

(I'm too lazy to type /poudriere all the time 8-)

> and do:
> 
> # zpool create poudriere /dev/da1
> 
> and restore the poudriere.tgz into this having
> 
> /poudriere/poudriere/data/
> /poudriere/poudriere/jails/
> /poudriere/poudriere/ports/
> 
> or only
> 
> /poudriere/data/
> /poudriere/jails/
> /poudriere/ports/

The last one is correct.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Matthias Apitz]

El día Saturday, March 04, 2017 a las 12:57:40PM +0100, Kurt Jaeger escribió:

> Hi!
> 
> > > Did you use ZFS as file system ?
> > 
> > When I did the disk setup, I was also thinking in ZFS, but this would have 
> > reduced the netto disk space from 2* 280 GB to the half.
> 
> It does not have to be that way. Use only one disk for the system
> and the second disk for a poudriere ZFS pool ?
> 
> > I have / for 
> > all.the system and /usr/local for poudriere on the 2nd disk.
> 
> Use the same setup, but with ZFS.

I now have made tar-backups of /usr/local/poudiere und /usr/local
(without) poudriere; I will umount the 2nd disk (da1) and restore
/usr/local, which will end up in the 1st disk;

about the ZFS I'm unsure; the poudriere.conf says:

...
#ZPOOL=tank
...
# root of the poudriere zfs filesystem, by default /poudriere
# ZROOTFS=/poudriere

Does this mean I should set:

ZPOOL=poudriere
ZROOTFS=/poudriere

and do:

# zpool create poudriere /dev/da1

and restore the poudriere.tgz into this having

/poudriere/poudriere/data/
/poudriere/poudriere/jails/
/poudriere/poudriere/ports/

or only

/poudriere/data/
/poudriere/jails/
/poudriere/ports/

Thanks for clarification.

matthias

-- 
Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎ 
+49-176-38902045
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Matthew D. Fuller]

On Sat, Mar 04, 2017 at 12:57:40PM +0100 I heard the voice of
Kurt Jaeger, and lo! it spake thus:
>
> /pou is on an Samsung SSD 840 EVO 250GB:
> 
> smartctl says:
> 
>   9 Power_On_Hours  0x0032   091   091   Always   -   41885
>  12 Power_Cycle_Count   0x0032   099   099   Always   -   8
> 177 Wear_Leveling_Count 0x0013   083   083   Always   -   199
> 
> The Wear_Leveling_Count shows that it's probably beyond the spec
> already 8-}

Actually, I'm pretty sure it means the opposite   8-}

It's my understanding that the "Raw" value on them is pretty much
roughly the actual count of levelled drive writes, and the cooked
"Value" (/worst, though that doesn't make a difference on that
attribute) winds up basically meaning "percent of rated life left"[0].
So you've got ~200 drive writes, and have used about 17% of its life.


On a 250GB 840 Evo I have, in somewhat easier use than being
poundriered senseless:

ID# ATTRIBUTE_NAME  FLAGSVALUE WORST THRESH FAIL RAW_VALUE
177 Wear_Leveling_Count PO--C-   095   095   000-57

It's in a mirror with a similarly sized MX100, which (with its
different attributes; can't be consistent, y'know) says

202 Percent_Lifetime_Used   P---CK   095   095   000-5

which agrees; I've only got 19/20ths of their lifespan to go before I
gotta cough up umpty-dozens of dollars to replace them.



[0] For most SMART attributes, the cooked 'value' actually starts at
100 and counts down, with the Threshold level being where it
starts claiming to be deeply unhappy.


-- 
Matthew Fuller (MF4839)   |  fulle...@over-yonder.net
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
   On the Internet, nobody can hear you scream.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Miroslav Lachman]

Kurt Jaeger wrote on 2017/03/04 12:57:


/pou is on an Samsung SSD 840 EVO 250GB:

smartctl says:

   9 Power_On_Hours  0x0032   091   091   Always   -   41885
  12 Power_Cycle_Count   0x0032   099   099   Always   -   8
177 Wear_Leveling_Count 0x0013   083   083   Always   -   199

The Wear_Leveling_Count shows that it's probably beyond the spec already 8-}


I don't think it's worn out. Wear Leveling starts at 100 and count down 
until there are 000.


# smartctl -A /dev/ada2 | egrep 'ID|Wear_Level|Power'
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE 
UPDATED  WHEN_FAILED RAW_VALUE
  9 Power_On_Hours  0x0032   094   094   000Old_age 
Always   -   28767
 12 Power_Cycle_Count   0x0032   099   099   000Old_age 
Always   -   6
177 Wear_Leveling_Count 0x0013   064   064   000Pre-fail  Always 
  -   1299


Miroslav Lachman
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Thomas Zander]

On 4 March 2017 at 13:20, Tommi Pernila  wrote:
> By default MAKE_JOBS is disabled to allow only one process per cpu. To
> allow it anyway, ALLOW_MAKE_JOBS=yes in your /usr/local/etc/poudriere.conf:
>
> echo "ALLOW_MAKE_JOBS=yes" >> /usr/local/etc/poudriere.conf

Consider allowing this only for selected ports, e.g. via setting
ALLOW_MAKE_JOBS_PACKAGES="pkg gcc* py* "

My machine tends to become close to unresponsive for (UI heavy tasks
like browsing) when both firefox and chromium builds run in parallel
with ALLOW_MAKE_JOBS.

Riggs
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Tommi Pernila]

On Mar 4, 2017 13:57, "Kurt Jaeger"  wrote:

Hi!

> > Did you use ZFS as file system ?
>
> When I did the disk setup, I was also thinking in ZFS, but this would have
> reduced the netto disk space from 2* 280 GB to the half.

It does not have to be that way. Use only one disk for the system
and the second disk for a poudriere ZFS pool ?

> I have / for
> all.the system and /usr/local for poudriere on the 2nd disk.

Use the same setup, but with ZFS.

I think, but have not checked it, that poudriere uses zfs clones,
which is ***way*** more efficient than copying the whole tree for
each poudriere builder jail.

> > SSD instead of spinning disks ?

> No. Seagate SAS disks.

3.5" or 2.5" ?

Putting the poudriere storage on a SSD might be very helpful.

This is my builder box (8 threads, Intel(R) Xeon(R) CPU E3-1230 V2 @
3.30GHz,
32 GB RAM):

zpool list
NAMESIZE  ALLOC   FREE  EXPANDSZ   FRAGCAP  DEDUP  HEALTH  ALTROOT
zroot   428G   216G   212G -47%50%  1.00x  ONLINE  -
pou 232G  49.9G   182G -39%21%  1.00x  ONLINE  -

/pou is on an Samsung SSD 840 EVO 250GB:

smartctl says:

  9 Power_On_Hours  0x0032   091   091   Always   -   41885
 12 Power_Cycle_Count   0x0032   099   099   Always   -   8
177 Wear_Leveling_Count 0x0013   083   083   Always   -   199

The Wear_Leveling_Count shows that it's probably beyond the spec already 8-}

--
p...@opsec.eu+49 171 3101372 3 years to
go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"


Have you checked poudriere configurations MAKE_JOBS parameter?
Here is a snippet from poudriere wiki:

By default MAKE_JOBS is disabled to allow only one process per cpu. To
allow it anyway, ALLOW_MAKE_JOBS=yes in your /usr/local/etc/poudriere.conf:

echo "ALLOW_MAKE_JOBS=yes" >> /usr/local/etc/poudriere.conf


Br,Tommi
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Kurt Jaeger]

Hi!

> > Did you use ZFS as file system ?
> 
> When I did the disk setup, I was also thinking in ZFS, but this would have 
> reduced the netto disk space from 2* 280 GB to the half.

It does not have to be that way. Use only one disk for the system
and the second disk for a poudriere ZFS pool ?

> I have / for 
> all.the system and /usr/local for poudriere on the 2nd disk.

Use the same setup, but with ZFS.

I think, but have not checked it, that poudriere uses zfs clones,
which is ***way*** more efficient than copying the whole tree for
each poudriere builder jail.

> > SSD instead of spinning disks ?

> No. Seagate SAS disks.

3.5" or 2.5" ?

Putting the poudriere storage on a SSD might be very helpful.

This is my builder box (8 threads, Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz,
32 GB RAM):

zpool list  
NAMESIZE  ALLOC   FREE  EXPANDSZ   FRAGCAP  DEDUP  HEALTH  ALTROOT
zroot   428G   216G   212G -47%50%  1.00x  ONLINE  -
pou 232G  49.9G   182G -39%21%  1.00x  ONLINE  -

/pou is on an Samsung SSD 840 EVO 250GB:

smartctl says:

  9 Power_On_Hours  0x0032   091   091   Always   -   41885
 12 Power_Cycle_Count   0x0032   099   099   Always   -   8
177 Wear_Leveling_Count 0x0013   083   083   Always   -   199

The Wear_Leveling_Count shows that it's probably beyond the spec already 8-}

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Matthias Apitz]

On Saturday, 4 March 2017 12:33:39 CET, Kurt Jaeger  wrote:




Did you use ZFS as file system ?


When I did the disk setup, I was also thinking in ZFS, but this would have 
reduced the netto disk space from 2* 280 GB to the half. I have / for 
all.the system and /usr/local for poudriere on the 2nd disk.




SSD instead of spinning disks ?



No. Seagate SAS disks.



--
Sent from my Ubuntu phone
http://www.unixarea.de/
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: running poudriere with 8 builders

[Kurt Jaeger]

Hi!

> During this time (00:09:20 - 00:33:47) the CPUs are idling and top
> shows 8x 'cpdup' (I do not see any man page for this, what proc is
> this?).
> 
> The overall throughput now is around 70 ports per hour. And top shows
> little CPU usage and a lot of pkg-static processes, perhaps for
> unpacking dependency packages. 
> 
> Any hints about how to improve the setup?

Did you use ZFS as file system ?

SSD instead of spinning disks ?

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

running poudriere with 8 builders

[Matthias Apitz]

Hello,

I acquired a server which my company sent to dust (called it
"decommissioning"), a Dell PowerEdge r210 with 8 CPU 3.3 GHz and 16 GByte
RAM. I have had to insert 2 SAS disks, each of 300 GByte.

The beast compiled CURRENT's world in an hour, and the kernel in 6
minutes, wow!

Now I setup poudriere and a ports tree in it, all from CURRENT too. I
started with 8 builders and poudriere needs halv an hour to start/clone
the builders:

root@jet:/usr/home/guru # poudriere bulk -f /usr/local/etc/poudriere-list -J 8 
-j freebsd-r314251 -p ports-20170303
[00:00:00] >> Creating the reference jail... done
[00:03:46] >> Mounting system devices for freebsd-r314251-ports-20170303
[00:03:46] >> Mounting ports/packages/distfiles
[00:03:46] >> Using packages from previously failed build
[00:03:46] >> Mounting packages from: 
/usr/local/poudriere/data/packages/freebsd-r314251-ports-20170303
[00:03:46] >> Copying /var/db/ports from: 
/usr/local/etc/poudriere.d/freebsd-r314251-options
[00:03:47] >> Appending to make.conf: 
/usr/local/etc/poudriere.d/freebsd-r314251-make.conf
/etc/resolv.conf -> 
/usr/local/poudriere/data/.m/freebsd-r314251-ports-20170303/ref/etc/resolv.conf
[00:03:47] >> Starting jail freebsd-r314251-ports-20170303
[00:03:48] >> Logs: 
/usr/local/poudriere/data/logs/bulk/freebsd-r314251-ports-20170303/2017-03-04_09h50m18s
[00:03:48] >> Loading MOVED
[00:04:06] >> Calculating ports order and dependencies
[00:09:17] >> pkg package missing, skipping sanity
[00:09:17] >> Skipping incremental rebuild and repository sanity checks
[00:09:17] >> Cleaning the build queue
[00:09:20] >> Recording filesystem state for prepkg... done
[00:09:20] >> Building 1760 packages using 8 builders
[00:09:20] >> Starting/Cloning builders
[00:33:47] >> Hit CTRL+t at any time to see build progress and stats
[00:33:47] >> [01][00:00:00] Starting build of ports-mgmt/pkg
[00:35:01] >> [01][00:01:14] Finished build of ports-mgmt/pkg: Success
...

During this time (00:09:20 - 00:33:47) the CPUs are idling and top
shows 8x 'cpdup' (I do not see any man page for this, what proc is
this?).

The overall throughput now is around 70 ports per hour. And top shows
little CPU usage and a lot of pkg-static processes, perhaps for
unpacking dependency packages. 

Any hints about how to improve the setup?

Thanks

matthias

-- 
Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎ 
+49-176-38902045
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"