from:"Martin Matuska"

Re: crash zfs_clone_range()

2023-11-10 Thread Martin Matuska


Hi Ronald,

hitting the panic with a DEBUG kernel would be great and it would be 
very nice if I could somehow reproduce the panic.
I have the option to rent an cheap arm64 virtual host at Hetzner so I 
could test that at an environment close to yours.


Please try compiling a GENERIC-DEBUG kernel with:

include GENERIC

ident GENERIC-DEBUG

options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options WITNESS_SKIPSPIN
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC
options DDB

Cheers,
mm

On 10. 11. 2023 11:12, Ronald Klop wrote:

Hi,

Had this crash today on RPI4/15-CURRENT.

FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #19 
main-b0203aaa46-dirty: Sat Nov  4 11:48:33 CET 2023 
ronald@rpi4:/home/ronald/dev/freebsd/obj/home/ronald/dev/freebsd/src/arm64.aarch64/sys/GENERIC-NODEBUG 
arm64


$ sysctl -a | grep bclon
vfs.zfs.bclone_enabled: 1

I started a jail with poudriere to build a package. The jail uses null 
mounts over ZFS.


[root]# cu -s 115200 -l /dev/cuaU0
Connected

db> bt
Tracing pid 95213 tid 100438 td 0xe1e97900
db_trace_self() at db_trace_self
db_stack_trace() at db_stack_trace+0x120
db_command() at db_command+0x2e4
db_command_loop() at db_command_loop+0x58
db_trap() at db_trap+0x100
kdb_trap() at kdb_trap+0x334
handle_el1h_sync() at handle_el1h_sync+0x18
--- exception, esr 0xf200
kdb_enter() at kdb_enter+0x48
vpanic() at vpanic+0x1dc
panic() at panic+0x48
data_abort() at data_abort+0x2fc
handle_el1h_sync() at handle_el1h_sync+0x18
--- exception, esr 0x9604
rms_rlock() at rms_rlock+0x1c
zfs_clone_range() at zfs_clone_range+0x68
zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x19c
null_bypass() at null_bypass+0x118
vn_copy_file_range() at vn_copy_file_range+0x18c
kern_copy_file_range() at kern_copy_file_range+0x36c
sys_copy_file_range() at sys_copy_file_range+0x8c
do_el0_sync() at do_el0_sync+0x634
handle_el0_sync() at handle_el0_sync+0x48
--- exception, esr 0x5600


Oh.. While typing this I rebooted the machine and it happened again. I 
didn't start anything in particular although the machine runs some jails.


x0: 0x00e0
  x1: 0xa00090317a48
  x2: 0xa000f79d4f00
  x3: 0xa000c61a44a8
  x4: 0xdeefe460 ($d.2 + 0xdd776560)
  x5: 0xa001250e4c00
  x6: 0xe54025b5 ($d.5 + 0xc)
  x7: 0x030a
  x8: 0xe1559000 ($d.2 + 0xdfdd1100)
  x9: 0x0001
 x10: 0x
 x11: 0x0001
 x12: 0x0002
 x13: 0x
 x14: 0x0001
 x15: 0x
 x16: 0x016dce88 (__stop_set_modmetadata_set + 0x1310)
 x17: 0x004e0d44 (rms_rlock + 0x0)
 x18: 0xdeefe280 ($d.2 + 0xdd776380)
 x19: 0x
 x20: 0xdeefe460 ($d.2 + 0xdd776560)
 x21: 0x7fff
 x22: 0xa00090317a48
 x23: 0xa000f79d4f00
 x24: 0xa001067ef910
 x25: 0x00e0
 x26: 0xa000158a8000
 x27: 0x
 x28: 0xa000158a8000
 x29: 0xdeefe280 ($d.2 + 0xdd776380)
  sp: 0xdeefe280
  lr: 0x01623564 (zfs_clone_range + 0x6c)
 elr: 0x004e0d60 (rms_rlock + 0x1c)
spsr: 0xa045
 far: 0x0108
 esr: 0x9604
panic: data abort in critical section or under mutex
cpuid = 1
time = 1699610885
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x38
vpanic() at vpanic+0x1a0
panic() at panic+0x48
data_abort() at data_abort+0x2fc
handle_el1h_sync() at handle_el1h_sync+0x18
--- exception, esr 0x9604
rms_rlock() at rms_rlock+0x1c
zfs_clone_range() at zfs_clone_range+0x68
zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x19c
null_bypass() at null_bypass+0x118
vn_copy_file_range() at vn_copy_file_range+0x18c
kern_copy_file_range() at kern_copy_file_range+0x36c
sys_copy_file_range() at sys_copy_file_range+0x8c
do_el0_sync() at do_el0_sync+0x634
handle_el0_sync() at handle_el0_sync+0x48
--- exception, esr 0x5600
KDB: enter: panic
[ thread pid 3792 tid 100394 ]
Stopped at  kdb_enter+0x48: str xzr, [x19, #768]
db>

I'll keep the debugger open for a while. Can I type something for 
additional info?


Regards,
Ronald.

Re: vfs.zfs.bclone_enabled (was: FreeBSD 14.0-BETA2 Now Available)

2023-09-17 Thread Martin Matuska


I vote for enabling block cloning on main :-)

mm

On 16. 9. 2023 19:14, Alexander Motin wrote:

On 16.09.2023 01:25, Graham Perrin wrote:

On 16/09/2023 01:28, Glen Barber wrote:

o A fix for the ZFS block_cloning feature has been implemented.


Thanks

I see 
, 
with 
 
in stable/14.


As vfs.zfs.bclone_enabled is still 0 (at least, with 15.0-CURRENT 
n265350-72d97e1dd9cc): should we assume that additional fixes, not 
necessarily in time for 14.0-RELEASE, will be required before 
vfs.zfs.bclone_enabled can default to 1?


I am not aware of any block cloning issues now.  All this thread about 
bclone_enabled actually started after I asked why it is still 
disabled. Thanks to Mark Millard for spotting this issue I could fix, 
but now we are back at the point of re-enabling it again.  Since the 
tunable does not even exist anywhere outside of FreeBSD base tree, I'd 
propose to give this code another try here too.  I see no point to 
have it disabled at least in main unless somebody needs time to run 
some specific tests first.

Re: main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics

2023-09-08 Thread Martin Matuska


Hi Alexander,

I can confirm that the patch fixes the panic caused by the provided 
script on my test systems.
Mark, would it be possible to try poudriere on your system with a 
patched kernel?


Thanks
mm

On 9. 9. 2023 0:09, Alexander Motin wrote:

On 08.09.2023 09:52, Martin Matuska wrote:
I digged a little and was able to reproduce the panic without 
poudriere with a shell script.


#!/bin/sh
nl='
'
sed_script=s/aaa/b/ 


for ac_i in 1 2 3 4 5 6 7; do
 sed_script="$sed_script$nl$sed_script"
done
echo "$sed_script" 2>/dev/null | sed 99q >conftest.sed

repeats=8
count=0
echo -n 0123456789 >"conftest.in"
while :
do
 cat "conftest.in" "conftest.in" >"conftest.tmp"
 mv "conftest.tmp" "conftest.in"
 cp "conftest.in" "conftest.nl"
 echo '' >> "conftest.nl"
 sed -f conftest.sed < "conftest.nl" >"conftest.out" 2>/dev/null 
|| break

 diff "conftest.out" "conftest.nl" >/dev/null 2>&1 || break
 count=$(($count + 1))
 echo "count: $count"
 # 10*(2^10) chars as input seems more than enough
 test $count -gt $repeats && break
done
rm -f conftest.in conftest.tmp conftest.nl conftest.out


Thank you, Martin.  I was able to reproduce the issue with your script 
and found the cause.


I first though the issue is triggered by the `cp`, but it appeared to 
be triggered by `cat`.  It also got copy_file_range() support, but 
later than `cp`.  That is probably why it slipped through testing.  
This patch fixes it for me: https://github.com/openzfs/zfs/pull/15251 .


Mark, could you please try the patch?

Re: main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics

2023-09-08 Thread Martin Matuska

I digged a little and was able to reproduce the panic without poudriere 
with a shell script.


You may want to increase "repeats".
The script causes the panic in dmu_buf_hold_array_by_dnode() on my 
VirtualBox with the cat command on 9th iteration.


Here is the script:

#!/bin/sh
nl='
'
sed_script=s/aaa/b/
for ac_i in 1 2 3 4 5 6 7; do
    sed_script="$sed_script$nl$sed_script"
done
echo "$sed_script" 2>/dev/null | sed 99q >conftest.sed

repeats=8
count=0
echo -n 0123456789 >"conftest.in"
while :
do
    cat "conftest.in" "conftest.in" >"conftest.tmp"
    mv "conftest.tmp" "conftest.in"
    cp "conftest.in" "conftest.nl"
    echo '' >> "conftest.nl"
    sed -f conftest.sed < "conftest.nl" >"conftest.out" 2>/dev/null || 
break

    diff "conftest.out" "conftest.nl" >/dev/null 2>&1 || break
    count=$(($count + 1))
    echo "count: $count"
    # 10*(2^10) chars as input seems more than enough
    test $count -gt $repeats && break
done
rm -f conftest.in conftest.tmp conftest.nl conftest.out


On 8. 9. 2023 6:32, Mark Millard wrote:

[Today's main-snapshot kernel panics as well.]

On Sep 7, 2023, at 16:32, Mark Millard  wrote:


On Sep 7, 2023, at 13:07, Alexander Motin  wrote:


Thanks, Mark.

On 07.09.2023 15:40, Mark Millard wrote:

On Sep 7, 2023, at 11:48, Glen Barber  wrote:

On Thu, Sep 07, 2023 at 11:17:22AM -0700, Mark Millard wrote:

When I next have time, should I retry based on a more recent
vintage of main that includes 969071be938c ?

Yes, please, if you can.

As stands, I rebooted that machine into my normal
enviroment, so the after-crash-with-dump-info
context is preserved. I'll presume lack of a need
to preserve that context unless I hear otherwise.
(But I'll work on this until later today.)
Even my normal environment predates the commit in
question by a few commits. So I'll end up doing a
more general round of updates overall.
Someone can let me know if there is a preference
for debug over non-debug for the next test run.

It is not unknown when some bugs disappear once debugging is enabled due to 
different execution timings, but generally debug may to detect the problem 
closer to its origin instead of looking on random consequences. I am only 
starting to look on this report (unless Pawel or somebody beat me on it), and 
don't have additional requests yet, but if you can repeat the same with debug 
kernel (in-base ZFS's ZFS_DEBUG setting follows kernel's INVARIANTS), it may 
give us some additional information.

So I did a zpool import, rewinding to the checkpoint.
(This depends on the questionable zfs doing fully as
desired for this. Notably the normal environment has
vfs.zfs.bclone_enabled=0 , including when it was
doing this activity.) My normal environment reported
no problems.

Note: the earlier snapshot from my first setup was
still in place since it was made just before the
original checkpoint used above.

However, the rewind did remove the /var/crash/
material that had been added.

I did the appropriate zfs mount.

I installed a debug kernel and world to the import. Again,
no problems reported.

I did the appropriate zfs umount.

I did the appropriate zpool export.

I rebooted with the test media.

# sysctl vfs.zfs.bclone_enabled
vfs.zfs.bclone_enabled: 1

# zpool trim -w zamd64

# zpool checkpoint zamd64

# uname -apKU
FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 150 #74 
main-n265188-117c54a78ccd-dirty: Tue Sep  5 21:29:53 PDT 2023 
root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-DBG
 amd64 amd64 150 150

(So, before the 969071be938c vintage, same sources as for
my last run but a debug build.)

# poudriere bulk -jmain-amd64-bulk_a -a
. . .
[00:03:23] Building 34214 packages using up to 32 builders
[00:03:23] Hit CTRL+t at any time to see build progress and stats
[00:03:23] [01] [00:00:00] Builder starting
[00:04:19] [01] [00:00:56] Builder started
[00:04:20] [01] [00:00:01] Building ports-mgmt/pkg | pkg-1.20.6
[00:05:33] [01] [00:01:14] Finished ports-mgmt/pkg | pkg-1.20.6: Success
[00:05:53] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1
[00:05:53] [02] [00:00:00] Builder starting
. . .
[00:05:54] [32] [00:00:00] Builder starting
[00:06:11] [01] [00:00:18] Finished print/indexinfo | indexinfo-0.3.1: Success
[00:06:12] [01] [00:00:00] Building devel/gettext-runtime | 
gettext-runtime-0.22_1
[00:08:24] [01] [00:02:12] Finished devel/gettext-runtime | 
gettext-runtime-0.22_1: Success
[00:08:31] [01] [00:00:00] Building devel/libtextstyle | libtextstyle-0.22
[00:10:06] [05] [00:04:13] Builder started
[00:10:06] [05] [00:00:00] Building devel/autoconf-switch | 
autoconf-switch-20220527
[00:10:06] [31] [00:04:12] Builder started
[00:10:06] [31] [00:00:00] Building devel/libatomic_ops | libatomic_ops-7.8.0
. . .

Crashed again, with 158 *.pkg files in .building/All/ after
rebooting.

The crash is similar to the non-debug one. No extra

Re: ZFS deadlock in 14

2023-08-22 Thread Martin Matuska


Hi Alexander,

as 15107 is a prerequisite for 15122,
would it be possible to have https://github.com/openzfs/zfs/pull/15107  
merged into the OpenZFS zfs-2.2-release branch (and of course later  
15122)?


If the patches help I can cherry-pick them into main.

Cheers,
mm


Alexander Motin  wrote:


On 17.08.2023 15:41, Dag-Erling Smørgrav wrote:

Alexander Motin  writes:

Trying to run your test (so far without reproduction) I see it
producing a substantial amount of ZIL writes.  The range of commits
you reduced the scope to so far includes my ZIL locking refactoring,
where I know for sure are some deadlocks.  I am already waiting for 3
weeks now for reviews and tests for PR that should fix it:
https://github.com/openzfs/zfs/pull/15122 .  It would be good if you
could test it, though it seems to depend on few more earlier patches
not merged to FreeBSD yet.


Do you have a FreeBSD branch with your patch applied?


I don't have a FreeBSD branch, but these two patches apply clean and  
build on top of today's FreeBSD main branch:


https://github.com/openzfs/zfs/pull/15107
https://github.com/openzfs/zfs/pull/15122

And if you still experience the issue, please show all stacks, or at  
least include ZFS sync threads.


--
Alexander Motin

Re: panic(s) in ZFS on CURRENT

2023-06-09 Thread Martin Matuska


I will wait with my prepared merge until #14954 gets merged.

On 9. 6. 2023 15:59, Alexander Motin wrote:

Hi Gleb,

There are two probably related PRs upstream:
https://github.com/openzfs/zfs/pull/14939
https://github.com/openzfs/zfs/pull/14954

On 09.06.2023 00:57, Gleb Smirnoff wrote:

On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote:
T> I'm switching to INVARIANTS kernel right now and will see if that 
panics earlier.


This is what I got with INVARIANTS:

panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed 
(225142071296 <= 225142063104)


cpuid = 17
time = 1686286015
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 
0xfe0160dcea90

kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe0160dceb40
vpanic() at vpanic+0x21f/frame 0xfe0160dcebe0
spl_panic() at spl_panic+0x4d/frame 0xfe0160dcec60
l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 
0xfe0160dcedf0

l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfe0160dceec0
fork_exit() at fork_exit+0x122/frame 0xfe0160dcef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfe0160dcef30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 1m4s
Dumping 5473 out of 65308 
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%


(kgdb) frame 4
#4  0x804342ea in l2arc_write_buffers 
(spa=0xfe022e942000, dev=0xfe023116a000, target_sz=16777216)

 at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445
9445    ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict);
(kgdb) p dev
$1 = (l2arc_dev_t *) 0xfe023116a000
(kgdb) p dev->l2ad_hand
$2 = 225142071296
(kgdb) p dev->l2ad_evict
$3 = 225142063104
(kgdb) p *dev
value of type `l2arc_dev_t' requires 66136 bytes, which is more than 
max-value-size


Never seen kgdb not being able to print a structure that reported to 
be too big.

Re: another crash and going forward with zfs

2023-04-18 Thread Martin Matuska


Btw. I am open for setting up a pre-merge stress testing

I will check out if I can use the hourly-billed amd64 and arm64 cloud 
boxes at Hetzner with FreeBSD.

Otherwise there are monthly-billed as well.

Cheers,
mm

On 17. 4. 2023 22:14, Mateusz Guzik wrote:

On 4/17/23, Pawel Jakub Dawidek  wrote:

On 4/18/23 03:51, Mateusz Guzik wrote:

After bugfixes got committed I decided to zpool upgrade and sysctl
vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very
quickly got a new crash:

panic: VERIFY(arc_released(db->db_buf)) failed

cpuid = 9
time = 1681755046
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0a90b8e5f0
vpanic() at vpanic+0x152/frame 0xfe0a90b8e640
spl_panic() at spl_panic+0x3a/frame 0xfe0a90b8e6a0
dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfe0a90b8e6c0
dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame
0xfe0a90b8e700
dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame
0xfe0a90b8e780
dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfe0a90b8e7b0
zfs_write() at zfs_write+0x672/frame 0xfe0a90b8e960
zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfe0a90b8e980
VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfe0a90b8ea90
vn_write() at vn_write+0x325/frame 0xfe0a90b8eb20
vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfe0a90b8eb80
vn_io_fault1() at vn_io_fault1+0x161/frame 0xfe0a90b8ecc0
vn_io_fault() at vn_io_fault+0x1b5/frame 0xfe0a90b8ed40
dofilewrite() at dofilewrite+0x81/frame 0xfe0a90b8ed90
sys_write() at sys_write+0xc0/frame 0xfe0a90b8ee00
amd64_syscall() at amd64_syscall+0x157/frame 0xfe0a90b8ef30
fast_syscall_common() at fast_syscall_common+0xf8/frame
0xfe0a90b8ef30
--- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp =
0x103cdc85dd48, rbp = 0x103cdc85dd80 ---
KDB: enter: panic
[ thread pid 95000 tid 135035 ]
Stopped at  kdb_enter+0x32: movq$0,0x9e4153(%rip)

The posted 14.0 schedule which plans to branch stable/14 on May 12 and
one cannot bet on the feature getting beaten up into production shape
by that time. Given whatever non-block_clonning and not even zfs bugs
which are likely to come out I think this makes the feature a
non-starter for said release.

I note:
1. the current problems did not make it into stable branches.
2. there was block_cloning-related data corruption (fixed) and there may
be more
3. there was unrelated data corruption (see
https://github.com/openzfs/zfs/issues/14753), sorted out by reverting
the problematic commit in FreeBSD, not yet sorted out upstream

As such people's data may be partially hosed as is.

Consequently the proposed plan is as follows:
1. whack the block cloning feature for the time being, but make sure
pools which upgraded to it can be mounted read-only
2. run ztest and whatever other stress testing on FreeBSD, along with
restoring openzfs CI -- I can do the first part, I'm sure pho will not
mind to run some tests of his own
3. recommend people create new pools and restore data from backup. if
restoring from backup is not an option, tar or cp (not zfs send) from
the read-only mount

block cloning beaten into shape would use block_cloning_v2 or whatever
else, key point that the current feature name would be considered
bogus (not blocking RO import though) to prevent RW usage of the
current pools with it enabled.

Comments?

Correct me if I'm wrong, but from my understanding there were zero
problems with block cloning when it wasn't in use or now disabled.

The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly
avoid mess like this and give us more time to sort all the problems out
while making it easy for people to try it.

If there is no plan to revert the whole import, I don't see what value
removing just block cloning will bring if it is now disabled by default
and didn't cause any problems when disabled.


The feature definitely was not properly stress tested and what not and
trying to do it keeps running into panics. Given the complexity of the
feature I would expect there are many bug lurking, some of which
possibly related to the on disk format. Not having to deal with any of
this is can be arranged as described above and is imo the most
sensible route given the timeline for 14.0

Re: another crash and going forward with zfs

2023-04-18 Thread Martin Matuska


On 18. 4. 2023 3:16, Warner Losh wrote:



Related question: what zfs branch is stable/14 going to track? With 13 
it was whatever the next stable branch was.


Warner

FreeBSD 14.0 is about to track soon-to-be-branched OpenZFS 2.2

Tachyum Prodigy processors

2022-06-13 Thread Martin Matuska


Hi everybody,

does anyone have any information regarding the new Tachyum Prodigy 
processors?


They have announced FreeBSD support as well as a porting kit of their 
instruction set architecture for FreeBSD:

https://www.tachyum.com/en-eu/media/press-releases/2022/04/05/tachyum-successfully-runs-freebsd-in-prodigy-ecosystem-expands-open-source-os-support/

They are also offering pre-orders for their evaluation platform:
https://www.tachyum.com/media/press-releases/2022/06/01/tachyum-begins-pre-orders-for-prodigy-evaluation-platform/

If their proclamations and statements about Prodigy's performance and 
features are right this might be the next game-changer in the server 
industry.


Cheers,
mm

Re: zpool upgrade can't enable new features

2021-02-25 Thread Martin Matuska


I have submitted a pull request to fix this in OpenZFS:
https://github.com/openzfs/zfs/pull/11653

On 25. 2. 2021 17:20, John Kennedy wrote:

On Thu, Feb 25, 2021 at 11:09:17AM -0300, Renato Botelho wrote:

I recently upgraded a CURRENT system to main-n244932-248a47a4c2f and
zpool status shows:

pool: zroot
   state: ONLINE
status: Some supported and requested features are not enabled on the pool.
  The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
  the pool may no longer be accessible by software that does not support
  the features. See zpool-features(5) for details.

   I noticed that the other day with main-n245037-6e822e99570f.

  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the 
pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not 
support
the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:10 with 0 errors on Mon Feb  1 
18:34:43 2021
config:

NAME   STATE READ WRITE CKSUM
zroot  ONLINE   0 0 0
  vtbd0p3.eli  ONLINE   0 0 0

   I didn't see anything at all that seemed missing.

zpool get all zroot | grep feature | sed -E 's/^([^ ]+)[ ]+([^ ]+)[ 
]+([^ ]+)[ ]+([^ ]+).*$/\1 \3 \4/' | sort | uniq -c
  12 zroot active local
  21 zroot enabled local


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: zpool upgrade can't enable new features

2021-02-25 Thread Martin Matuska


Looks like Ryan didn't think it all the way through in:
https://github.com/openzfs/zfs/commit/35ec51796f0aa8d4fe322b48e7d1d5a65e38a4ce

I am preparing a patch for OpenZFS.

On 25. 2. 2021 17:20, John Kennedy wrote:

On Thu, Feb 25, 2021 at 11:09:17AM -0300, Renato Botelho wrote:

I recently upgraded a CURRENT system to main-n244932-248a47a4c2f and
zpool status shows:

pool: zroot
   state: ONLINE
status: Some supported and requested features are not enabled on the pool.
  The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
  the pool may no longer be accessible by software that does not support
  the features. See zpool-features(5) for details.

   I noticed that the other day with main-n245037-6e822e99570f.

  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the 
pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not 
support
the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:10 with 0 errors on Mon Feb  1 
18:34:43 2021
config:

NAME   STATE READ WRITE CKSUM
zroot  ONLINE   0 0 0
  vtbd0p3.eli  ONLINE   0 0 0

   I didn't see anything at all that seemed missing.

zpool get all zroot | grep feature | sed -E 's/^([^ ]+)[ ]+([^ ]+)[ 
]+([^ ]+)[ ]+([^ ]+).*$/\1 \3 \4/' | sort | uniq -c
  12 zroot active local
  21 zroot enabled local


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libarchive update SVN r299529 breaks "ezjail update"

2016-05-14 Thread Martin Matuska

That switch is "--insecure" and is supported in all libarchive versions
freebsd ever used.


On 15.05.2016 01:36, Ngie Cooper (yaneurabeya) wrote:
>> On May 14, 2016, at 16:29, Martin Matuska <m...@freebsd.org> wrote:
>>
>> Ian, we are here talking about cpio, not libarchive. The flag in
>> libarchive is not active by default.
>>
>> On 14.05.2016 22:08, Ian Lepore wrote:
>>> The real damage will happen to out-of-tree users.  I think this will
>>> impact our software updater for $work for example, and it has to work
>>> with both old and new versions of libarchive, and now the new version
>>> will require a flag that the old version will reject as unknown.
>>>
>>> Ick.
> Ian’s comment was valid.. cpio doesn’t recognize the new switch on older 
> versions, so something like cpio `cpio --help | grep -- switch && echo 
> switch` would need to be employed everywhere for backwards compatibility — ew.
> Thanks,
> -Ngie

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libarchive update SVN r299529 breaks "ezjail update"

2016-05-14 Thread Martin Matuska

Ian, we are here talking about cpio, not libarchive. The flag in
libarchive is not active by default.


On 14.05.2016 22:08, Ian Lepore wrote:
> On Sat, 2016-05-14 at 15:51 -0400, michael butler wrote:
>>  From the looks of this, I think it's likely better to have the
>> default 
>> be "secure" and ezjail-admin use the "--insecure" flag as an explicit
>> override. That's the only place I've noticed the need for it although
>> I've not done an extensive search for any other instances in which it
>> might be required,
>>
>>  imb
>>
> The real damage will happen to out-of-tree users.  I think this will
> impact our software updater for $work for example, and it has to work
> with both old and new versions of libarchive, and now the new version
> will require a flag that the old version will reject as unknown.
>
> Ick.
>
> -- Ian
>
>> On 5/14/2016 3:46 PM, Tim Kientzle wrote:
>>> A little history about this issue:
>>>
>>> http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-2304
>>>
>>>
>>>> On May 14, 2016, at 12:17 PM, Tim Kientzle <t...@kientzle.com>
>>>> wrote:
>>>>
>>>> Many people consider the traditional behavior to be a security
>>>> risk, which is why this was changed.
>>>>
>>>> FreeBSD is welcome to make --insecure the default on FreeBSD, but
>>>> I'm reluctant to do that in the upstream libarchive project.
>>>>
>>>> Tim
>>>>
>>>>
>>>>> On May 12, 2016, at 8:54 AM, Martin Matuska <m...@freebsd.org>
>>>>> wrote:
>>>>>
>>>>> Looks like we have to remove line #174 from cpio/cpio.c:
>>>>> cpio->extract_flags |= ARCHIVE_EXTRACT_SECURE_NOABSOLUTEPATHS;
>>>>>
>>>>> This breaks traditional cpio behavior.
>>>>>
>>>>> Quoting Martin Matuska <m...@freebsd.org>:
>>>>>
>>>>>> Hi Michael, I have looked at the source and this is an
>>>>>> intended change in 3.2.0.
>>>>>>
>>>>>> An absolute path security check was added, cpio refuses to
>>>>>> extract or copy over absolute paths. To do this anyway the "-
>>>>>> -insecure" flag must be used.
>>>>>>
>>>>>> Here is the commit:
>>>>>> https://github.com/libarchive/libarchive/commit/59357157706d4
>>>>>> 7c365b2227739e17daba3607526
>>>>>>
>>>>>> Quoting Michael Butler <i...@protected-networks.net>:
>>>>>>
>>>>>>> It seems that today's libarchive update breaks cpio's
>>>>>>> behaviour:
>>>>>>>
>>>>>>> sudo ezjail-admin update -i -s /usr/src
>>>>>>>
>>>>>>> [ .. ]
>>>>>>>
>>>>>>> cd /usr/src/etc/..; install -o root -g wheel -m 444 
>>>>>>>  COPYRIGHT
>>>>>>> /usr/local/jails/fulljail/
>>>>>>> install -o root -g wheel -m 444
>>>>>>> /usr/src/etc/../sys/i386/conf/GENERIC.hints
>>>>>>> /usr/local/jails/fulljail/boot/device.hints
>>>>>>> /usr/local/jails/basejail/bincpio: bin: Path is absolute:
>>>>>>> Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/catcpio: bin/cat: Path is
>>>>>>> absolute:
>>>>>>> Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/chflagscpio: bin/chflags:
>>>>>>> Path is
>>>>>>> absolute: Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/chiocpio: bin/chio: Path is
>>>>>>> absolute:
>>>>>>> Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/chmodcpio: bin/chmod: Path is
>>>>>>> absolute:
>>>>>>> Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/cpcpio: bin/cp: Path is
>>>>>>> absolute: Unknown
>>>>>>> error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/datecpio: bin/date: Path is
>>>>>>> absolute:
>>>>>>> Unknown error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/ddcpio: bin/dd: Path is
>>>>>>> absolute: Unknown
>>>>>>> error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/dfcpio: bin/df: Path is
>>>>>>> absolute: Unknown
>>>>>>> error: -1
>>>>>>>
>>>>>>> /usr/local/jails/basejail/bin/domainnamecpio:
>>>>>>> bin/domainname: Path is
>>>>>>> absolute: Unknown error: -1
>>>>>>> [ .. etc. .. ]
>>>>>>
>>>>>>
>>>>>> Martin Matuska
>>>>>> FreeBSD committer
>>>>>> http://blog.vx.sk
>>>>>
>>>>>
>>>>> Martin Matuska
>>>>> FreeBSD committer
>>>>> http://blog.vx.sk
>> ___
>> freebsd-current@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "
>> freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libarchive update SVN r299529 breaks "ezjail update"

2016-05-12 Thread Martin Matuska


 Looks like we have to remove line #174 from cpio/cpio.c:
cpio->extract_flags |= ARCHIVE_EXTRACT_SECURE_NOABSOLUTEPATHS;

This breaks traditional cpio behavior.

Quoting Martin Matuska <m...@freebsd.org>:

Hi Michael, I have looked at the source and this is an intended  
change in 3.2.0.


An absolute path security check was added, cpio refuses to extract  
or copy over absolute paths. To do this anyway the "--insecure" flag  
must be used.


Here is the commit:
https://github.com/libarchive/libarchive/commit/59357157706d47c365b2227739e17daba3607526

Quoting Michael Butler <i...@protected-networks.net>:


It seems that today's libarchive update breaks cpio's behaviour:

sudo ezjail-admin update -i -s /usr/src

[ .. ]

cd /usr/src/etc/..; install -o root -g wheel -m 444  COPYRIGHT
/usr/local/jails/fulljail/
install -o root -g wheel -m 444
/usr/src/etc/../sys/i386/conf/GENERIC.hints
/usr/local/jails/fulljail/boot/device.hints
/usr/local/jails/basejail/bincpio: bin: Path is absolute: Unknown error: -1

/usr/local/jails/basejail/bin/catcpio: bin/cat: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/chflagscpio: bin/chflags: Path is
absolute: Unknown error: -1

/usr/local/jails/basejail/bin/chiocpio: bin/chio: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/chmodcpio: bin/chmod: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/cpcpio: bin/cp: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/datecpio: bin/date: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/ddcpio: bin/dd: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/dfcpio: bin/df: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/domainnamecpio: bin/domainname: Path is
absolute: Unknown error: -1
[ .. etc. .. ]




---------
Martin Matuska
FreeBSD committer
http://blog.vx.sk

--
Martin Matuska
FreeBSD committer
http://blog.vx.sk
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libarchive update SVN r299529 breaks "ezjail update"

2016-05-12 Thread Martin Matuska

 Hi Michael, I have looked at the source and this is an intended  
change in 3.2.0.


An absolute path security check was added, cpio refuses to extract or  
copy over absolute paths. To do this anyway the "--insecure" flag must  
be used.


Here is the commit:
https://github.com/libarchive/libarchive/commit/59357157706d47c365b2227739e17daba3607526

Quoting Michael Butler <i...@protected-networks.net>:


It seems that today's libarchive update breaks cpio's behaviour:

sudo ezjail-admin update -i -s /usr/src

[ .. ]

cd /usr/src/etc/..; install -o root -g wheel -m 444  COPYRIGHT
/usr/local/jails/fulljail/
install -o root -g wheel -m 444
/usr/src/etc/../sys/i386/conf/GENERIC.hints
/usr/local/jails/fulljail/boot/device.hints
/usr/local/jails/basejail/bincpio: bin: Path is absolute: Unknown error: -1

/usr/local/jails/basejail/bin/catcpio: bin/cat: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/chflagscpio: bin/chflags: Path is
absolute: Unknown error: -1

/usr/local/jails/basejail/bin/chiocpio: bin/chio: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/chmodcpio: bin/chmod: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/cpcpio: bin/cp: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/datecpio: bin/date: Path is absolute:
Unknown error: -1

/usr/local/jails/basejail/bin/ddcpio: bin/dd: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/dfcpio: bin/df: Path is absolute: Unknown
error: -1

/usr/local/jails/basejail/bin/domainnamecpio: bin/domainname: Path is
absolute: Unknown error: -1
[ .. etc. .. ]

--
Martin Matuska
FreeBSD committer
http://blog.vx.sk
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Prompt Live-CD/DVD with support for ZFS v.5000

2013-10-10 Thread Martin Matuska

I have updated the amd64 images with rsync without iconv linking.
Please re-download.

On 2013-10-10 23:12, Vladislav V. Prodan wrote:
 04.10.2013 16:56, Ollivier Robert wrote:
 According to Vladislav V. Prodan on Fri, Oct 04, 2013 at 01:48:16AM +0300:
 You want to add such a liveCD for automatic loading on PXE.
 MfsBSD built with ZFS v.28 :(
 mfsbsd will be updated soon I guess but in the meantime it is very easy to 
 generate your own.  Just get the code from github, modify a few config files 
 if needed and make.

 Thanks.

 In the latest build no libiconv.so.3

 root@mfsbsd:/tmp/oldpool/var/db/mysql # rsync -a
 /tmp/oldpool/var/db/mysql/ /tmp/newpool/var/db/mysql/
 Shared object libiconv.so.3 not found, required by rsync

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?

2013-06-29 Thread Martin Matuska

On 2013-06-29 12:01, Alexander Leidinger wrote:
 On Thu, 27 Jun 2013 23:58:33 +0200
 Kristof Provost kris...@sigsegv.be wrote:

 On 2013-06-24 22:08:01 (+0200), Alexander Leidinger
 alexan...@leidinger.net wrote:
 On Mon, 24 Jun 2013 12:15:18 +0200
 Kristof Provost kris...@sigsegv.be wrote:

 For what it's worth, I'm running into exactly the same problem.
 (amd64, stable/9). I have no custom settings in /etc/make.conf
 or /etc/src.conf
 I had a short discussion with the maintainer of our
 stress-test-suite, he was able to create a test-case which triggers
 the problem.

 I've been bisecting for a little bit, and while I'm not 100% sure yet,
 there is one likely culprit right now: r249643.
 It's an MFC with a number of ZFS changes relating to a refactoring of
 the ioctl() interface. 

 It is, unfortunately, also a rather large commit.
 Martin, the issue here is that starting a jail with a recent -current
 panics, if the jail has a dataset assigned to it during start (and
 the rc.d zfs scripts kicks in). At least in my case the jail contains an
 userland from before the change and the jail host a current userland.

 Any ideas / suggestions? pho@ has a test case for this.

 Bye,
 Alexander.

Hi Alexander,

some input would be great (at least the panic message - ideally from a
debug kernel).

Cheers,
mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?

2013-06-29 Thread Martin Matuska

This was an obvious error by me - I forgot to register zfs_ioc_jail and
zfs_ioc_unjail using the new functions.
Amazing noone noticed this by now until it got merged down to stable/8.

In addition, I see no need to log these operations to the zpool history
as they cause no on-disk changes, so I have disabled logging for these
calls.
Please test the patch from current in r252380.

http://svnweb.freebsd.org/base?view=revisionrevision=252380

On 29.6.2013 17:00, Alexander Leidinger wrote:
 On Sat, 29 Jun 2013 14:02:35 +0200
 Martin Matuska m...@freebsd.org wrote:

 some input would be great (at least the panic message - ideally from a
 debug kernel).
 The bt in the minidump is useless, but I made a bt directly
 in the kernel debugger:
 ---snip---
 Fatal trap 12: page fault while in kernel mode
 cpuid = 2; apic id = 02
 fault virtual address   = 0x0
 fault code  = supervisor read instruction, page not present
 instruction pointer = 0x20:0x0
 stack pointer   = 0x28:0xff839e79d930
 frame pointer   = 0x28:0xff839e79d9e0
 code segment= base 0x0, limit 0xf, type 0x1b
 = DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags= interrupt enabled, resume, IOPL = 0
 current process = 2183 (zfs)

 db bt  
 Tracing pid 2356
 uart_sab82532_class() at 0
 devfs_ioctl_f() at devfs_ioctl_f+0xf0
 kern_ioctl() at kern_ioctl+0x1d7
 sys_ioctl() at sys_ioctl+0x142
 ---snip---

 The test case is easy, create a dataset for a jail, add something like
 this to /etc/devfs.rules:
 ---snip---
 [devfsrules_unhide_zfs=12]
 add path zfs unhide

 [devfsrules_jail_withzfs=17]
 add include $devfsrules_hide_all
 add include $devfsrules_unhide_basic
 add include $devfsrules_unhide_login
 add include $devfsrules_unhide_zfs
 ---snip---

 Attach the dataset to the jail and see the box panicing on the use of
 the zfs command in the jail (maybe you don't even need to attach the
 dataset to the jail, maybe just the above devfs stuff is enough).

 The default jail scripts don't attach a ZFS dataset to a jail, the
 corresponding ezjail-script code is

   # Attach ZFS-datasets to the jail
   for zfs in ${ezjail_zfs_datasets}; do
 /sbin/zfs jail ${ezjail_id} ${zfs} || echo -n Error: ${zfs} could 
 not be configured
   done

 Bye,
 Alexander.



-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CRASH] ZFS recv (fwd)/CURRENT

2013-04-05 Thread Martin Matuska

You can use the attached patch, it should fix the problem.
We are still waiting for code review and a final solution by illumos,
maybe I will commit this preliminary (or final) fix to head.

mm

On 5.4.2013 16:49, Larry Rosenman wrote:
 On 2013-04-02 16:26, Martin Matuska wrote:
 On 1. 4. 2013 22:33, Martin Matuska wrote:
 This error seems to be limited to sending deduplicated streams. Does
 sending without -D work ok? This might be a vendor error as well.

 On 1.4.2013 20:05, Larry Rosenman wrote:
 Re-Sending.  Any ideas, guys/gals?

 This really gets in my way.

 This may be also related to:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=176978
 Taking off -D does get around the panic.

 What information can I provide to help fix it?

 I *CAN* provide access to both sides via SSH.




-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c
===
--- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c   (revision 
249165)
+++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c   (working copy)
@@ -990,6 +990,7 @@ free_guid_map_onexit(void *arg)
 
while ((gmep = avl_destroy_nodes(ca, cookie)) != NULL) {
dsl_dataset_long_rele(gmep-gme_ds, gmep);
+   dsl_dataset_rele(gmep-gme_ds, FTAG);
kmem_free(gmep, sizeof (guid_map_entry_t));
}
avl_destroy(ca);
@@ -1698,7 +1699,6 @@ add_ds_to_guidmap(const char *name, avl_tree_t *gu
gmep-gme_ds = snapds;
avl_add(guid_map, gmep);
dsl_dataset_long_hold(snapds, gmep);
-   dsl_dataset_rele(snapds, FTAG);
}
 
dsl_pool_rele(dp, FTAG);
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CRASH] ZFS recv (fwd)/CURRENT

2013-04-05 Thread Martin Matuska

This is a patch against -CURRENT, so the receiving side in your case.

On 5.4.2013 18:31, Larry Rosenman wrote:
 On 2013-04-05 11:29, Martin Matuska wrote:
 You can use the attached patch, it should fix the problem.
 We are still waiting for code review and a final solution by illumos,
 maybe I will commit this preliminary (or final) fix to head.

 Which side does this need to be on?

 (sending? (since it's dmu_send)?

 (which in my case is 8-STABLE)

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CRASH] ZFS recv (fwd)/CURRENT

2013-04-02 Thread Martin Matuska

On 1. 4. 2013 22:33, Martin Matuska wrote:
 This error seems to be limited to sending deduplicated streams. Does
 sending without -D work ok? This might be a vendor error as well.

 On 1.4.2013 20:05, Larry Rosenman wrote:
 Re-Sending.  Any ideas, guys/gals?

 This really gets in my way.

This may be also related to:
http://www.freebsd.org/cgi/query-pr.cgi?pr=176978

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CRASH] ZFS recv (fwd)/CURRENT

2013-04-01 Thread Martin Matuska

This error seems to be limited to sending deduplicated streams. Does
sending without -D work ok? This might be a vendor error as well.

On 1.4.2013 20:05, Larry Rosenman wrote:
 Re-Sending.  Any ideas, guys/gals?

 This really gets in my way.

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: ZFS problems

2013-03-01 Thread Martin Matuska

Fixed in r247540. If you are using a post-247265 zfs module please
install the new module first.

On 27.2.2013 22:56, Derrick Dantavious Edwards wrote:
   I updated sources a couple of days ago and when I rebooted to continue 
 to 
 the upgrade process I received errors when I attempted to mount zfs 
 filesystem. The error looked like this.

 zpool mount -a

 internal error: Invalid arugment
 pid 25 (zfs), uid 0, exited on signal 6
 Abort trap

 I get the same error if I just type zpool status -v, except the (zfs) is 
 replace with (zpool). The kernel module is loaded and visible by kldstat.
 Any ideas?
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r238860: bsdtar: eating up 100% CPU, hanging

2012-07-30 Thread Martin Matuska

Please check with CURRENT r238909. I have backported the NFSv4 ACL fix
for now.
It will be reverted and re-merged when fixed in libarchive's release branch.

On 30.7.2012 0:17, O. Hartmann wrote:
 Am 07/29/12 19:19, schrieb Martin Matuska:
 Do you still have this problem after r238882?

 The specific problem has gone - bsdtar now performs well. But another
 issue occurs now:

 Ports like mail/thunderbird or www/firefox which seems to install using
 bsdtar, install files with weird access bits set: --xr-xr-x
 I need to adjust the access bits manually. I do not know whether this is
 also libarchive related.
 I did not investigate further due to time constraints.

 Regards,
 Oliver

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r238860: bsdtar: eating up 100% CPU, hanging

2012-07-29 Thread Martin Matuska

I am also looking into it:

1. It happens only with libarchive 3.0.4 (3.0.3 works fine)
2. It happens only if archiving files located on ZFS (UFS works fine)
3. Backtrace:
#0  setup_acl_posix1e (a=0x801c45100, entry=0x801d69100, acl=0x801d8a000,
archive_entry_acl_type=256)
at
/base/head/lib/libarchive/../../contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c:474
#1  0x000800879036 in archive_read_disk_entry_from_file (_a=0x801c45100,
entry=0x801d69100, fd=6, st=Variable st is not available.
)
at
/base/head/lib/libarchive/../../contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c:434
#2  0x00080087831e in _archive_read_next_header2 (_a=0x801c45100,
entry=0x801d69100)
at
/base/head/lib/libarchive/../../contrib/libarchive/libarchive/archive_read_disk_posix.c:1070
#3  0x004078d5 in write_hierarchy (bsdtar=0x7fffd940,
a=0x801c17140, path=Variable path is not available.
)
at /base/head/usr.bin/tar/../../contrib/libarchive/tar/write.c:825
#4  0x00407d51 in write_archive (a=0x801c17140,
bsdtar=0x7fffd940)
at /base/head/usr.bin/tar/../../contrib/libarchive/tar/write.c:471
#5  0x00404fbd in main (argc=4, argv=Variable argv is not
available.
)
at /base/head/usr.bin/tar/../../contrib/libarchive/tar/bsdtar.c:668

This infinite loop has been fixed in libarchive master:
https://github.com/libarchive/libarchive/commit/d8b9dbd
d8b9dbd6dac0125957b997c2fe8d246237ec9f94

I suggest you backport to release also the following:
f67370d5c33b336b103c43fe35984defe7e0e473
https://github.com/libarchive/libarchive/commit/f67370d
c6d3cd33aecdc579966c3fbe7b9424cea83c7555
https://github.com/libarchive/libarchive/commit/c6d3cd3


Dňa 29. 7. 2012 3:18 Tim Kientzle  wrote / napísal(a):
 
 On Jul 28, 2012, at 10:21 AM, O. Hartmann wrote:
 
 When updating ports (like databases/sqlite3 or graphics/png via portmaster 
 graphics/png), the installation process comes to a point where a backup of 
 the old port is created with bsdtar. The process hangs then …
 
 My operating system is
 FreeBSD 10.0-CURRENT #0 r238860: Sat Jul 28 11:28:38 CEST 2012
 
 That's newer than my -CURRENT system here; I'm updating now.
 Martin imported a few changes from upstream just recently, so this
 is likely a new problem.
 
 What to do?
 
 Can you get the full command line for the command that's
 hanging?
 
 $ ps auxww | grep tar
 
 Knowing the exact options that were used will help narrow
 it down.
 
 Thanks for reporting it,
 
 Tim
 
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r238860: bsdtar: eating up 100% CPU, hanging

2012-07-29 Thread Martin Matuska

Do you still have this problem after r238882?

Dňa 28. 7. 2012 19:21 O. Hartmann  wrote / napísal(a):
 When updating ports (like databases/sqlite3 or graphics/png via
 portmaster graphics/png), the installation process comes to a point
 where a backup of the old port is created with bsdtar. The process hangs
 then:
 
 === Starting build for graphics/png ===
 
 === All dependencies are up to date
 
 
 === Creating a backup package for old version png-1.5.12
 load: 1.38  cmd: bsdtar 99286 [running] 1301.04r 1296.34u 0.00s 100% 5656k
 
 
 And a look on top:
 
 last pid:  3365;  load averages:  1.49,  1.44,  1.41
 up 0+04:39:08
 19:17:44
 65 processes:  2 running, 63 sleeping
 CPU: 50.4% user,  0.0% nice,  1.0% system,  0.0% interrupt, 48.6% idle
 Mem: 521M Active, 3599M Inact, 3424M Wired, 32M Cache, 826M Buf, 323M Free
 ARC: 1970M Total, 672M MRU, 1224M MFU, 48K Anon, 46M Header, 28M Other
 Swap: 32G Total, 32G Free
 
   PID USERNAME   THR PRI NICE   SIZERES STATE   C   TIME   WCPU
 COMMAND
 99286 root 1 1030 71724K  5672K CPU11  24:10 100.00%
 bsdtar
  1339 root 1  210  3221M 38008K select  1   3:02  1.71%
 Xorg
  3364 ohartmann   28  200   634M   301M uwait   1   0:06  0.63%
 thunderbird
   737 root 1  200 16520K  1492K select  0   0:42  0.00%
 moused
  3286 ohartmann   22  200   681M   368M uwait   1   0:14  0.00%
 firefox
  1469 ohartmann1  200 72364K 10612K select  1   0:05  0.00%
 xterm
 
 
 I can circumvent by doing a make reinstall in the port's directory, but
 this doesn't work for ports which copy files around using tar - like
 www/firefox and mail/thunderbird (which also get stuck when bsdtar is
 involved).
 
 My operating system is
  FreeBSD 10.0-CURRENT #0 r238860: Sat Jul 28 11:28:38 CEST 2012
 
 buildworld and kernel from today's sources, ports seem to be up to date,
 I updated everything successfully before installing the new world which
 seems to be faulty.
 
 I also recompiled usr.bin/tar separately and installed it, but without
 success.
 
 What to do?
 
 regards,
 
 Oliver
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: binutils-2.22: ld and --copy-dt-needed-entries

2011-12-06 Thread Martin Matuska

On 6.12.2011 17:48, Andriy Gapon wrote:
 Just for your information.
 It seems that ld from binutils-2.22 by default has --no-copy-dt-needed-entries
 behavior, and so explicit --copy-dt-needed-entries is now needed where the
 previous default behavior is relied upon.

 A short excerpt from the man page for your convenience:

 This option also has an effect on the resolution of symbols in
 dynamic libraries.  With --copy-dt-needed-entries dynamic libraries
 mentioned on the command line will be recursively searched,
 following their DT_NEEDED tags to other libraries, in order to
 resolve symbols required by the output binary.  With the default
 setting however the searching of dynamic libraries that follow it
 will stop with the dynamic library itself.  No DT_NEEDED links will
 be traversed to resolve symbols.
What do we do with this?
We can go back, patch to behave as before or to continue.
Are there any serious complaints?

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[PATCH] updated /etc/rc.d/jail (ZFS support, persistent jails and other features)

2011-11-08 Thread Martin Matuska

I have improved the jail etc script significantly (in addition to ZFS
support and other improvements).

- you can now set a jail_name= parameter to set the name for your jail
- the jails are still searched by name, so you cannot manage the jail
with the script if name in /etc/rc.conf changes while running.
- the status subcommand now also shows the number of running
processes, this way you can identify an empty jail
- there are also two new subcommands - create and remove, intended
for persistent jail operation
- if a jail is set to persistent, you can do the following sequence:
create start stop remove.
- non-persistent jails may also be created (won't be started) but will
be removed on a stop

http://people.freebsd.org/~mm/patches/jail/jail_etc.v2.patch
http://people.freebsd.org/~mm/patches/jail/jail_etc.v2.nowhitespace.patch

On 31. 7. 2011 0:32, Jamie Gritton wrote:
 Yes, that looks good.  It keeps what I'd call expected behavior for
 persist (at least on the startup side).

 - Jamie


 On 07/29/11 14:53, Martin Matuska wrote:
 So what do you think about this updated patch (attached)?
 Here we leave everything possible for jail_example_params.
 Btw. you can also set jid=xxx in params to have a static jail id for
 this jail.

 Also stopping a persistent jail doesn't delete it (but you cannot start
 it again).

 Dňa 28. 7. 2011 20:47, Jamie Gritton  wrote / napísal(a):
 Yes, it was intentional to move away from the global sysctls and to
 the per-jail parameters instead.  It makes more sense once config
 files come into play, which can do a better job of providing global
 defaults as well as per-jail parameters.

 The connection between ZFS and persist makes sense.  So for ZFS-based
 jail you'd want to set (and then reset) persist.  For others, this
 could be left to the user.  The changes to jail(8) for config files
 also sets persist when creating jails, and then clears it at a later
 stage unless the user specifies to keep it set.  It looks like I might
 want to add some ZFS support to the new jail(8).

 I would prefer to keep things simpler regarding create/start and
 remove/stop, and keep them tied together.

 - Jamie


 On 07/28/11 12:00, Martin Matuska wrote:
 If you start jail(8) witth -c (the new param way,) the values
 of the
 actual security.jail. variables are not initialized inside the jail,
 default values are used instead. I don't know if this is intentional,
 but probably yes. Default enforce_statfs=2, allow.mount=0.
 As of me we can leave everything for ${_params}, but then ${_zfs}
 makes
 sense only if enforce_statfs2 and allow.mount=1.

 Regarding zfs, if you want to operate zfs from the very start of a
 jail
 (and e.g. make use of /etc/rc.d/zfs which has jail support), you
 have to
 pair datasets with an existing jail. In simple words, you have to
 create
 a process-less jail (persist=1), attach zfs datasets and then run the
 command. The persist option can be made optional - but we always start
 with persist=1, then we can set (or not) persist=0 depending on user
 setting.

 The question that opens, should we remove a persisting jail on stop?
 Or should we support new commands create and remove in addition to
 start and stop? Create would just make a processless jail, remove
 would wipe out a jail and start/stop would just deal with the
 processes
 (if persist=0 the old way, of course)?

 Cheers,
 mm

 Dňa 28. 7. 2011 18:25, Jamie Gritton wrote / napísal(a):
 Since I missed the 9.0 boat with jail config file capability,
 something
 like this seems necessary; rc.d/jail has long been unable to
 handle the
 full scale of what jail(8) can do.

 I gather that setting persist is necessary for the ZFS operation. As
 long as we're making the parameter setting more generic from rc, we
 should handle the case where persist is specified in ${_params}, and
 not
 always set/reset it around the jail creation unless ZFS is used.

 Also, why the specific inclusion of the security-related parameters?
 They could just be folded into ${_params}, and if left unspecified
 then
 jail(8) should by default do the right thing.

 - Jamie


 On 07/28/11 08:11, Martin Matuska wrote:
 The attached patch allows better fine-tuning of jails started via
 /etc/rc.d, uses the new jail(8) flags (-c -m), the persist
 parameter and
 adds ZFS support.
 Patch is fully backward compatible.

 Please review, comment and/or test my attached patch.

 Cheers,
 mm


-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: makefs(8) broken iso9660 images

2011-08-10 Thread Martin Matuska

On 10. 8. 2011 16:54, Test Rat wrote:
 A quick example

   $ mkdir -p q/a q/b q/c q/d
   $ touch q/a/foo.c q/b/foo.c q/c/foo.c q/d/foo.c

   $ makefs -t cd9660 q.iso q
   $ tar tf q.iso
   .
   A
   B
   A/FOO.C
   B/FOO.C
   C
   D

   $ mkisofs -quiet -o q.iso q
   $ tar tf q.iso
   .
   A
   B
   C
   D
   D/FOO.C
   C/FOO.C
   B/FOO.C
   A/FOO.C

   $ makefs -t cd9660 inc.iso /usr/include
   $ tar tvvf inc.iso
   tar: Invalid location of extent of file
   Archive Format: ISO9660,  Compression: none
   tar: Error exit delayed from previous errors.

   $ mkisofs -quiet -o inc.iso /usr/include
   mkisofs: Symlink /usr/include/float.h ignored - continuing.
   mkisofs: Symlink /usr/include/syslog.h ignored - continuing.
   mkisofs: Symlink /usr/include/sched.h ignored - continuing.
   [...]
   $ tar tvvf inc.iso
   drwx--  54 0  0   12288 Aug 10 15:26 .
   drwx--  2 0  02048 Jun 14 01:40 NETATALK
   [...]
   -r  1 0  06324 Jun 14 01:40 GSSAPI/GSSAPI_K.H
   Archive Format: ISO9660,  Compression: none

 And for more real example grab a bootonly image from allbsd.org though
 official BETA1 would suffice, too, and try to extract kernel e.g.

   $ sha256 -q FreeBSD-9.0-HEAD-20110810-JPSNAP-bootonly.iso
   9b8beabe007f88f85f3fc59dd1b40ce212132dde173e03d4a93d48a5477989a4

   $ tar tf FreeBSD-9.0-HEAD-20110810-JPSNAP-bootonly.iso | fgrep -i kernel
   [nothing]
   $ mount -t cd9660 /dev/$(mdconfig -f 
 FreeBSD-9.0-HEAD-20110810-JPSNAP-bootonly.iso) /media
   $ ls -1 /media/boot/kernel
   aac.ko
   accf_data.ko
   accf_dns.ko
   accf_http.ko
   acpi_asus.ko
   acpi_dock.ko
   acpi_fujitsu.ko
   acpi_hp.ko
   acpi_ibm.ko
   acpi_panasonic.ko
   ^C

 And the following is probably known but doesn't happen with 8.2-RELEASE image.

   $ find /media/usr/include /dev/null
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/basic_tree_policy: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/binary_heap_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/binomial_heap_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/binomial_heap_base_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/bin_search_tree_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/cc_hash_table_map_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/eq_fn: Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/gp_hash_table_map_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/hash_fn: Input/output 
 error
   find: 
 /media/usr/include/c++/4.2/ext/pb_ds/detail/left_child_next_sibling_heap_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/list_update_map_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/list_update_policy: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/ov_tree_map_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/pairing_heap_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/pat_trie_: Input/output 
 error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/rb_tree_map_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/rc_binomial_heap_: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/resize_policy: 
 Input/output error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/splay_tree_: Input/output 
 error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/thin_heap_: Input/output 
 error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/tree_policy: Input/output 
 error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/trie_policy: Input/output 
 error
   find: /media/usr/include/c++/4.2/ext/pb_ds/detail/unordered_iterator: 
 Input/output error

 Am I alone in seeing this?

To resolve problems as quickly as possible, libarchive errors (tar)
should also be reported as issues to libarchive.googlecode.com, makefs
errors filed as NetBSD PR's.
The first two examples is libarchive with problems to read properly a
makefs-created ISO images.
The last example is bug completely fixed in r224762, going to be
commited to 8-stable and 7-stable soon.

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: bsdtar(1) can't extract new ISO images

2011-08-06 Thread Martin Matuska

The error is in FreeBSD ISO images.
They are created using makefs and that doesn't create ISO files that
strictly comple to the ECMA-119 (ISO9660 standard).

I have already filed a PR at NetBSD (bin/45217):
http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=45217

The volume_set_id doesn't have to be filled with 0x20 characters, too.
I am also preparing a patch for libarchive (different things), but that
won't fix that one bug - makefs needs to be fixed.

Dňa 3. 8. 2011 15:22, Test Rat wrote / napísal(a):
 It's often convenient to extract pieces of iso9660 images for recovery
 purposes or a jail. As libarchive no longer recognizes them one has to
 resort to mdconfig + mount_cd9660. On a zfs-only system this populates
 bufspace unused by arc cache and never gives memory back... nevermind.

   $ tar tf FreeBSD-9.0-BETA1-amd64-bootonly.iso
   $ cpio -ti FreeBSD-9.0-BETA1-amd64-bootonly.iso
   2 blocks
   $ tar tf FreeBSD-8.2-RELEASE-amd64-bootonly.iso
   .
   boot
   boot/zfs
   boot/firmware
   boot/kernel
   ^C
   $ cpio -ti FreeBSD-8.2-RELEASE-amd64-bootonly.iso
   .
   boot
   boot/zfs
   boot/firmware
   boot/kernel
   ^C

 I think it's also reproducable on daily snapshots from allbsd.org
 /stable/8 vs. /head. So, does this count as regression?
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: panic: share - excl @r224632

2011-08-05 Thread Martin Matuska

, fsflags, optlist);
 + error = vfs_domount_first(td, vfsp, vp, fspath, fsflags, 
 optlist);
   else
   error = vfs_domount_update(td, vp, fsflags, optlist);
   mtx_unlock(Giant);
  
 - ASSERT_VI_UNLOCKED(vp, __func__);
 - ASSERT_VOP_UNLOCKED(vp, __func__);
 -
   return (error);
  }
  

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: sys/kern/vfs_mount.c
===
--- sys/kern/vfs_mount.c	(revision 224654)
+++ sys/kern/vfs_mount.c	(working copy)
@@ -362,6 +362,64 @@ vfs_mergeopts(struct vfsoptlist *toopts, struct vf
 }
 
 /*
+ * Verify vnode's global path
+ */
+static int
+vfs_verify_global_path(struct thread *td, struct vnode *vp, char *fspath)
+{
+	struct nameidata nd;
+	struct vnode *vp1;
+	char *rpath, *fbuf;
+	int error;
+
+	ASSERT_VOP_ELOCKED(vp, __func__);
+
+	/* Construct global filesystem path from vp. */
+	VOP_UNLOCK(vp, 0);
+	error = vn_fullpath_global(td, vp, rpath, fbuf);
+	if (error != 0) {
+		vrele(vp);
+		return (error);
+	}
+	if (strlen(rpath) = MNAMELEN) {
+		vrele(vp);
+		free(fbuf, M_TEMP);
+		return (ENAMETOOLONG);
+	}
+	if ((vp-v_iflag  VI_DOOMED) != 0) {
+		vrele(vp);
+		free(fbuf, M_TEMP);
+		return (EBADF);
+	}
+
+	/*
+	 * Re-lookup the vnode by path. As a side effect, the vnode is
+	 * relocked.  If vnode was renamed, return ENOENT.
+	 */
+	NDINIT(nd, LOOKUP, FOLLOW | LOCKLEAF | MPSAFE | AUDITVNODE1,
+	UIO_SYSSPACE, fspath, td);
+	error = namei(nd);
+	if (error != 0) {
+		vrele(vp);
+		free(fbuf, M_TEMP);
+		return (error);
+	}
+	NDFREE(nd, NDF_ONLY_PNBUF);
+	vp1 = nd.ni_vp;
+	vrele(vp);
+	if (vp1 != vp) {
+		vput(vp1);
+		free(fbuf, M_TEMP);
+		return (ENOENT);
+	}
+
+	strlcpy(fspath,rpath,MNAMELEN);
+	free(fbuf, M_TEMP);
+
+	return (error);
+}
+
+/*
  * Mount a filesystem.
  */
 int
@@ -745,6 +803,7 @@ static int
 vfs_domount_first(
 	struct thread *td,		/* Calling thread. */
 	struct vfsconf *vfsp,		/* File system type. */
+	char *fspath,			/* Mount path. */
 	struct vnode *vp,		/* Vnode to be covered. */
 	int fsflags,			/* Flags common to all filesystems. */
 	struct vfsoptlist **optlist	/* Options local to the filesystem. */
@@ -753,25 +812,12 @@ vfs_domount_first(
 	struct vattr va;
 	struct mount *mp;
 	struct vnode *newdp;
-	char *fspath, *fbuf;
 	int error;
 
 	mtx_assert(Giant, MA_OWNED);
 	ASSERT_VOP_ELOCKED(vp, __func__);
 	KASSERT((fsflags  MNT_UPDATE) == 0, (MNT_UPDATE shouldn't be here));
 
-	/* Construct global filesystem path from vp. */
-	error = vn_fullpath_global(td, vp, fspath, fbuf);
-	if (error != 0) {
-		vput(vp);
-		return (error);
-	}
-	if (strlen(fspath) = MNAMELEN) {
-		vput(vp);
-		free(fbuf, M_TEMP);
-		return (ENAMETOOLONG);
-	}
-
 	/*
 	 * If the user is not root, ensure that they own the directory
 	 * onto which we are attempting to mount.
@@ -793,14 +839,12 @@ vfs_domount_first(
 	}
 	if (error != 0) {
 		vput(vp);
-		free(fbuf, M_TEMP);
 		return (error);
 	}
 	VOP_UNLOCK(vp, 0);
 
 	/* Allocate and initialize the filesystem. */
 	mp = vfs_mount_alloc(vp, vfsp, fspath, td-td_ucred);
-	free(fbuf, M_TEMP);
 	/* XXXMAC: pass to vfs_mount_alloc? */
 	mp-mnt_optnew = *optlist;
 	/* Set the mount level flags. */
@@ -1083,15 +1127,15 @@ vfs_domount(
 		mtx_lock(Giant);
 	NDFREE(nd, NDF_ONLY_PNBUF);
 	vp = nd.ni_vp;
-	if ((fsflags  MNT_UPDATE) == 0)
-		error = vfs_domount_first(td, vfsp, vp, fsflags, optlist);
-	else
+	if ((fsflags  MNT_UPDATE) == 0) {
+		error = vfs_verify_global_path(td, vp, fspath);
+		if (error == 0)
+			error = vfs_domount_first(td, vfsp, fspath, vp,
+			fsflags, optlist);
+	} else
 		error = vfs_domount_update(td, vp, fsflags, optlist);
 	mtx_unlock(Giant);
 
-	ASSERT_VI_UNLOCKED(vp, __func__);
-	ASSERT_VOP_UNLOCKED(vp, __func__);
-
 	return (error);
 }
 
@@ -1118,7 +1162,7 @@ unmount(td, uap)
 {
 	struct mount *mp;
 	struct nameidata nd;
-	char *pathbuf, *rpathbuf, *fbuf;
+	char *pathbuf;
 	int error, id0, id1;
 
 	AUDIT_ARG_VALUE(uap-flags);
@@ -1164,15 +1208,10 @@ unmount(td, uap)
 			UIO_SYSSPACE, pathbuf, td);
 			if (namei(nd) == 0) {
 NDFREE(nd, NDF_ONLY_PNBUF);
-if (vn_fullpath_global(td, nd.ni_vp, rpathbuf,
-fbuf) == 0) {
-	if (strlen(rpathbuf)  MNAMELEN) {
-		strlcpy(pathbuf, rpathbuf,
-		MNAMELEN);
-	}
-	free(fbuf, M_TEMP);
-}
-vput(nd.ni_vp);
+error = vfs_verify_global_path(td, nd.ni_vp,
+pathbuf);
+if (error == 0)
+	vput(nd.ni_vp);
 			}
 		}
 		mtx_lock(mountlist_mtx);
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: panic: share - excl @r224632

2011-08-05 Thread Martin Matuska

Patch updated.

On 05.08.2011 10:26, Kostik Belousov wrote:
 On Fri, Aug 05, 2011 at 10:18:49AM +0200, Martin Matuska wrote:
 I agree to Kostik's  approach, but I suggest implementing it in a
 separate function and also use for the unmount() part.

 Please review attached patch.
 Since you are moving the fragment to a function, you may somewhat reduce
 the code duplication by moving at least free() and return to the end
 of function and jumping to it.

 Also, after more looking at this, I think now that the check for VI_DOOMED
 is not needed, due to relookup and comparision of vp and vp1.

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: sys/kern/vfs_mount.c
===
--- sys/kern/vfs_mount.c	(revision 224654)
+++ sys/kern/vfs_mount.c	(working copy)
@@ -362,6 +362,58 @@
 }
 
 /*
+ * Verify vnode's global path
+ */
+static int
+vfs_verify_global_path(struct thread *td, struct vnode *vp, char *fspath)
+{
+	struct nameidata nd;
+	struct vnode *vp1;
+	char *rpath, *fbuf;
+	int error;
+
+	ASSERT_VOP_ELOCKED(vp, __func__);
+
+	/* Construct global filesystem path from vp. */
+	VOP_UNLOCK(vp, 0);
+	error = vn_fullpath_global(td, vp, rpath, fbuf);
+	if (error != 0) {
+		vrele(vp);
+		return (error);
+	}
+	if (strlen(rpath) = MNAMELEN) {
+		vrele(vp);
+		error = ENAMETOOLONG;
+		goto out;
+	}
+
+	/*
+	 * Re-lookup the vnode by path. As a side effect, the vnode is
+	 * relocked.  If vnode was renamed, return ENOENT.
+	 */
+	NDINIT(nd, LOOKUP, FOLLOW | LOCKLEAF | MPSAFE | AUDITVNODE1,
+	UIO_SYSSPACE, fspath, td);
+	error = namei(nd);
+	if (error != 0) {
+		vrele(vp);
+		goto out;
+	}
+	NDFREE(nd, NDF_ONLY_PNBUF);
+	vp1 = nd.ni_vp;
+	vrele(vp);
+	if (vp1 != vp) {
+		vput(vp1);
+		error = ENOENT;
+		goto out;
+	}
+
+	strlcpy(fspath,rpath,MNAMELEN);
+out:
+	free(fbuf, M_TEMP);
+	return (error);
+}
+
+/*
  * Mount a filesystem.
  */
 int
@@ -745,6 +797,7 @@
 vfs_domount_first(
 	struct thread *td,		/* Calling thread. */
 	struct vfsconf *vfsp,		/* File system type. */
+	char *fspath,			/* Mount path. */
 	struct vnode *vp,		/* Vnode to be covered. */
 	int fsflags,			/* Flags common to all filesystems. */
 	struct vfsoptlist **optlist	/* Options local to the filesystem. */
@@ -753,25 +806,12 @@
 	struct vattr va;
 	struct mount *mp;
 	struct vnode *newdp;
-	char *fspath, *fbuf;
 	int error;
 
 	mtx_assert(Giant, MA_OWNED);
 	ASSERT_VOP_ELOCKED(vp, __func__);
 	KASSERT((fsflags  MNT_UPDATE) == 0, (MNT_UPDATE shouldn't be here));
 
-	/* Construct global filesystem path from vp. */
-	error = vn_fullpath_global(td, vp, fspath, fbuf);
-	if (error != 0) {
-		vput(vp);
-		return (error);
-	}
-	if (strlen(fspath) = MNAMELEN) {
-		vput(vp);
-		free(fbuf, M_TEMP);
-		return (ENAMETOOLONG);
-	}
-
 	/*
 	 * If the user is not root, ensure that they own the directory
 	 * onto which we are attempting to mount.
@@ -793,14 +833,12 @@
 	}
 	if (error != 0) {
 		vput(vp);
-		free(fbuf, M_TEMP);
 		return (error);
 	}
 	VOP_UNLOCK(vp, 0);
 
 	/* Allocate and initialize the filesystem. */
 	mp = vfs_mount_alloc(vp, vfsp, fspath, td-td_ucred);
-	free(fbuf, M_TEMP);
 	/* XXXMAC: pass to vfs_mount_alloc? */
 	mp-mnt_optnew = *optlist;
 	/* Set the mount level flags. */
@@ -1083,15 +1121,15 @@
 		mtx_lock(Giant);
 	NDFREE(nd, NDF_ONLY_PNBUF);
 	vp = nd.ni_vp;
-	if ((fsflags  MNT_UPDATE) == 0)
-		error = vfs_domount_first(td, vfsp, vp, fsflags, optlist);
-	else
+	if ((fsflags  MNT_UPDATE) == 0) {
+		error = vfs_verify_global_path(td, vp, fspath);
+		if (error == 0)
+			error = vfs_domount_first(td, vfsp, fspath, vp,
+			fsflags, optlist);
+	} else
 		error = vfs_domount_update(td, vp, fsflags, optlist);
 	mtx_unlock(Giant);
 
-	ASSERT_VI_UNLOCKED(vp, __func__);
-	ASSERT_VOP_UNLOCKED(vp, __func__);
-
 	return (error);
 }
 
@@ -1118,7 +1156,7 @@
 {
 	struct mount *mp;
 	struct nameidata nd;
-	char *pathbuf, *rpathbuf, *fbuf;
+	char *pathbuf;
 	int error, id0, id1;
 
 	AUDIT_ARG_VALUE(uap-flags);
@@ -1164,15 +1202,10 @@
 			UIO_SYSSPACE, pathbuf, td);
 			if (namei(nd) == 0) {
 NDFREE(nd, NDF_ONLY_PNBUF);
-if (vn_fullpath_global(td, nd.ni_vp, rpathbuf,
-fbuf) == 0) {
-	if (strlen(rpathbuf)  MNAMELEN) {
-		strlcpy(pathbuf, rpathbuf,
-		MNAMELEN);
-	}
-	free(fbuf, M_TEMP);
-}
-vput(nd.ni_vp);
+error = vfs_verify_global_path(td, nd.ni_vp,
+pathbuf);
+if (error == 0)
+	vput(nd.ni_vp);
 			}
 		}
 		mtx_lock(mountlist_mtx);
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [PATCH] updated /etc/rc.d/jail and added ZFS support

2011-07-31 Thread Martin Matuska

Dňa 30. 7. 2011 17:29, Alexander Leidinger wrote / napísal(a):
 On Thu, 28 Jul 2011 16:11:37 +0200 Martin Matuska m...@freebsd.org
 wrote:


 The attached patch allows better fine-tuning of jails started via
 /etc/rc.d, uses the new jail(8) flags (-c -m), the persist parameter
 and adds ZFS support.
 Patch is fully backward compatible.

 Please review, comment and/or test my attached patch.
 Can you please have a look at the jail part of
 http://www.leidinger.net/FreeBSD/current-patches/etc:rc.d.diff and take
 some parts which you didn't take care about
 (jailname/securelevel/correctness check for fstab entries)?

 Bye,
 Alexander.

I have added the check for fstab entries to my patch. The
jailname/securelevel part is questionable. As to discussion with Jamie
Gritton (jamie@) we should go the jail_example_params way for as many
parameters as possible so we don't unnecessarily pollute rc.conf. This
is not possible for persist because it has to be set to 1 on creation
time for ZFS support.

This way a user can set something like:
jail_example_params=name=test securelevel=1 enforce_statfs=1 allow.mount=1

Patch available at:
http://people.freebsd.org/~mm/patches/jail/jail_etc.patch

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[PATCH] Updated jail mount/unmount patch

2011-07-29 Thread Martin Matuska

After implementing a suggestion from pjd@, a new version of the patch is
attached, now using a more universal solution - vn_fullpath_global() in
the mount part.

Dňa 28. 7. 2011 16:59, Martin Matuska wrote / napísal(a):
 Please review my attached patch.

 The patch fixes f_mntonname with mount/unmount inside a jail with allow.mount 
 enabled.
 Filesystems mountable in a jail require the VFCF_JAIL flag (currently only 
 ZFS).

 With this patch, mount and unmount works both with enforce_statfs = 0 and 
 enforce_statfs = 1.
 I suggest disabling mount/unmount for jails with enforce_statfs = 2, as this 
 is contradictory and does not play well with or without this patch.

 I have successfully tested this patch with ZFS, nullfs and tmpfs.

 To enable nullfs for a jail, you have to modify tmpfs/tmpfs_vfsops.c and 
 recompile the tmpfs module:
 -VFS_SET(tmpfs_vfsops, tmpfs, 0);
 +VFS_SET(tmpfs_vfsops, tmpfs, VFCF_JAIL);

 To enable tmpfs for a jail, you have to modify nullfs/null_vfsops.c and 
 recompile the nullfs module:
 -VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK);
 +VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK | VFCF_JAIL);

 The filesystems can be successfully mounted/unmounted inside a jail and also 
 unmounted from the parent host without problems.

 The mount inside jail, a jail needs allow.mount=1 and enforce.statfs=0 or 
 enforce.statfs=1, for more information see jail(8)
 I assume other filesystem not dealing with devices may work correctly with 
 this patch, too (e.g. nfs).

 With jailed nullfs we can run tinderbox in a jail ;)

 Please review, comment and/or test my attached patch.

 Cheers,
 mm
-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: sys/kern/kern_jail.c
===
--- sys/kern/kern_jail.c(revision 224471)
+++ sys/kern/kern_jail.c(working copy)
@@ -3858,7 +3858,8 @@
case PRIV_VFS_UNMOUNT:
case PRIV_VFS_MOUNT_NONUSER:
case PRIV_VFS_MOUNT_OWNER:
-   if (cred-cr_prison-pr_allow  PR_ALLOW_MOUNT)
+   if (cred-cr_prison-pr_allow  PR_ALLOW_MOUNT 
+   cred-cr_prison-pr_enforce_statfs != 2)
return (0);
else
return (EPERM);
Index: sys/kern/vfs_mount.c
===
--- sys/kern/vfs_mount.c(revision 224471)
+++ sys/kern/vfs_mount.c(working copy)
@@ -745,7 +745,6 @@
 vfs_domount_first(
struct thread *td,  /* Calling thread. */
struct vfsconf *vfsp,   /* File system type. */
-   char *fspath,   /* Mount path. */
struct vnode *vp,   /* Vnode to be covered. */
int fsflags,/* Flags common to all filesystems. */
struct vfsoptlist **optlist /* Options local to the filesystem. */
@@ -755,11 +754,23 @@
struct mount *mp;
struct vnode *newdp;
int error;
+   char *fspath, *fbuf;
 
mtx_assert(Giant, MA_OWNED);
ASSERT_VOP_ELOCKED(vp, __func__);
KASSERT((fsflags  MNT_UPDATE) == 0, (MNT_UPDATE shouldn't be here));
 
+   /* Construct global filesystem path from vp */
+   error = vn_fullpath_global(td, vp, fspath, fbuf);
+   if (error == 0  strlen(fspath) = MNAMELEN)
+   error = ENAMETOOLONG;
+   if (error != 0) {
+   vput(vp);
+   if (fbuf != NULL)
+   free(fbuf, M_TEMP);
+   return(error);
+   }
+
/*
 * If the user is not root, ensure that they own the directory
 * onto which we are attempting to mount.
@@ -781,12 +792,14 @@
}
if (error != 0) {
vput(vp);
+   free(fbuf, M_TEMP);
return (error);
}
VOP_UNLOCK(vp, 0);
 
/* Allocate and initialize the filesystem. */
mp = vfs_mount_alloc(vp, vfsp, fspath, td-td_ucred);
+   free(fbuf, M_TEMP);
/* XXXMAC: pass to vfs_mount_alloc? */
mp-mnt_optnew = *optlist;
/* Set the mount level flags. */
@@ -1070,7 +1083,7 @@
NDFREE(nd, NDF_ONLY_PNBUF);
vp = nd.ni_vp;
if ((fsflags  MNT_UPDATE) == 0) {
-   error = vfs_domount_first(td, vfsp, fspath, vp, fsflags,
+   error = vfs_domount_first(td, vfsp, vp, fsflags,
optlist);
} else {
error = vfs_domount_update(td, vp, fsflags, optlist);
@@ -1105,7 +1118,7 @@
} */ *uap;
 {
struct mount *mp;
-   char *pathbuf;
+   char *pathbuf, *rpathbuf;
int error, id0, id1;
 
AUDIT_ARG_VALUE(uap-flags);
@@ -1140,12 +1153,27 @@
mtx_unlock(mountlist_mtx);
} else {
AUDIT_ARG_UPATH1(td, pathbuf);
+   /*
+* If we are jailed and enforce_statfs=1
+* construct real filesystem

Re: [PATCH] updated /etc/rc.d/jail and added ZFS support

2011-07-29 Thread Martin Matuska

So what do you think about this updated patch (attached)?
Here we leave everything possible for jail_example_params.
Btw. you can also set jid=xxx in params to have a static jail id for
this jail.

Also stopping a persistent jail doesn't delete it (but you cannot start
it again).

Dňa 28. 7. 2011 20:47, Jamie Gritton  wrote / napísal(a):
 Yes, it was intentional to move away from the global sysctls and to
 the per-jail parameters instead.  It makes more sense once config
 files come into play, which can do a better job of providing global
 defaults as well as per-jail parameters.

 The connection between ZFS and persist makes sense.  So for ZFS-based
 jail you'd want to set (and then reset) persist.  For others, this
 could be left to the user.  The changes to jail(8) for config files
 also sets persist when creating jails, and then clears it at a later
 stage unless the user specifies to keep it set.  It looks like I might
 want to add some ZFS support to the new jail(8).

 I would prefer to keep things simpler regarding create/start and
 remove/stop, and keep them tied together.

 - Jamie


 On 07/28/11 12:00, Martin Matuska wrote:
 If you start jail(8) witth -c (the new param way,) the values of the
 actual security.jail. variables are not initialized inside the jail,
 default values are used instead. I don't know if this is intentional,
 but probably yes. Default enforce_statfs=2, allow.mount=0.
 As of me we can leave everything for ${_params}, but then ${_zfs} makes
 sense only if enforce_statfs2 and allow.mount=1.

 Regarding zfs, if you want to operate zfs from the very start of a jail
 (and e.g. make use of /etc/rc.d/zfs which has jail support), you have to
 pair datasets with an existing jail. In simple words, you have to create
 a process-less jail (persist=1), attach zfs datasets and then run the
 command. The persist option can be made optional - but we always start
 with persist=1, then we can set (or not) persist=0 depending on user
 setting.

 The question that opens, should we remove a persisting jail on stop?
 Or should we support new commands create and remove in addition to
 start and stop? Create would just make a processless jail, remove
 would wipe out a jail and start/stop would just deal with the processes
 (if persist=0 the old way, of course)?

 Cheers,
 mm

 Dňa 28. 7. 2011 18:25, Jamie Gritton wrote / napísal(a):
 Since I missed the 9.0 boat with jail config file capability, something
 like this seems necessary; rc.d/jail has long been unable to handle the
 full scale of what jail(8) can do.

 I gather that setting persist is necessary for the ZFS operation. As
 long as we're making the parameter setting more generic from rc, we
 should handle the case where persist is specified in ${_params}, and
 not
 always set/reset it around the jail creation unless ZFS is used.

 Also, why the specific inclusion of the security-related parameters?
 They could just be folded into ${_params}, and if left unspecified then
 jail(8) should by default do the right thing.

 - Jamie


 On 07/28/11 08:11, Martin Matuska wrote:
 The attached patch allows better fine-tuning of jails started via
 /etc/rc.d, uses the new jail(8) flags (-c -m), the persist
 parameter and
 adds ZFS support.
 Patch is fully backward compatible.

 Please review, comment and/or test my attached patch.

 Cheers,
 mm


-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: etc/rc.d/jail
===
--- etc/rc.d/jail   (revision 224494)
+++ etc/rc.d/jail   (working copy)
@@ -43,6 +43,9 @@
eval _ip=\\$jail_${_j}_ip\
eval _interface=\\${jail_${_j}_interface:-${jail_interface}}\
eval _exec=\\$jail_${_j}_exec\
+   eval _params=\\$jail_${_j}_params\
+   eval _persist=\\$jail_${_j}_persist\
+   eval _zfs=\\${jail_${_j}_zfs:-}\
 
i=0
while : ; do
@@ -98,6 +101,9 @@
fi
 
# The default jail ruleset will be used by rc.subr if none is specified.
+   if [ -n jail_devfs_ruleset -a -n _zfs ]; then
+   jail_devfs_ruleset=devfsrules_jail_zfs
+   fi
eval _ruleset=\\${jail_${_j}_devfs_ruleset:-${jail_devfs_ruleset}}\
eval _devfs=\\${jail_${_j}_devfs_enable:-${jail_devfs_enable}}\
[ -z ${_devfs} ]  _devfs=NO
@@ -345,6 +351,36 @@
mount -a -F ${_fstab}
 }
 
+# jail_zfs_jailin
+#  Make zfs datasets manageable from inside a jail
+#  the jailed dataset property must be set to on
+jail_zfs_jailin()
+{
+   if [ -n ${_zfs} ]; then
+   for _ds in ${_zfs}; do
+   _jailed=`zfs get -H jailed ${_ds} 2/dev/null | awk '{ 
print $3 }'`
+   if [ $_jailed = on ]; then
+   zfs jail ${_jail_id} ${_ds} 2/dev/null
+   fi
+   done
+   fi
+}
+
+# jail_zfs_jailout
+#  Unjail zfs datasets
+#  the jailed dataset property must be set

[PATCH] updated /etc/rc.d/jail and added ZFS support

2011-07-28 Thread Martin Matuska

The attached patch allows better fine-tuning of jails started via
/etc/rc.d, uses the new jail(8) flags (-c -m), the persist parameter and
adds ZFS support.
Patch is fully backward compatible.

Please review, comment and/or test my attached patch.

Cheers,
mm

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: etc/rc.d/jail
===
--- etc/rc.d/jail	(revision 224471)
+++ etc/rc.d/jail	(working copy)
@@ -43,6 +43,7 @@
 	eval _ip=\\$jail_${_j}_ip\
 	eval _interface=\\${jail_${_j}_interface:-${jail_interface}}\
 	eval _exec=\\$jail_${_j}_exec\
+	eval _params=\\$jail_${_j}_params\
 
 	i=0
 	while : ; do
@@ -83,6 +84,8 @@
 		i=$((i + 1))
 	done
 
+	eval _zfs=\\${jail_${_j}_zfs:-}\
+
 	if [ -n ${_exec} ]; then
 		#   simple/backward-compatible execution
 		_exec_start=${_exec}
@@ -98,6 +101,9 @@
 	fi
 
 	# The default jail ruleset will be used by rc.subr if none is specified.
+	if [ -n jail_devfs_ruleset -a -n _zfs ]; then
+		jail_devfs_ruleset=devfsrules_jail_zfs
+	fi
 	eval _ruleset=\\${jail_${_j}_devfs_ruleset:-${jail_devfs_ruleset}}\
 	eval _devfs=\\${jail_${_j}_devfs_enable:-${jail_devfs_enable}}\
 	[ -z ${_devfs} ]  _devfs=NO
@@ -200,6 +206,58 @@
 	if [ -z ${_rootdir} ]; then
 		err 3 $name: No root directory has been defined for ${_j}
 	fi
+
+	# Security-related parameters
+	eval _enforce_statfs=\\$jail_${_j}_enforce_statfs\
+	eval _allow_set_hostname=\\$jail_${_j}_allow_set_hostname\
+	eval _allow_sysvipc=\\$jail_${_j}_allow_sysvipc\
+	eval _allow_raw_sockets=\\$jail_${_j}_allow_raw_sockets\
+	eval _allow_chflags=\\$jail_${_j}_allow_chflags\
+	eval _allow_mount=\\$jail_${_j}_allow_mount\
+	eval _allow_socket_af=\\$jail_${_j}_allow_socket_af\
+	eval _allow_quotas=\\$jail_${_j}_allow_quotas:-0\
+
+	if [ -z ${_enforce_statfs} ]; then
+		_enforce_statfs=`${SYSCTL} -n security.jail.enforce_statfs`
+	fi
+
+	if [ -z ${_allow_set_hostname} ]; then
+		_allow_set_hostname=`${SYSCTL} -n security.jail.set_hostname_allowed`
+	fi
+
+	if [ -z ${_allow_sysvipc} ]; then
+		_allow_sysvipc=`${SYSCTL} -n security.jail.sysvipc_allowed`
+	fi
+
+	if [ -z ${_allow_raw_sockets} ]; then
+		_allow_raw_sockets=`${SYSCTL} -n security.jail.allow_raw_sockets`
+	fi
+
+	if [ -z ${_allow_chflags} ]; then
+		_allow_chflags=`${SYSCTL} -n security.jail.chflags_allowed`
+	fi
+
+	if [ -z ${_allow_mount} ]; then
+		_allow_mount=`${SYSCTL} -n security.jail.mount_allowed`
+	fi
+
+	if [ -z ${_allow_socket_af} ]; then
+		_tmpval=`${SYSCTL} -n security.jail.socket_unixiproute_only`
+		if [ ${_tmpval} = 0 ]; then
+			_allow_socket_af=1
+		else
+			_allow_socket_af=0
+		fi
+	fi
+
+	_security_params=enforce_statfs=${_enforce_statfs} \
+	allow.set_hostname=${_allow_set_hostname} \
+	allow.sysvipc=${_allow_sysvipc} \
+	allow.raw_sockets=${_allow_raw_sockets} \
+	allow.chflags=${_allow_chflags} \
+	allow.mount=${_allow_mount} \
+	allow.socket_af=${_allow_socket_af} \
+	allow.quotas=${allow_quotas}
 }
 
 # set_sysctl rc_knob mib msg
@@ -345,6 +403,36 @@
 	mount -a -F ${_fstab}
 }
 
+# jail_zfs_jailin
+#	Make zfs datasets manageable from inside a jail
+#	the jailed dataset property must be set to on
+jail_zfs_jailin()
+{
+	if [ -n ${_zfs} ]; then
+		for _ds in ${_zfs}; do
+			_jailed=`zfs get -H jailed ${_ds} 2/dev/null | awk '{ print $3 }'`
+			if [ $_jailed = on ]; then
+zfs jail ${_jail_id} ${_ds} 2/dev/null
+			fi
+		done
+	fi
+}
+
+# jail_zfs_jailout
+#	Unjail zfs datasets
+#	the jailed dataset property must be set to on
+jail_zfs_jailout()
+{
+	if [ -n ${_zfs} ]; then
+		for _ds in ${_zfs}; do
+			_jailed=`zfs get -H jailed ${_ds} 2/dev/null | awk '{ print $3 }'`
+			if [ $_jailed = on ]; then
+zfs unjail ${_jail_id} ${_ds} 2/dev/null
+			fi
+		done
+	fi
+}
+
 # jail_show_addresses jail
 #	Debug print the input for the given _multi aliases
 #	for a jail for init_variables().
@@ -483,10 +571,27 @@
 		*)	;;
 		esac
 
-		# Append address to list of addresses for the jail command.
-		case ${_addrl} in
-		)	_addrl=${_addr} ;;
-		*)	_addrl=${_addrl},${_addr} ;;
+		case ${_type} in
+		inet)
+			# Append address to list of ipv4 addresses for the
+			# jail command.
+			case ${_addrl} in
+			)	_addrl=${_addr} ;;
+			*)	_addrl=${_addrl},${_addr} ;;
+			esac
+			;;
+		inet6)
+			# Append address to list of ipv6 addresses for the
+			# jail command.
+			case ${_addrl6} in
+			)	_addrl6=${_addr} ;;
+			*)	_addrl6=${_addrl6},${_addr} ;;
+			esac
+			;;
+		*)	warn Could not determine address family.  Not going \
+			to set address '${_addr}' for ${_jail}.
+			continue
+			;;
 		esac
 
 		# Configure interface alias if requested by a given interface
@@ -494,14 +599,7 @@
 		case ${_iface} in
 		)	continue ;;
 		esac
-		case ${_type} in
-		inet)	;;
-		inet6)	;;
-		*)	warn Could not determine address family.  Not going \
-			to ${_action} address '${_addr}' for ${_jail}.
-			continue
-			;;
-		esac
+
 		case ${_action} in
 		add)	ifconfig ${_iface

[PATCH] jail mount/unmount patch

2011-07-28 Thread Martin Matuska

Please review my attached patch.

The patch fixes f_mntonname with mount/unmount inside a jail with allow.mount 
enabled.
Filesystems mountable in a jail require the VFCF_JAIL flag (currently only ZFS).

With this patch, mount and unmount works both with enforce_statfs = 0 and 
enforce_statfs = 1.
I suggest disabling mount/unmount for jails with enforce_statfs = 2, as this is 
contradictory and does not play well with or without this patch.

I have successfully tested this patch with ZFS, nullfs and tmpfs.

To enable nullfs for a jail, you have to modify tmpfs/tmpfs_vfsops.c and 
recompile the tmpfs module:
-VFS_SET(tmpfs_vfsops, tmpfs, 0);
+VFS_SET(tmpfs_vfsops, tmpfs, VFCF_JAIL);

To enable tmpfs for a jail, you have to modify nullfs/null_vfsops.c and 
recompile the nullfs module:
-VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK);
+VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK | VFCF_JAIL);

The filesystems can be successfully mounted/unmounted inside a jail and also 
unmounted from the parent host without problems.

The mount inside jail, a jail needs allow.mount=1 and enforce.statfs=0 or 
enforce.statfs=1, for more information see jail(8)
I assume other filesystem not dealing with devices may work correctly with this 
patch, too (e.g. nfs).

With jailed nullfs we can run tinderbox in a jail ;)

Please review, comment and/or test my attached patch.

Cheers,
mm

-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

Index: src/sys/kern/kern_jail.c
===
--- src/sys/kern/kern_jail.c	(revision 224297)
+++ src/sys/kern/kern_jail.c	(working copy)
@@ -3858,7 +3858,8 @@
 	case PRIV_VFS_UNMOUNT:
 	case PRIV_VFS_MOUNT_NONUSER:
 	case PRIV_VFS_MOUNT_OWNER:
-		if (cred-cr_prison-pr_allow  PR_ALLOW_MOUNT)
+		if (cred-cr_prison-pr_allow  PR_ALLOW_MOUNT 
+			cred-cr_prison-pr_enforce_statfs != 2)
 			return (0);
 		else
 			return (EPERM);
Index: src/sys/kern/vfs_mount.c
===
--- src/sys/kern/vfs_mount.c	(revision 224297)
+++ src/sys/kern/vfs_mount.c	(working copy)
@@ -1007,6 +1007,7 @@
 	struct vfsconf *vfsp;
 	struct nameidata nd;
 	struct vnode *vp;
+	char realfspath[MNAMELEN];
 	int error;
 
 	/*
@@ -1023,6 +1024,21 @@
 	}
 
 	/*
+	 * If we are jailed, construct real filesystem path
+	 */
+	if (jailed(td-td_ucred) 
+	strcmp(td-td_ucred-cr_prison-pr_path, /) != 0) {
+		if (strlen(td-td_ucred-cr_prison-pr_path) +
+		strlen(fspath) = MNAMELEN)
+			return (ENAMETOOLONG);
+		strlcpy(realfspath, td-td_ucred-cr_prison-pr_path,
+		sizeof(realfspath));
+		strlcat(realfspath, fspath, sizeof(realfspath));
+	} else {
+		strlcpy(realfspath, fspath, sizeof(realfspath));
+	}
+
+	/*
 	 * Do not allow NFS export or MNT_SUIDDIR by unprivileged users.
 	 */
 	if (fsflags  MNT_EXPORTED) {
@@ -1070,7 +1086,7 @@
 	NDFREE(nd, NDF_ONLY_PNBUF);
 	vp = nd.ni_vp;
 	if ((fsflags  MNT_UPDATE) == 0) {
-		error = vfs_domount_first(td, vfsp, fspath, vp, fsflags,
+		error = vfs_domount_first(td, vfsp, realfspath, vp, fsflags,
 		optlist);
 	} else {
 		error = vfs_domount_update(td, vp, fsflags, optlist);
@@ -1107,6 +1123,7 @@
 	struct mount *mp;
 	char *pathbuf;
 	int error, id0, id1;
+	char realfspath[MNAMELEN];
 
 	AUDIT_ARG_VALUE(uap-flags);
 	if (jailed(td-td_ucred) || usermount == 0) {
@@ -1139,10 +1156,23 @@
 		}
 		mtx_unlock(mountlist_mtx);
 	} else {
-		AUDIT_ARG_UPATH1(td, pathbuf);
+		/*
+		 * If we are jailed and enforce_statfs == 1
+		 * construct real filesystem path
+		 */
+		if (jailed(td-td_ucred) 
+		td-td_ucred-cr_prison-pr_enforce_statfs == 1 
+		strcmp(td-td_ucred-cr_prison-pr_path, /) != 0) {
+			strlcpy(realfspath, td-td_ucred-cr_prison-pr_path,
+			sizeof(realfspath));
+			strlcat(realfspath, pathbuf, sizeof(realfspath));
+		} else {
+			strlcpy(realfspath, pathbuf, sizeof(realfspath));
+		}
+		AUDIT_ARG_UPATH1(td, realfspath);
 		mtx_lock(mountlist_mtx);
 		TAILQ_FOREACH_REVERSE(mp, mountlist, mntlist, mnt_list) {
-			if (strcmp(mp-mnt_stat.f_mntonname, pathbuf) == 0)
+			if (strcmp(mp-mnt_stat.f_mntonname, realfspath) == 0)
 break;
 		}
 		mtx_unlock(mountlist_mtx);
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [PATCH] updated /etc/rc.d/jail and added ZFS support

2011-07-28 Thread Martin Matuska

If you start jail(8) witth -c (the new param way,) the values of the
actual security.jail. variables are not initialized inside the jail,
default values are used instead. I don't know if this is intentional,
but probably yes. Default enforce_statfs=2, allow.mount=0.
As of me we can leave everything for ${_params}, but then ${_zfs} makes
sense only if enforce_statfs2 and allow.mount=1.

Regarding zfs, if you want to operate zfs from the very start of a jail
(and e.g. make use of /etc/rc.d/zfs which has jail support), you have to
pair datasets with an existing jail. In simple words, you have to create
a process-less jail (persist=1), attach zfs datasets and then run the
command. The persist option can be made optional - but we always start
with persist=1, then we can set (or not) persist=0 depending on user
setting.

The question that opens, should we remove a persisting jail on stop?
Or should we support new commands create and remove in addition to
start and stop? Create would just make a processless jail, remove
would wipe out a jail and start/stop would just deal with the processes
(if persist=0 the old way, of course)?

Cheers,
mm

Dňa 28. 7. 2011 18:25, Jamie Gritton wrote / napísal(a):
 Since I missed the 9.0 boat with jail config file capability, something
 like this seems necessary; rc.d/jail has long been unable to handle the
 full scale of what jail(8) can do.

 I gather that setting persist is necessary for the ZFS operation. As
 long as we're making the parameter setting more generic from rc, we
 should handle the case where persist is specified in ${_params}, and not
 always set/reset it around the jail creation unless ZFS is used.

 Also, why the specific inclusion of the security-related parameters?
 They could just be folded into ${_params}, and if left unspecified then
 jail(8) should by default do the right thing.

 - Jamie


 On 07/28/11 08:11, Martin Matuska wrote:
 The attached patch allows better fine-tuning of jails started via
 /etc/rc.d, uses the new jail(8) flags (-c -m), the persist parameter and
 adds ZFS support.
 Patch is fully backward compatible.

 Please review, comment and/or test my attached patch.

 Cheers,
 mm
-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [PATCH] jail mount/unmount patch

2011-07-28 Thread Martin Matuska

For vfs_mount, the easiest way to look at this is to follow the path of
the realfspath argument.
It goes to vfs_domount_first() and ends up in vfs_mount_alloc() where is
the only place it is consumed:
strlcpy(mp-mnt_stat.f_mntonname, fspath, MNAMELEN);

The original fspath (not realfspath) is correctly consumed in vnode
initialization - vfs_domount(): NDINIT(...)
that ends in namei() of vfs_lookup.c - here we are processing jail paths
again so it gets translated properly and the correct vnode is picked.

We cannot write the supplied fspath to f_mntonname as this won't reflect
real path and we cannot unmount by name anymore
from host and from jail. If reading, f_mntonname gets translated by
prison_enforce_statfs so we should translate it correctly if writing so
that it matches the vnode.

We cannot add the check to vfs_mount_alloc() because we may blow MNAMELEN.
The check could be theoretically moved to vfs_domount_first() before mp
= vfs_mount_alloc(...) at the latest.

For the enforce_statfs privilege check: with enforce_statfs=2 you are
unable to unmount filesystems in a jail as they
are simply not found (mount works).

Cheers,
mm

Dňa 28. 7. 2011 18:12, Jamie Gritton wrote / napísal(a):
 There's a curious asymmetry here: enforce_statfs==1 is checked for
 munging the name on unmounting, but not on mounting. I see the point on
 the unmount side, as statfs would give the full un-jailed pathname and
 an admin would naturally want to unmount what he sees mounted, but
 without the same logic on the mount side, it would mean the unmount path
 is different from the mount path which would make fstab not use for
 mounting inside a jail. But then, enforce_statfs==0 is a strange world
 to be in anyway.

 I'm not sure about enforce_statfs!=2 in the privilege check. It seems a
 reasonable response to a contradictory set of permissions, but then so
 does the strange case if being able to mount a filesystem and then not
 being able to see it in statfs.

 - Jamie


 On 07/28/11 08:59, Martin Matuska wrote:
 Please review my attached patch.

 The patch fixes f_mntonname with mount/unmount inside a jail with
 allow.mount enabled.
 Filesystems mountable in a jail require the VFCF_JAIL flag (currently
 only ZFS).

 With this patch, mount and unmount works both with enforce_statfs = 0
 and enforce_statfs = 1.
 I suggest disabling mount/unmount for jails with enforce_statfs = 2,
 as this is contradictory and does not play well with or without this
 patch.

 I have successfully tested this patch with ZFS, nullfs and tmpfs.

 To enable nullfs for a jail, you have to modify tmpfs/tmpfs_vfsops.c
 and recompile the tmpfs module:
 -VFS_SET(tmpfs_vfsops, tmpfs, 0);
 +VFS_SET(tmpfs_vfsops, tmpfs, VFCF_JAIL);

 To enable tmpfs for a jail, you have to modify nullfs/null_vfsops.c
 and recompile the nullfs module:
 -VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK);
 +VFS_SET(null_vfsops, nullfs, VFCF_LOOPBACK | VFCF_JAIL);

 The filesystems can be successfully mounted/unmounted inside a jail
 and also unmounted from the parent host without problems.

 The mount inside jail, a jail needs allow.mount=1 and
 enforce.statfs=0 or enforce.statfs=1, for more information see jail(8)
 I assume other filesystem not dealing with devices may work correctly
 with this patch, too (e.g. nfs).

 With jailed nullfs we can run tinderbox in a jail ;)

 Please review, comment and/or test my attached patch.

 Cheers,
 mm
-- 
Martin Matuska
FreeBSD committer
http://blog.vx.sk

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-14 Thread Martin Matuska

After my recent patches to HEAD not anymore.
I have also a SSSE3 patch and a general gcc 4.2 update patch pending.

Dňa 12.03.2011 09:42, Jakub Lach  wrote / napísal(a):
 
 Core i7 based procesors run slower with -march=core2 (new option) on the
 system 
 compiler than with -march=nocona
 
 Sorry for double mail, isn't CPUTYPE=core2 just alias to nocona with base
 compiler?
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Martin Matuska

Hi Poul-Henning,

I have redone the test for majority of the processors, this time taking
5 samples of each whole testrun, calculating the average, standard
deviation, relative standard deviation, standard error and relative
standard error.

The relative standard error is below 0.25% for ~91%, between 0.25% and
0.5% for ~7%, 0.5%-1.0% for ~1% and between 1.0%-2.0% for 1% of the
tests. Under a test I mean 5 runs for the same setting of the same
compiler on the same preocessor.

So let's say I have now the string/base64 test for a core i7 showing the
following (score +/- standard deviation):
gcc421: 82.7892 points +/- 0.8314 (1%)
gcc45-nocona: 96.0882 points +/- 1.1652 (1.21%)

For a relative comparsion of two settings of the same test I could
calculate the difference of averages = 13.299 (16.06%) points and sum of
standard deviations = 2.4834 points (3.00%)

Therefore if assuming normal distribution intervals I could say that:
With a 95% probability gcc45-nocona is faster than gcc421 by at least
10.18% (16.06 - 1.96x3.00) or with a 99.9% probability by at least 6.12%
(16,06 - 3.2906x3.00).

So I should probably take a significance level (e.g. 95%, 99% or 99.9%)
and normalize all the test scores for this level. Results out of the
interval (difference is below zero) are then not significant.

What significance level should I take?

I hope this approach is better :)

Dňa 11.03.2011 17:46, Poul-Henning Kamp  wrote / napísal(a):
 In message 4d7a42cc.8020...@freebsd.org, Martin Matuska writes:
 
 But what I can say, e.g. for the Intel Atom processor, if there are
 performance gains in all but one test (that falls 2% behind), generic
 perl code (the routines benchmarked) on this processor is very likely to
 run faster with that setup.
 
 No, actually you cannot say that, unless you run all the tests at
 least three times for each compiler(+flag), calculate the average
 and standard deviation of all the tests, and see which, if any of
 the results are statistically significant.
 
 Until you do that, you numbers are meaningless, because we have no
 idea what the signal/noise ratio is.
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Martin Matuska

I don't take this personally and fully understand your point.

But even if all conditions you described are met, I am still not able to
say this is better as I am not doing a microbenchmark. The +x% score
is just an average of all test scores weightened by factor 1 - this does
not reflect any real application out there, as these applications don't
use the tested functions in that exact weighting ratio. If one function
had score 0%, the program actually would be stale forever when executing
this function but the score of this average would still look promising :-)

But what I can say, e.g. for the Intel Atom processor, if there are
performance gains in all but one test (that falls 2% behind), generic
perl code (the routines benchmarked) on this processor is very likely to
run faster with that setup.
On the other hand, if clang generated code falls short in all tests, I
can say it is very likely that it will run slower. But again, I am
benchmarking just a subset of generic perl functions.

Cheers,
mm

Dňa 11.03.2011 15:01, Poul-Henning Kamp wrote / napísal(a):
In message 4d7943b1.1030...@freebsd.org, Martin Matuska writes:

More information, detailed test results and test configuration are at
our blog:
http://blog.vx.sk/archives/25-FreeBSD-Compiler-Benchmark-gcc-base-vs-gcc-ports-vs-clang.html
Please don't take this personally Martin, but you have triggered
my periodic rant about proper running, evaluation and reporting of
benchmarks.

These results are not published at a level of detail that allows
anybody to draw any kind of conclusions from them.

In particular, your use of overall best result selection is totally
bogus from a statistical point of view.

At the very least, we need to see standard-deviations on your numbers,
and preferably, when you claim that X is N% better than Y, you should
also provide the confidence interval on that judgment, Student's T
being the canonical test.

The ministat(1) program does both of these things, and is now in
FreeBSD/src, so there is absolutely no excuse for not using it.

In practice this means that you have to run each test at least three
times, to get a standardeviation, and you have to make sure that
your testconditions are as identical as possible.

Therefore, proper benchmarking procedure is something like:

(boot machine single-user // Improves reproducibility)
(mount md(4)/malloc filesystem // ditto)
(newfs test-partition // ditto)
for at least 4 iterations:
run test A
run test B
run test C
...
Throw first result away for all tests
Run remaining results through ministat(1)

This was a public service announcement.

Poul-Henning

PS: Recommended reading: http://www.larrygonick.com/html/pub/books/sci7.html

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[TESTING] base gcc update to latest GPLv2 version

2011-03-10 Thread Martin Matuska

Here is a base gcc upgrade to the latest GPLv2 version (rev. 127959).

http://people.freebsd.org/~mm/patches/head-gcc-422-prerelease.patch

Open questions:
Do we want the 4.2.2 prerelase 20070831 version tag or stick to 4.2.1
20070831?

Testing and comments are welcome.

Originally suggested by Pedro F. Giffuni in:
http://www.freebsd.org/cgi/query-pr.cgi?pr=gnu/153298

Resolved GCC bugs in this patch:

c++:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=17763
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29365
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30535
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30917
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=31337
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=31941
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32108
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32112
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32346
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32898
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32992

debug:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32610
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32914

libstdc++:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=33128
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=33084

middle-end:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32563

rtl-optimization:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=33148

tree-optimization:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25413
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32723

target:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32218
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-10 Thread Martin Matuska

Hi everyone,

we have performed a benchmark of the perl binary compiled with base gcc,
ports gcc and ports clang using the perlbench benchmark suite.
Our benchmark was performed solely on amd64 with 10 different processors
and we have tried different -march= flags to compare binary performance
of the same compiler with different flags.

Here is some statistics from the results:
- clang falls 10% behind the base gcc 4.2.1 (test average)
- gcc 4.5 from ports gives 5-10% better average performance than the
base gcc 4.2.1
- 4% average penalty for Intel Atom and -march=nocona (using gcc from base)
- core i7 class processors run best with -march=nocona (using gcc from base)

This benchmark speaks only for perl, but it tests quite a lot of
generic features so we a are seriously considering using ports gcc for
heavily used ports (e.g. PHP, MySQL, PostgreSQL) and suggesting that an
user should be provided with a easily settable choice of using gcc 4.5
for ports.

A first step in this direction is in this PR (allowing build-only
dependency on GCC):
http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/155408

More information, detailed test results and test configuration are at
our blog:
http://blog.vx.sk/archives/25-FreeBSD-Compiler-Benchmark-gcc-base-vs-gcc-ports-vs-clang.html
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[TESTING] ssse3 backport from gcc 4.3

2011-03-09 Thread Martin Matuska

I have prepared a patch that finishes the core2 support part and
backports from gcc-4.3
the SSSE3 instruction set (-mssse3, -mno-ssse3).
It is enabled for -march=core2 by default.

Testing and comments are welcome.

Patch:
http://people.freebsd.org/~mm/patches/head-gcc-ssse3.patch

The backport covers three GPLv2 revisions from gcc 4.3:
http://gcc.gnu.org/viewcvs?view=revisionrevision=117958 (applies cleanly)
http://gcc.gnu.org/viewcvs?view=revisionrevision=121687 (small adjustment)
http://gcc.gnu.org/viewcvs?view=revisionrevision=121726 (small adjustment)
http://gcc.gnu.org/viewcvs?view=revisionrevision=123639 (small adjustment)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r219385 build error.

2011-03-07 Thread Martin Matuska

This actually does not happen at the bootstrap stage, but when
building 32-bit compat libs under amd64. It looks like the system
compiler is used here instead (should it be this way, isn't it a bug
somewhere around Makefile.inc1?).

Yes, building + installing world without this optimization makes it
work again (you can build the whole world).

And generally, I am putting the LIB32CPUFLAGS in question, why are
we using here a 64-bit cpu type at all? In bsd.cpu.mk we map nocona and
core2 to prescott for i386.


Dňa 07.03.2011 22:29, Kostik Belousov  wrote / napísal(a):
 On Mon, Mar 07, 2011 at 11:19:40PM +0200, George Liaskos wrote:
 What process did you follow to get here?
 I did a make toolchain followed by make buildworld.

 that's because the latest gcc commits have support for core2 and thus it no
 longer is being expanded to nocona. please note that having core2 in 
 make.conf
 has always been *wrong*. hence the need to reset it to nocona.
 the best way to fix this would be to set CPUYTYPE?=native. if you want core2
 support now's the chance to actually get it. just update world and you can 
 use
 CPUTYPE?=core2 and this time it *really* is supported. ;)
 I saw the relevant commits about core2, this is the reason i decided
 to do a rebuild.
 I didn't know that core2 was wrong, it's in the make.conf
 documentation, native it's not and after serious googling i found
 out that i should actually avoid it.

 I always believed that core2 was there [make.conf] as a future proof
 upgrade path for when the base toolchain actually supports core2.

 So, should i use native cputype?
 You did not shown the actual point where the error was raised.
 Applying some psychic powers, I could guess that it happens at the
 bootstrap stage. And this would be reasonable indeed, since bootstrap
 needs to use the system compiler, until the new cross toolchain is
 ready. And obviously system compiler not yet supports -march=core2,
 since you are only compiling the code that supports.

 Of course, all this assuming that error indeed happens at bootstrap,
 and the referenced commit does not introduce regressions, which I
 think is the case.

 I believe the solution for you would be to remove any CPU model settings
 from make.conf, make and install new world, then try new buildworld
 with desired settings. As a side note, I do not believe that you would
 get any measurable changes.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r219385 build error.

2011-03-07 Thread Martin Matuska

This change did NOT add SSSE3 or any other new instruction sets to our
base compiler.

The only change of using -march=core2 vs -march=nocona is actually
different instruction costs that may result
in binaries more optimized for your core2 and later CPUs (and less
optimized for nocona and earlier CPUs - but they will run there if the
CPU supports sse3).

We support newer instruction sets for base compiling starting with the
latest base binutils upgrade and
that is available only in CURRENT.

I might take a look at the possibility of backporting SSSE3, but that is
a way more intrusive change than this one. I will also run some more
benchmarks.

Dňa 08.03.2011 00:14, George Liaskos  wrote / napísal(a):
 native doesn't get handled by bsd.cpu.mk at all! it gets passed to gcc
 directly and gcc choses -m{tune,arch} on it's own.

 don't add -march=* directly to CFLAGS. this is bound to go wrong at some
 point. use CPUTYPE to set the cpu and CFLAGS for -O*, -pipe, etc.

 also please keep in mind that the optimisations that can be achieved by
 finetuning make.conf are rather minor. some people think that with
 cflags and cpu juju they can boost the OS. i don't believe that's true. the
 chances are much greater that you're adding a problematic switch and end up
 with binaries during installworld that segfault. so it's not really worth
 getting into this kinda trouble just for the sake of optimisation.

 a simple

 CPUTYPE ?= native
 COPTFLAGS = -O0 -pipe
 CFLAGS = -O2 -pipe

 should be close to perfekt. ;)

 cheers.
 alex
 Thank you again.

 It's not so much about the base system but the ports. Now that the
 assembler and binutils support newer SIMD commands it makes sense to
 exploit them, I know that they are not used during kernel compilation.
 Using newer / different compiler from ports makes things more complicated.
 .
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Next ZFSv28 patchset ready for testing.

2010-12-14 Thread Martin Matuska

Thanks for the notice.

I have found the cause of this error (wrong constants), tested the code
in both directions again (v15-v28 and v28-v15) + fixed it in perforce.

Bugfix patch (apply after pjd's patch):
http://people.freebsd.org/~mm/patches/zfs/v28/head-zfs_ioctl_compat.c.patch

Dňa 14.12.2010 16:44, Pawel Jakub Dawidek  wrote / napísal(a):
 On Tue, Dec 14, 2010 at 03:20:05PM +0100, Olivier Smedts wrote:
 make installworld
 That's what I wanted to do, and why I rebooted single-user on the new
 kernel. But isn't the v13-v15 userland supposed to work with the v28
 kernel ?
 Yes, it is suppose to work. Exactly to be able to follow FreeBSD common
 upgrade path. Martin was working on this (CCed).

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: ZFS v28 is ready for wider testing.

2010-09-06 Thread Martin Matuska

 Hi everyone,
I have put together a slightly improved patch of Pawel's that compiles
correctly and supports booting from ZFS v19 pools.

You can download the patch here:
http://people.freebsd.org/~mm/patches/zfs/head-zfs-v28-20100831.patch

For users who don't want to compile I have created a mfsBSD ISO image
with a zfs-only install of 9-CURRENT-amd64:
http://mfsbsd.vx.sk/iso/mfsbsd-se-head-zfsv28-amd64.iso

You can read more about all of this here:
http://blog.vx.sk/archives/9-Help-testing-ZFS-v28-in-FreeBSD.html

More information about ZFS pool and filesystem versions:
http://blog.vx.sk/archives/4-ZFS-pool-and-filesystem-versions.html

Thanks to everybody participating in testing,
mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: ZFS v28 is ready for wider testing.

2010-09-06 Thread Martin Matuska

To avoid user and developer confusion, my patch was just a chain  of
pjd's patch + pjd's atomic.h fix + my v19 boot patch.

I have removed (= split) the chained patch in my posting and altered my
blog article with updated build instructions that actually just
summarize what has been written on this list.

Thanks for understanding!

Dňa 6. 9. 2010 13:50, Martin Matuska  wrote / napísal(a):
  Hi everyone,
 I have put together a slightly improved patch of Pawel's that compiles
 correctly and supports booting from ZFS v19 pools.
 
 You can download the patch here:
 http://people.freebsd.org/~mm/patches/zfs/head-zfs-v28-20100831.patch
 
 For users who don't want to compile I have created a mfsBSD ISO image
 with a zfs-only install of 9-CURRENT-amd64:
 http://mfsbsd.vx.sk/iso/mfsbsd-se-head-zfsv28-amd64.iso
 
 You can read more about all of this here:
 http://blog.vx.sk/archives/9-Help-testing-ZFS-v28-in-FreeBSD.html
 
 More information about ZFS pool and filesystem versions:
 http://blog.vx.sk/archives/4-ZFS-pool-and-filesystem-versions.html
 
 Thanks to everybody participating in testing,
 mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[CFT] ZFS ACL and rrwlock speedup

2010-08-24 Thread Martin Matuska

Dear FreeBSD community,

in my last CFT I presented a patch that improves ZFS write speed. There
have been easily portable improvements on the read side in OpenSolaris,
too. Of course, the improvement here is by far not that dramatic as in
the zfs_metaslab.patch, but OpenSolaris developers claim this also
improves the speed of stat() calls.

I would like to Call For Testing for the following patch (9-CURRENT):
http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch

This patch adds the following:
- better ACL caching and speedup of the ACL permission checks
- lowered mutex contention in the read/writer lock (rrwlock)
- several bugfixes

This patch does not interfere with the zfs_metaslab.patch

To apply the patch against 8-STABLE, you need to apply the v15 update first:
http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch

The patch imports the following OpenSolaris onnv revisions:
9749 (without zlook), 9866, 9981, 10143, 10232, 10295, 10250, 10269

And covers the following OpenSolaris Bug IDs:
6802734 Support for Access Based Enumeration (not used on FreeBSD)
6844861 inconsistent xattr readdir behavior with too-small buffer
6848431 zfs with rstchown=0 or file_chown_self privilege allows user to
take ownership
6775100 stat() performance on files on zfs should be improved
6827779 rrwlock is overly protective of its counters
6857433 memory leaks found at: zfs_acl_alloc/zfs_acl_node_alloc
6860318 truncate() on zfsroot succeeds when file has a component of its
path set without access permission
6865875 zfs sometimes incorrectly giving search access to a dir
6870564 panic in zfs_getsecattr
6867395 zpool_upgrade_007_pos testcase panic'd with BAD TRAP: type=e
(#pf Page fault)
6868276 zfs_rezget() can be hazardous when znode has a cached ACL

Cheers,
mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[CFT] Improved ZFS metaslab code (faster write speed)

2010-08-22 Thread Martin Matuska

Dear FreeBSD community,

many of our [2] (and Solaris [3]) users today are complaining about slow
ZFS writes. One of the causes for these writes is the selection of the
proper allocation method for allocation of new blocks [3] [4]. Another
issue a write slowdown during TXG sync times.

Solaris 10 (and OpenSolaris up to november 2009) have the
following scenario:

- pool has more than 30% free space: use first fit method [1]
- pool has less than 30% free space: use best fit method [1]

This causes a major slowdown of the writes if we go below 30% of free
space. On large pools, 30% may be terabytes of free space.

OpenSolaris has changed this in November 2009 and the Oracle Storage
Appliances also included the new code in Q1/2010 [1].

The source [1] states, that with this change they archieved a speedup
of: 50% Improved OLTP Performance, 70% Reduced Variability, 200%
Improvement on MS Exchange

I would like to issue a Call For Testing for the following 9-CURRENT patch:
http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab.patch

To apply the patch against 8-STABLE, you need to apply the v15 update first:
http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch

The patch includes the following OpenSolaris onnv revisions:
10921 (partial), 11146, 11728, 12047

And covers the following Bug IDs:
6826241 Sync write IOPS drops dramatically during TXG sync
6869229 zfs should switch to shiny new metaslabs more frequently
6917066 zfs block picking can be improved
6918420 zdb -m has issues printing metaslab statistics

References:
[1] http://blogs.sun.com/roch/entry/doubling_exchange_performance
[2] http://forums.freebsd.org/showthread.php?t=8270
[3]
http://blogs.everycity.co.uk/alasdair/2010/07/zfs-runs-really-slowly-when-free-disk-usage-goes-above-80/
[4] http://blogs.sun.com/bonwick/entry/zfs_block_allocation
[5] http://blogs.sun.com/bonwick/entry/space_maps
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] Improved ZFS metaslab code (faster write speed)

2010-08-22 Thread Martin Matuska

 Thank you, I have updated the v15 patch for 8-STABLE.

Dňa 22. 8. 2010 17:44, Olivier Smedts wrote / napísal(a):
 2010/8/22 Martin Matuska m...@freebsd.org:
 Dear FreeBSD community,

 many of our [2] (and Solaris [3]) users today are complaining about slow
 ZFS writes. One of the causes for these writes is the selection of the
 proper allocation method for allocation of new blocks [3] [4]. Another
 issue a write slowdown during TXG sync times.

 Solaris 10 (and OpenSolaris up to november 2009) have the
 following scenario:

 - pool has more than 30% free space: use first fit method [1]
 - pool has less than 30% free space: use best fit method [1]

 This causes a major slowdown of the writes if we go below 30% of free
 space. On large pools, 30% may be terabytes of free space.

 OpenSolaris has changed this in November 2009 and the Oracle Storage
 Appliances also included the new code in Q1/2010 [1].

 The source [1] states, that with this change they archieved a speedup
 of: 50% Improved OLTP Performance, 70% Reduced Variability, 200%
 Improvement on MS Exchange

 I would like to issue a Call For Testing for the following 9-CURRENT patch:
 http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab.patch

 To apply the patch against 8-STABLE, you need to apply the v15 update first:
 http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch
 This one doesn't apply cleanly since few minutes :
 # svn log -l 1 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
 
 r211599 | avg | 2010-08-22 10:18:32 +0200 (Dim 22 aoû 2010) | 7 lignes

 Fix a mismerge in r211581, MFC of r210427

 This is a direct commit.

 Reported by:many
 Pointyhat to:   avg

 

 But it does not seem hard to correct. Do you want me to submit an
 updated patch for 8-stable ?

 The patch includes the following OpenSolaris onnv revisions:
 10921 (partial), 11146, 11728, 12047

 And covers the following Bug IDs:
 6826241 Sync write IOPS drops dramatically during TXG sync
 6869229 zfs should switch to shiny new metaslabs more frequently
 6917066 zfs block picking can be improved
 6918420 zdb -m has issues printing metaslab statistics

 References:
 [1] http://blogs.sun.com/roch/entry/doubling_exchange_performance
 [2] http://forums.freebsd.org/showthread.php?t=8270
 [3]
 http://blogs.everycity.co.uk/alasdair/2010/07/zfs-runs-really-slowly-when-free-disk-usage-goes-above-80/
 [4] http://blogs.sun.com/bonwick/entry/zfs_block_allocation
 [5] http://blogs.sun.com/bonwick/entry/space_maps
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-14 Thread Martin Matuska

 Without head-12636.patch you are unable to reproduce the deadlock?

Dňa 14. 7. 2010 2:14, Peter Jeremy  wrote / napísal(a):
 On 2010-Jul-08 23:30:33 +0200, Martin Matuska m...@freebsd.org wrote:
 On 8. 7. 2010 22:04, Peter Jeremy  wrote / napísal(a):
 Without patching arc_memory_throttle(), a system behaves especially
 poorly if it uses ZFS with any of mmap(2), UFS or NFS client - in my
 case, ports/mail/mairix was almost guaranteed to wedge the system.
 This is the problem that the following hack is intended to work around:
   perl -e '$x = x x 100;'

   
 Regarding ARC, you might want to try the revision 209227 from head that
 is scheduled for MFC on 18.7.2010:
 http://people.freebsd.org/~mm/patches/zfs/head-12636.patch
 I have done some testing with 8-STABLE with head-12636.patch and have
 managed to successfully reproduce a deadlock.  The system is amd64
 with 2GB RAM running a mixed UFS+ZFS environment.  On a freshly booted
 system, I unmount/remount my ZFS /home and a UFS scratch filesystem
 that contains a 1.5GB file [ensuring there is no cached data from
 either FS].  I then dd(1) the 1.5GB UFS file to /dev/null and, once
 that is finished, start mairix on my ~6GB mail directory (on ZFS
 /home).  After some time, I get the following 'systat -v' output:

 4 usersLoad  9.30  8.97  8.33  Jul 14 09:49

 Mem:KBREALVIRTUAL   VN PAGER   SWAP PAGER
 Tot   Share  TotShareFree   in   out in   out
 Act  1223084436   721892 7876   59824  count  
 All  4183767020 1074594k38920  pages  
 Proc:Interrupts
   r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Fltcow4031 total
   4  76  133k3  194   30  135 zfodata0 
 irq14
   ozfod30 bge0 
 irq16
 99.8%Sys   0.2%Intr  0.0%User  0.0%Nice  0.0%Idle%ozfod   atapci1 
 20
 |||||||||||   daefr   uhci0 
 ehci
 ==prcfr   uhci1 22
dtbuf  totfr  2000 cpu0: 
 time
 Namei Name-cache   Dir-cache10 desvn  react  2001 cpu1: 
 time
Callshits   %hits   %   918 numvn  pdwak
273 frevn  pdpgs
   intrn
 Disks   ad0   ad1  540404 wire
 KB/t   0.00  0.00  297512 act
 tps   0 0 1122808 inact
 MB/s   0.00  0.00   57876 cache
 %busy 0 01948 free
218192 buf

 Apart from normal daemons, the only processes running are vmstat,
 systat and mairix (via SSH sessions).  Note that the system is running
 at virtually 100%sys with extremely low free memory and extremely high
 context switches but no obviously useful activity.  At this stage, the
 system is basically unusable (I can't even kill the mairix process).

 My understanding of the problem is that the VM system sees available
 RAM as the sum of cache and free - which is reasonably high so
 there is no pressure to free up inact RAM.  OTOH, ZFS ARC only
 counts free RAM - which is critically low so it throttles itself
 but has no way to get the VM system to move RAM onto the free list.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-14 Thread Martin Matuska

 What about the OpenSolaris revision 9701 for starters?
Could that help your case?

9701:cc5b64682e64

6803605 should be able to offline log devices
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6803605

6726045 vdev_deflate_ratio is not set when offlining a log device
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6726045

Dňa 14. 7. 2010 2:45, jhell wrote / napísal(a):
 On 07/13/2010 12:30, Jason J. W. Williams wrote:
 If there's any way to backport ZFS log device removal that would be
 very helpful. That's the primary hold up for ops folks moving our
 OpenSolaris servers to FreeBSD.
 I second this.

 In explanation I have one machine that needs to be rearranged for a
 upgrade/update for a different purpose than what it is currently doing
 that is holding up some of the other tasks that are planned. I already
 made the mistake once in a test situation by removing a log device that
 was da0 during a reboot  popped in another device, something happened,
 rebooted the machine, plugged in original and the situation was
 irreparable causing the pool to be corrupt.

 Needless to say it was only a test environment but this has stopped me
 cold from using further log devices because of this. ;(

 This is not really top priority right now but would be a great Christmas
 present ;)

 If someone points me in the right direction I guess I could always start
 reviewing the changes for inclusion on a patch against head  stable/8.


 Regards, Thanks  Good Luck,

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-14 Thread Martin Matuska

 In other words:
The problem is not caused by head-12636.patch

And this is important, otherwise we are seeking for an error where it isn't.
I know about the current ARC problem and we have to seek a reasonable
solution for it, but no ugly hacks that work only in specific cases /
workloads.

Dňa 14. 7. 2010 10:23, Peter Jeremy  wrote / napísal(a):
 On 2010-Jul-14 09:16:09 +0200, Martin Matuska m...@freebsd.org wrote:
 Without head-12636.patch you are unable to reproduce the deadlock?
 The deadlock occurs with either stock 8-stable or with head-12636.patch.

 I have also been testing arc.patch2 from
 http://lists.freebsd.org/pipermail/freebsd-stable/2010-July/057696.html
 I am unable to reproduce the deadlock when using the combination of
 arc.patch2 and head-12636.patch but have not yet tested arc.patch2 alone.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[HEADSUP] ZFS version 15 committed to head

2010-07-13 Thread Martin Matuska

 Dear community,

as you may have noticed, ZFS v15 support with many bugfixes was
committed to head in revision 209962.
The commit was tagged for MFC, so if there are no stopper issues I am
going to commit it to stable/8 in 2 months from today.

Here is a short summary of what we gained with this update:

1. Stability - more mature ZFS code, comparable to Solaris 10 update 8
with latest bugfixes
2. Compatibility - Solaris 10 update 8 default ZFS pools can now be
imported into FreeBSD
3. Features - user and group quotas (ZFS-internal) are very useful for
intranet fileservers that want to use ZFS
4. Speed - my real-world benchmarks give a 15-20% RPS yield in PHP
webserver workloads with codebase on ZFS

I would like to thank everybody who helped testing the upgrade before it
got commited.
For the future, it would be very nice to reply not only to my e-mail but
to the mailing list, too, so we can share the experience with others.

For people interested in running this on 8.1 I will provide patches for
releng/8.1 and stable/8 as soon as 8.1 gets released.

Feel free to test everything and don't forget to report any bugs found.

Cheers,
mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-12 Thread Martin Matuska

Upgrading your pool to version 15, compared to version 14, you get only
these additional features:
- user and group quotas
- getting rid of the old version message in zpool status (-x)

and these disadvantages:
- not importable and/or operable with pre-v15 kernel module and updated
boot loader code

If you do not need user and group quotas and the message does not
disturb you, there is no need to upgrade the pool.

Dňa 12. 7. 2010 19:09, Olivier Smedts  wrote / napísal(a):
 I'm currently trying your patchset, I have updated versions of kernel
 and userland zfs code.

 Now, are there any benefits to upgrade the zpool and the zfs to the
 latest versions, besides quotas ?
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-08 Thread Martin Matuska


 using head from 3 hours ago, this patch does not apply cleanly

 http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3-extension.patch
   

The patch you are trying is just for experimental testing and can be
applied only on top of
head-v15-v3.patch (so you need to apply head-v15-v3.patch first, don't
forget -p0).

http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.patch

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-08 Thread Martin Matuska

User and group quotas is no important enhancement?

We have to see the whole thing from a stability perspective as well -
OpenSolaris has by far less testing than Solaris 10.
Oracle cannot afford to feed his enterprise customers (and these are not
few) with untested code.

Dňa 7. 7. 2010 20:30, Sam Fourman Jr. wrote / napísal(a):
 On Wed, Jul 7, 2010 at 1:25 PM, Jason J. W. Williams
 jasonjwwilli...@gmail.com wrote:
   
 If the target is FreeBSD 9 instead of 8.1, why not merge ZFS v19? 15
 really doesn't give any major enhancements over 14 and FreeBSD 9 isn't
 coming out any time.

 19 would give much need log device removal and triple parity RAID-Z.
 Both of which are well tested at this point via OpenSolaris.

 
 these are very valid points, but I am not sure that anyone has zfs v19 patches

   
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-08 Thread Martin Matuska

Hi Jason,

as for me, I am ready to stand for the stability of my v15 upgrade, it
has been discussed with our zfs team, and we also see it as a kind of a
starting point.

We generally have two options:
a) push ZFS v15 now
- it has been already disussed
- we can continue with incremental upgrades that do bugfixes or
introduce non-intrusive features like we did until now
- upgrade to higher versions in the future

b) do not push anything, wait for a uncertain ammount of time and import
a higher version
-  this might take months or even more than 1 year
- our project is not an anarchy or a dictatorship, so it has to be
argued, discussed, evaluated and publicly tested again

As for me, I go for a).

The recent ZFS code contains even more OpenSolaris specific parts, zvol
code probably needs to be reprogrammed once again and there are also
other features like autoexpansion of pools that need polishing. If you
want some future information, there are plans and already work to make
the very latest ZFS available. But its uncertainity again, I cannot give
you any dates but what is very probable that you won't see anything that
early. Of course any volunteers that are willing to help us porting ZFS
features are welcome :-)

Now to the performance fixes for DB workloads - can you point me to the
code or tell me what do they do?
We have already now the prefetch improvements from v19 and ARC
improvements from v15 in stable/8.
Many parts of the OpenSolaris code can be very easily integrated without
breaking existing stuff and again, v15 is a very good starting point for
this.

Regarding performance, e.g. my PHP web servers with codebase in ZFS
yield 15-20% more req/s with v15 patch (as compared to v14).

Cheers,
mm

Dňa 8. 7. 2010 9:47, Jason J. W. Williams  wrote / napísal(a):
 Hi Martin,

 If you're using it for NFS then that can be a good feature, but I see a lot 
 more folks complaining about lack of removal for log devices. 

 We've been using ZFS on OpenSolaris for DB servers since 2006 and OpenSolaris 
 bits are very stable. In most cases we've found ZFS under OSol to be more 
 stable than Solaris. Normally this is due to the youth of ZFS and the speed 
 with which bugs are being corrected...which end up in OSol while Solaris 
 languishes under it's long release cycle.  I'll posit Joyent as an example 
 here of the stability of OSol bits...they use the SXCE distro recently 
 discontinued. 

 v19 also includes a number of performance fixes for DB workloads. 

 -J

 Sent via iPhone

 Is your e-mail Premiere?

 On Jul 8, 2010, at 1:32, Martin Matuska m...@freebsd.org wrote:

   
 User and group quotas is no important enhancement?

 We have to see the whole thing from a stability perspective as well -
 OpenSolaris has by far less testing than Solaris 10.
 Oracle cannot afford to feed his enterprise customers (and these are not
 few) with untested code.

 Dňa 7. 7. 2010 20:30, Sam Fourman Jr. wrote / napísal(a):
 
 On Wed, Jul 7, 2010 at 1:25 PM, Jason J. W. Williams
 jasonjwwilli...@gmail.com wrote:

   
 If the target is FreeBSD 9 instead of 8.1, why not merge ZFS v19? 15
 really doesn't give any major enhancements over 14 and FreeBSD 9 isn't
 coming out any time.

 19 would give much need log device removal and triple parity RAID-Z.
 Both of which are well tested at this point via OpenSolaris.


 
 these are very valid points, but I am not sure that anyone has zfs v19 
 patches


   
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-08 Thread Martin Matuska

On 8. 7. 2010 22:04, Peter Jeremy  wrote / napísal(a):
 On 2010-Jul-05 13:50:52 +0200, Martin Matuska m...@freebsd.org wrote:
   
 As ZFS v15 is already being used in the Solaris 10 enterprise world, we
 can consider it well-tested.
 
 So we know if the ZFS in Solaris 10 includes any fixes that aren't
 publicly available?

   
All fixes for ZFS in Solaris 10 are not publicly available and to
customers only in binary form.
But the list of fixed bug IDs is available and they match the
OpenSolaris Bug IDs ;-)
 Direct link to the patch:
 http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.patch

 The patch applies cleanly against head and stable/8.
 
 In order to apply it to a two-week-old stable/8, I needed to:
 # mkdir cddl/contrib/opensolaris/lib/pyzfs
 # mkdir cddl/contrib/opensolaris/lib/pyzfs/common
 # mkdir cddl/contrib/opensolaris/cmd/pyzfs
   
 Other than verifying that it applies (with the above change), compiles
 and runs, I haven't attempted any stress tests yet.
   
The -p0 flag creates all needed directories:
patch -p0  head-v15-v3.patch
 Looking at the patchset, the most critical issue (IMHO) that doesn't
 appear to have been addressed is the interaction between ZFS ARC and
 the VM cache used by UFS/NFS: arc_memory_throttle() is still making
 decisions solely on the amount of free memory, without considering
 inactive or cache.  I am running a slight variant of a patch by
 Artem Belevich (see http://pastebin.com/ZCkzkWcs) but he acknowledges
 that patch is incomplete (and I've managed to wedge one of my systems
 a couple of times whilst doing zfs send/recv).

 Without patching arc_memory_throttle(), a system behaves especially
 poorly if it uses ZFS with any of mmap(2), UFS or NFS client - in my
 case, ports/mail/mairix was almost guaranteed to wedge the system.
 This is the problem that the following hack is intended to work around:
   perl -e '$x = x x 100;'

   
Regarding ARC, you might want to try the revision 209227 from head that
is scheduled for MFC on 18.7.2010:
http://people.freebsd.org/~mm/patches/zfs/head-12636.patch

OpenSolaris onnv-revision: 12636:13b5d698941e

OpenSolaris Bug IDs:
6950219 large ghost eviction causes high write latency
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6950219

6953403 arc_adjust might adjust MRU unnecessarily
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6953403

6951024 arc_adapt can lead to wild arc_p adjustment
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6951024
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [CFT] ZFS v15 patch (version 3)

2010-07-07 Thread Martin Matuska

We decided not to go with v16 - the feature difference for FreeBSD
between v15 and v16 is zero.
(v16 = Common Multiprotocol SCSI Target (COMSTAR) for ISCSI export of
ZVOLS - we don't do ISCSI export (yet))//

Of course v16 pools cannot be downgraded. But I provided the patch for
testing only, not for production :-)
You can consider the latest patch (v15 v3) to be more in direction
production - it is much more complete and very closely follows Solaris 10.
If someone strictly needs v16 (maybe because upgrading a pool) it is
just a small patch to be added, I can provide it if requested.

Dňa 7. 7. 2010 18:04, jhell  wrote / napísal(a):
 On 07/05/2010 07:50, Martin Matuska wrote:
   
 Direct link to the patch:
 http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.patch

 For full operation (commands zfs allow, unallow, userspace, grouspace)
 the python port must be installed, otherwise these comands don't work or
 have only limited functionality. The port will be added to the ports
 tree soon, you can download it from:
 http://people.freebsd.org/~mm/patches/zfs/v15/sysutils_py-zfs.tar.gz
 
   It should be worth noting for those that have been using the v16 patch
 in the previous v16 CFT that have installed this port and are
 downgrading their system to v15 that they will have to recompile the
 port once the downgrade is complete otherwise this will end up in a
 Segmentation fault when (zfs [allow/unallow/groupspace/userspace]) is run.
   
 I would be very grateful for testing on different architectures, mainly
 on amd64, i386 and sprarc.
 
 Martin,

 ***
   The above python port calls /usr/lib/zfs/pyzfs.py which is installed
 from /usr/src/cddl/contrib/opensolaris/cmd/pyzfs.py that has the
 following bang line for both the noted files.

 #! /usr/bin/python2.4 -S

 Could this be changed to:

 #! /usr/bin/env python -S

 ***
   
The python port (the one updated with v15-3) does install a pyzfs.py
with a correct header, see the Makefile:
http://people.freebsd.org/~mm/patches/zfs/v15/sysutils_py-zfs.tar.gz

I will commit the python port to the ports tree right after it gets
tagged, we are still having a discussion about it's versioning (maybe
starting from 1.0?)

   I think since the integration of python for use with ZFS commands has
 worked its way in it might just be worth STRONGLY noting somewhere that
 python is now heavily depended on for these functions.

   
The pythonized functions are: zfs allow/unallow (don't work without
python), userspace/groupspace (limited functionality without python)
   As for stable/8 i386, ZFSv15 has been stable as a rock. I was running
 v16 with no problems but since it seems unlikely that v16 will make it
 down to stable/8 I downgraded to v15 on revision 209732. As for the
 python testing I am still a little bit shaky with but getting used to
 it. If there is more information that I can provide you with for i386
 please let me know or Ill write back with anything if something should
 arise.

   
What is going to happen, v15 goes to head next week with a 2-month MFC
period.
I will provide patches for stable/8 and 8.1-release.
 Regards  Thanks for the great work on this.
   
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[CFT] ZFS v15 patch (version 3)

2010-07-05 Thread Martin Matuska

Dear FreeBSD community,

there has been a ZFS-related discussion at the meetBSD conference in
Krakow, Poland and we agreed to push ZFS version 15 (and not 16) to
-CURRENT.

An upgrade to version 16 gives us no valuable features (to be true, no
features at all besides ability to import v16 pools).
As ZFS v15 is already being used in the Solaris 10 enterprise world, we
can consider it well-tested.

The goal is to provide a filesystem compatible with Solaris 10 update 8,
which may attract new users to FreeBSD.
Existing users will get the userquota/groupquota features for ZFS and be
able to import Solaris 10 update 8 pools.

Import was done by walking through the path of bugfixes from Solaris 10,
including pre-v15 bugfixes and almost all post-v15 bugfixes.
Few patches are irrelevant to our code (Solaris-specific features) or
modify the zvol part, these have been left out.

I have prepared a new patch that includes almost all revision numbers
Solaris 10 has integrated (we have several of the revisions already in
our tree).
Patch also includes updated manpages and may be considered as a
candidate for head.

Link to the patch information file, including all imported revisions,
bug-ids and reference to Solairis 10 patch numbers:
http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.html
http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.txt

Direct link to the patch:
http://people.freebsd.org/~mm/patches/zfs/v15/head-v15-v3.patch

The patch applies cleanly against head and stable/8.

I am running a patched 8.1 (RC) without any problems so far.
To patch 8.1 (RC), you need to apply revision 209274 from stable/8
before the patch:
http://people.freebsd.org/~mm/patches/zfs/v15/stable-209274.patch

For full operation (commands zfs allow, unallow, userspace, grouspace)
the python port must be installed, otherwise these comands don't work or
have only limited functionality. The port will be added to the ports
tree soon, you can download it from:
http://people.freebsd.org/~mm/patches/zfs/v15/sysutils_py-zfs.tar.gz

For people just wanting to try the new features, I am providing mfsBSD
ISO's with ZFS-on-root install (don't forget the -V 15 flag to the
zfsinstall command):
http://mfsbsd.vx.sk/iso/8.1rc2-zfsv15-v3.iso (without symbols, 99 MB)
http://mfsbsd.vx.sk/iso/8.1rc2-zfsv15-v3-debug.iso (with symbols, 188 MB)

I would be very grateful for testing on different architectures, mainly
on amd64, i386 and sprarc.

Thank you for testing!
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

[CFT] ZFS v16 with stat() speedup

2010-06-24 Thread Martin Matuska

As I have imported some more improvements to the ZFS v15 patch that also
target speed,
I am now calling for testing of v16 with mainly the following important
(post-v16) enhancement (and some related bugfixes):

OpenSolaris Bug ID: 6775100 stat() performance on files on zfs should be
improved
This significantly improves stat() performance of ZFS and has a very
positive effect on e.g webservers using PHP
(I benchmarked a ~20% speed increase of PHP webserving)

The patch is available here:
http://people.freebsd.org/~mm/patches/zfs/v16/head-v16-extended.patch

List of all changes (compared to current head and v15-patch):
http://people.freebsd.org/~mm/patches/zfs/v16/head-v16.txt

And again some ISOs (patched 8.1rc1) for testing:
http://mfsbsd.vx.sk/iso/8.1rc1-zfsv16.iso (without symbols, 98.7MB)
http://mfsbsd.vx.sk/iso/8.1rc1-zfsv16-debug.iso (with kernel symbols,
187.6MB)

Login/password: root/mfsroot

You can test these ISO's in Virtualbox, other emulation software or
real-world systems.
They contain a full installable distribution, here is a zfsinstall
command example on an ad4 disk, with ZFS version 16, 4GB extra swap
partition, lzjb compression and fletcher4 checksum:

1. Login as root (root:mfsroot)
2. mount_cd9660 /dev/acd0 /cdrom
3. zfsinstall -V 16 -t /cdrom/8.1-RC1-amd64.tar.xz -s 4G -c -4 -d ad4

When mfsbsd (not the installed system) is booted, there is a python and
py-zfs package in /packages. You can copy and install these to your
installed system for full pyzfs functionality (zfs userspace, zfs
groupspace, zfs allow, zfs unallow).

Thanks for testing,
mm
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

CFT: ZFS v15 patch

2010-06-22 Thread Martin Matuska

Dear developers,

I would like to do a call for testing for my ZFS v15 patch.

As the user/group quotas feature is too much attractive for my needs,
I couldn't resist and have created (and debugged + tested) a ZFS v15
patch for head (applies cleanly against stable/8 as well).

It is a backport of several onnv-revisions, always consulting pjd's p4
tree and includes four post-9396 related user/groupquota bugfixes.
The bootcode (zfsimpl.h) is properly updated to support v15 as well, the
python part is modified (paths, smb support, ioctls).

The patch, list of imported revisions and a preliminary but working
version of the pyzfs port can be downloaded from this link:
http://people.freebsd.org/~mm/patches/zfs/v15/

You can download 8.1-RC1-amd64 with zfsv15 (incl. boot support) mfsBSD
ISOs from here:
http://mfsbsd.vx.sk/iso/8.1rc1-zfsv15.iso (without symbols, 98.7M)
http://mfsbsd.vx.sk/iso/8.1rc1-zfsv15-debug.iso (with symbols, 187.5M)

Login/password into the ISOs: root/mfsroot
run zfsinstall for a zfs-on-root installation (don't forget -V 15 if
trying to install)

The amd64 ISO's may be tested in virtualbox or real-world systems as well.

Regarding new ioctl's:
I have added all new ioctl's in the patch as new ones (numbers 49 to 52).
This makes the old kernel module work with new library / utilities
(tested) or any other binary that references these ioctl's.

Regarding the python port:
The current port installs the library into PYTHON_SITELIBDIR and
pyzfs.py into /usr/lib/zfs/

Later, we may install all necessary sources and pyzfs.py and the 
port may be a full-dummy that doesn't need kernel/world sources.

Version 15 introduces user and group quota accounting and could be a
very good step in preparation towards v26.
The code runs impressingly stable (I didn't manage to reproduce any
panics or deadlocks yet after polishing).
It would be very great if this could get widely tested.

I guess we could go with this code in direction stable/8 (head first),
because it is backwards compatible
(zfs and zpool work with older kernel module - only new features don't
work, the same for the boot changes, boot from older pools is supported).

Any ideas, suggetions, and bug reports are welcome!

Cheers,
mm

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

66 matches

Mail list logo