misc/157533: [nanobsd][patch] save_cfg improvements

2011-06-02 Thread Alex Bakhtin 

Number: 157533
Category:   misc
Synopsis:   [nanobsd][patch] save_cfg improvements
Confidential:   no
Severity:   non-critical
Priority:   low
Responsible:freebsd-bugs
State:  open
Quarter:
Keywords:   
Date-Required:
Class:  change-request
Submitter-Id:   current-users
Arrival-Date:   Thu Jun 02 17:30:12 UTC 2011
Closed-Date:
Last-Modified:
Originator: Alex Bakhtin
Release:
Organization:
Environment:
Description:
I have written the improved nanobsd save_cfg script with the following features:

1. It recursivelly handles new (modified by user) files in /etc and ask user 
what to do.
2. It recursivelly handles file deletion from /etc and ask user what to do.
3. It keeps 'ignore' list for files that user decided to never add to /cfg

I'm pretty sure that this script is a little bit better than 
http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/145962 because it doesn't 
require to remember - which files was modified. It scans the whole /etc, and if 
it found new file - it ask user - what to do with it.
User can:
a) Add file to /cfg.
b) Skip this file for current script run
c) Add this file to ignore list (/cfg/.ignore) and skip this file for all 
script runs.

One of the most annoying things I found in nanobsd administration is that user 
have to keep in mind ALL files he is going to add to /cfg. This script was used 
on about 10 nanobsd hosts for 1.5 years in production environment and is 
extremely helpfull. The chance that after adding some new configuration file to 
nanobsd user would forget to add it to /cfg was decreased drammatically.

Please consider adding this to Files.


How-To-Repeat:

Fix:


Patch attached with submission follows:

#!/bin/sh
#
# Copyright (c) 2006 Mathieu Arnold
# Copyright (c) 2010 Alex Bakhtin
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#notice, this list of conditions and the following disclaimer in the
#documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
# $FreeBSD: src/tools/tools/nanobsd/Files/root/save_cfg,v 1.1.4.1 2009/08/03 
08:13:06 kensmith Exp $
#

set -e

trap umount /cfg 1 2 15 EXIT
mount /cfg
(
cd /etc
for filename in $@ `find * -type f`
do
if [ ! -f /cfg/$filename -a ! -f /cfg/.ignore/$filename ]
then

#
# If file doesn't exist in /cfg and file is not in the 'ignore' 
list
# then check if this file is exactly the same as original file
# in nanobsd image
#
if ! cmp -s /etc/$filename /conf/base/etc/$filename 
then
file_path=`echo $filename | sed 's/\/[^/]*$//'`
if [ $file_path != $filename ]
then
if [ ! -d /etc/$file_path ]
then
# should never go here unless we have 
some errors in
# sed script extracting file path
echo Error: Path /etc/$file_path is 
not directory.
exit 1;
fi
fi

#
# Ask user - how should we handle this file.
# Add to cfg (y/n/i)?
#   y) - save this file in /cfg
#   n) - do not save this file in /cfg for current 
script invocation ONLY
#   i) - add file to ignore list (/cfg/.ignore 
hiereachy) and never save
# try to add this file to /cfg.
#
# touch is ised to add files to /cfg to keep the script 
flow straight and easy

Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image creation and 100% disk load on zfs

2011-05-25 Thread Alex Bakhtin 
The following reply was made to PR misc/145395; it has been noted by GNATS.

From: Alex Bakhtin alex.bakh...@gmail.com
To: bug-follo...@freebsd.org, i...@freebsd.org
Cc:  
Subject: Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image
 creation and 100% disk load on zfs
Date: Wed, 25 May 2011 17:35:50 +0400

 Hello,
 
 
 From-To:open-closed
 By:imp When:Fri May 13 13:44:33 MDT 2011
 Why:We've been doing async for a while now. Bug OBE.
 
I'm really sorry, but today I have tested the nanobsd.sh from
 9-CURRENT and I'm completely sure that the problem WAS NOT fixed. I
 discovered that building image is still extremelly slow and produces
 great disk load. After checking deeply, I found that image is not
 mounted in async mode.
 
 /dev/md2s1a on /mnt/system/obj/amtkit/_.mnt (ufs, local, soft-updates)
 
I tried to mout in async manually - and it seems that async option
 is ignored for MD-backed filesystem:
 
 bakhtin@tarzan(14) /mnt/system/nanobsd/amtkit
  sudo umount /mnt/system/obj/amtkit/_.mnt
 
 bakhtin@tarzan(14) /mnt/system/nanobsd/amtkit
  sudo mount -o async /dev/md2s1a //mnt/system/obj/amtkit/_.mnt
 
  mount | grep md
 /dev/md0 on /etc (ufs, local)
 /dev/md1 on /var (ufs, local)
 /dev/md2s1a on /mnt/system/obj/amtkit/_.mnt (ufs, local, soft-updates)
 
 As you can see - md2s1a is not mounted in async mode, and this causes
 extremely bad performance on ZFS.
 
 I'm pretty sure that attaching MD in async mode fixes this problem.
 Please review my patch and this bug.
 
 
 --
 ---
 Alex Bakhtin
___
freebsd-bugs@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to freebsd-bugs-unsubscr...@freebsd.org

Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image creation and 100% disk load on zfs

2011-05-25 Thread Alex Bakhtin 
The following reply was made to PR misc/145395; it has been noted by GNATS.

From: Alex Bakhtin alex.bakh...@gmail.com
To: bug-follo...@freebsd.org, i...@freebsd.org
Cc:  
Subject: Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image
 creation and 100% disk load on zfs
Date: Wed, 25 May 2011 21:33:23 +0400

 I made some additional performance testing.
 
 Original 9-current nanobsd.sh:
  sudo ./create.sh
 00:00:00 # NanoBSD image amtkit build starting
 00:00:00 ## Skipping buildworld (as instructed)
 00:00:00 ## Skipping buildkernel (as instructed)
 00:00:00 ## Clean and create world directory (/mnt/system/obj/amtkit/_.w)
 00:00:09 ## Construct install make.conf
 (/mnt/system/obj/amtkit/make.conf.install)
 00:00:09 ## installworld
 00:00:09 ### log: /mnt/system/obj/amtkit/_.iw
 00:02:55 ## install /etc
 00:02:55 ### log: /mnt/system/obj/amtkit/_.etc
 00:02:56 ## configure nanobsd /etc
 00:02:56 ## install kernel (GENERIC_FWD)
 00:02:56 ### log: /mnt/system/obj/amtkit/_.ik
 00:03:00 ## run customize scripts
 00:03:00 ## customize cust_allow_ssh_root
 00:03:00 ### log: /mnt/system/obj/amtkit/_.cust.cust_allow_ssh_root
 00:03:00 ## customize cust_install_files
 00:03:00 ### log: /mnt/system/obj/amtkit/_.cust.cust_install_files
 00:03:00 ## customize cust_ld32_cfg
 00:03:00 ### log: /mnt/system/obj/amtkit/_.cust.cust_ld32_cfg
 00:03:00 ## customize cust_install_packages
 00:03:00 ### log: /mnt/system/obj/amtkit/_.cust.cust_install_packages
 00:04:00 ## customize cust_etc_cfg
 00:04:00 ### log: /mnt/system/obj/amtkit/_.cust.cust_etc_cfg
 00:04:00 ## configure nanobsd setup
 00:04:00 ### log: /mnt/system/obj/amtkit/_.dl
 00:04:05 ## run late customize scripts
 00:04:05 ## late customize cust_dir_cfg
 00:04:05 ### log: /mnt/system/obj/amtkit/_.late_cust.cust_dir_cfg
 00:04:05 ## late customize copy_sources
 00:04:05 ### log: /mnt/system/obj/amtkit/_.late_cust.copy_sources
 00:04:57 ## build diskimage
 00:04:57 ### log: /mnt/system/obj/amtkit/_.di
 Gzipping disk image.
 00:41:02 # NanoBSD image amtkit completed
 
 
 The same nanobsd, my patch applied to 9-current nanobsd.sh and NANO_MD_ASYN=
 C=3D1:
 
 00:00:00 # NanoBSD image amtkit build starting
 00:00:00 ## Skipping buildworld (as instructed)
 00:00:00 ## Skipping buildkernel (as instructed)
 00:00:00 ## Clean and create world directory (/mnt/system/obj/amtkit/_.w)
 00:00:18 ## Construct install make.conf
 (/mnt/system/obj/amtkit/make.conf.install)
 00:00:18 ## installworld
 00:00:18 ### log: /mnt/system/obj/amtkit/_.iw
 00:02:26 ## install /etc
 00:02:26 ### log: /mnt/system/obj/amtkit/_.etc
 00:02:28 ## configure nanobsd /etc
 00:02:28 ## install kernel (GENERIC_FWD)
 00:02:28 ### log: /mnt/system/obj/amtkit/_.ik
 00:02:30 ## run customize scripts
 00:02:30 ## customize cust_allow_ssh_root
 00:02:30 ### log: /mnt/system/obj/amtkit/_.cust.cust_allow_ssh_root
 00:02:30 ## customize cust_install_files
 00:02:30 ### log: /mnt/system/obj/amtkit/_.cust.cust_install_files
 00:02:30 ## customize cust_ld32_cfg
 00:02:30 ### log: /mnt/system/obj/amtkit/_.cust.cust_ld32_cfg
 00:02:30 ## customize cust_install_packages
 00:02:30 ### log: /mnt/system/obj/amtkit/_.cust.cust_install_packages
 00:03:23 ## customize cust_etc_cfg
 00:03:23 ### log: /mnt/system/obj/amtkit/_.cust.cust_etc_cfg
 00:03:23 ## configure nanobsd setup
 00:03:23 ### log: /mnt/system/obj/amtkit/_.dl
 00:03:26 ## run late customize scripts
 00:03:26 ## late customize cust_dir_cfg
 00:03:26 ### log: /mnt/system/obj/amtkit/_.late_cust.cust_dir_cfg
 00:03:26 ## late customize copy_sources
 00:03:26 ### log: /mnt/system/obj/amtkit/_.late_cust.copy_sources
 00:04:56 ## build diskimage
 00:04:56 ### log: /mnt/system/obj/amtkit/_.di
 Gzipping disk image.
 00:09:34 # NanoBSD image amtkit completed
 
 4.5 minutes to buld and gzip disk image with patched nanobsd.sh
 and 36 minutes to build and gzip disk image with original.
 
 Alex Bakhtin
 
 2011/5/25 Alex Bakhtin alex.bakh...@gmail.com:
  Hello,
 
 
 From-To:open-closed
 By:imp When:Fri May 13 13:44:33 MDT 2011
 Why:We've been doing async for a while now. Bug OBE.
 
  =C2=A0 I'm really sorry, but today I have tested the nanobsd.sh from
  9-CURRENT and I'm completely sure that the problem WAS NOT fixed. I
  discovered that building image is still extremelly slow and produces
  great disk load. After checking deeply, I found that image is not
  mounted in async mode.
 
  /dev/md2s1a on /mnt/system/obj/amtkit/_.mnt (ufs, local, soft-updates)
 
  =C2=A0 I tried to mout in async manually - and it seems that async option
  is ignored for MD-backed filesystem:
 
  bakhtin@tarzan(14) /mnt/system/nanobsd/amtkit
  sudo umount /mnt/system/obj/amtkit/_.mnt
 
  bakhtin@tarzan(14) /mnt/system/nanobsd/amtkit
  sudo mount -o async /dev/md2s1a //mnt/system/obj/amtkit/_.mnt
 
  mount | grep md
  /dev/md0 on /etc (ufs, local)
  /dev/md1 on /var (ufs, local)
  /dev/md2s1a on /mnt/system/obj/amtkit/_.mnt (ufs, local, soft-updates)
 
  As you can see - md2s1a

Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image creation and 100% disk load on zfs

2011-05-25 Thread Alex Bakhtin 
The following reply was made to PR misc/145395; it has been noted by GNATS.

From: Alex Bakhtin alex.bakh...@gmail.com
To: Warner Losh i...@bsdimp.com, bug-follo...@freebsd.org
Cc:  
Subject: Re: misc/145395: [nanobsd] [patch] Extremely slow nanobsd disk image
 creation and 100% disk load on zfs
Date: Thu, 26 May 2011 00:08:25 +0400

 --000e0cd22f5e95771b04a41f45de
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 Warner,
 
 I tested it against head, attached.
 
 {bakhtin_nb_fbsd}/mnt/home/bakhtin/work/nanobsd/headsvn info
 Path: .
 URL: http://svn.freebsd.org/base/head/tools/tools/nanobsd
 Repository Root: http://svn.freebsd.org/base
 Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
 Revision: 87
 Node Kind: directory
 Schedule: normal
 Last Changed Author: imp
 Last Changed Rev: 221877
 Last Changed Date: 2011-05-14 08:32:30 +0400 (=D1=81=D0=B1, 14 =D0=BC=D0=B0=
 =D0=B9 2011)
 
 
 
 2011/5/25 Warner Losh i...@bsdimp.com:
  Can you rebase your changes on the latest nanobsd.sh and resend?
 
 
 
 -
 Alex Bakhtin
 
 --000e0cd22f5e95771b04a41f45de
 Content-Type: text/plain; charset=US-ASCII; name=nanobsd_async_head.patch.txt
 Content-Disposition: attachment; filename=nanobsd_async_head.patch.txt
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_go4pfioq0
 
 SW5kZXg6IG5hbm9ic2Quc2gKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gbmFub2JzZC5zaAkocmV2aXNpb24gMjIy
 Mjg3KQorKysgbmFub2JzZC5zaAkod29ya2luZyBjb3B5KQpAQCAtMTM1LDYgKzEzNSwxMCBAQAog
 IyBDYW4gYmUgImZpbGUiIG9yICJzd2FwIgogTkFOT19NRF9CQUNLSU5HPSJmaWxlIgogCisjIDAg
 LT4gVXNlIGRlZmF1bHQgb3B0aW9ucyBmb3IgbWRjb25maWcKKyMgMSAtPiBVc2UgLW8gYXN5bmMg
 b3B0aW9uIGZvciBtZGNvbmZpZyAoaW1wcm92ZSBwZXJmb3JtYW5jZSBvbiB6ZnMpCitOQU5PX01E
 X0FTWU5DPTAKKwogIyBQcm9ncmVzcyBQcmludCBsZXZlbAogUFBMRVZFTD0zCiAKQEAgLTUwOCw2
 ICs1MTIsMTEgQEAKIAlNTlQ9JHtOQU5PX09CSn0vXy5tbnQKIAlta2RpciAtcCAke01OVH0KIAor
 CV9OQU5PX01EQ09ORklHX09QVElPTlM9JyAnCisJaWYgWyAkTkFOT19NRF9BU1lOQyAtZ3QgMCBd
 IDsgdGhlbgorCQlfTkFOT19NRENPTkZJR19PUFRJT05TPSItbyBhc3luYyIKKwlmaQorCiAJaWYg
 WyAiJHtOQU5PX01EX0JBQ0tJTkd9IiA9ICJzd2FwIiBdIDsgdGhlbgogCQlNRD1gbWRjb25maWcg
 LWEgLXQgc3dhcCAtcyAke05BTk9fTUVESUFTSVpFfSAteCAke05BTk9fU0VDVFN9IFwKIAkJCS15
 ICR7TkFOT19IRUFEU31gCkBAIC01MTYsNyArNTI1LDcgQEAKIAkJcm0gLWYgJHtJTUd9CiAJCWRk
 IGlmPS9kZXYvemVybyBvZj0ke0lNR30gc2Vlaz0ke05BTk9fTUVESUFTSVpFfSBjb3VudD0wCiAJ
 CU1EPWBtZGNvbmZpZyAtYSAtdCB2bm9kZSAtZiAke0lNR30gLXggJHtOQU5PX1NFQ1RTfSBcCi0J
 CQkteSAke05BTk9fSEVBRFN9YAorCQkJLXkgJHtOQU5PX0hFQURTfSAke19OQU5PX01EQ09ORklH
 X09QVElPTlN9YAogCWZpCiAKIAl0cmFwICJlY2hvICdSdW5uaW5nIGV4aXQgdHJhcCBjb2RlJyA7
 IGRmIC1pICR7TU5UfSA7IHVtb3VudCAke01OVH0gfHwgdHJ1ZSA7IG1kY29uZmlnIC1kIC11ICRN
 RCIgMSAyIDE1IEVYSVQK
 --000e0cd22f5e95771b04a41f45de--
___
freebsd-bugs@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to freebsd-bugs-unsubscr...@freebsd.org

kern/145339: [zfs] deadlock after detaching block device from raidz pool

2010-04-03 Thread Alex Bakhtin 

Number: 145339
Category:   kern
Synopsis:   [zfs] deadlock after detaching block device from raidz pool
Confidential:   no
Severity:   serious
Priority:   medium
Responsible:freebsd-bugs
State:  open
Quarter:
Keywords:   
Date-Required:
Class:  sw-bug
Submitter-Id:   current-users
Arrival-Date:   Sat Apr 03 07:47:07 UTC 2010
Closed-Date:
Last-Modified:
Originator: Alex Bakhtin
Release:8.0-STABLE
Organization:
Environment:
FreeBSD tarzan-new.private.flydrag.ru 8.0-STABLE FreeBSD 8.0-STABLE #1: Sat Apr 
 3 04:54:06 UTC 2010 
bakh...@tarzan-new.private.flydrag.ru:/mnt/obj/usr/src.old/sys/DEBUG  amd64

Description:
Detaching (physically) block device when there is intensive writing to the pool 
causes deadlock. Tested on 8.0-STABLE/amd64 csuped at 02 Apr 2010. gmirror on 
the same system handles device detach properly. Detaching device when zfs is 
idle doesn't cause any problem.


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x48
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x805815f9
stack pointer   = 0x28:0xff865b80
frame pointer   = 0x28:0xff865bb0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 3 (g_up)
exclusive spin mutex uart_hwmtx (uart_hwmtx) r = 0 (0xff0002a62838) locked 
@ /usr/src.old/sys/dev/uart/uart_cpu.h:92
exclusive lockmgr zfs (zfs) r = 0 (0xff0123079098) locked @ 
/usr/src.old/sys/kern/vfs_vnops.c:607
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xff000c3728f0) locked @ 
/usr/src.old/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xff000c35a8f0) locked @ 
/usr/src.old/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xff0121a31648) locked @ 
/usr/src.old/sys/kern/uipc_sockbuf.c:148

0xff0123079000: tag zfs, type VREG
usecount 1, writecount 1, refcount 1 mountedhere 0
flags ()
v_object 0xff0126114e58 ref 0 pages 0
lock type zfs: EXCL by thread 0xff000c2d7740 (pid 2134)
#0 0x80579d27 at __lockmgr_args+0x777
#1 0x80613339 at vop_stdlock+0x39
#2 0x808d020b at VOP_LOCK1_APV+0x9b
#3 0x806300d7 at _vn_lock+0x57
#4 0x806316d8 at vn_write+0x218
#5 0x805d71e5 at dofilewrite+0x85
#6 0x805d89e0 at kern_writev+0x60
#7 0x805d8ae5 at write+0x55
#8 0x8087b488 at syscall+0x118
#9 0x80861611 at Xfast_syscall+0xe1


db:0:kdb.enter.default  bt
Tracing pid 3 tid 100010 td 0xff0002899740
_mtx_lock_flags() at _mtx_lock_flags+0x39
vdev_geom_io_intr() at vdev_geom_io_intr+0x62
g_io_schedule_up() at g_io_schedule_up+0xed
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xff865d30, rbp = 0 ---



tarzan-new# zdb -vvv
storage
version=14
name='storage'
state=0
txg=578
pool_guid=3309800284037274155
hostid=4266611921
hostname='tarzan-new.private.flydrag.ru'
vdev_tree
type='root'
id=0
guid=3309800284037274155
children[0]
type='raidz'
id=0
guid=11076638880661644944
nparity=1
metaslab_array=23
metaslab_shift=36
ashift=9
asize=10001970626560
is_log=0
children[0]
type='disk'
id=0
guid=134064330288565023
path='/dev/ad10'
whole_disk=0
DTL=33
children[1]
type='disk'
id=1
guid=6567589632071309972
path='/dev/ad12'
whole_disk=0
DTL=32
children[2]
type='disk'
id=2
guid=6024702546194706986
path='/dev/ad14'
whole_disk=0
DTL=27
children[3]
type='disk'
id=3
guid=10837092740689261565
path='/dev/ad16'
whole_disk=0
DTL=31
children[4]
type='disk'
id=4
guid=4165337351109841378
path='/dev/ad18'
whole_disk=0
DTL=30
tarzan-new#

Core:
http://flydrag.dyndns.org:9090/freebsd/zfs-deadlock/core.txt.9


How-To-Repeat