Re: [Ocfs2-users] Shared Writeable MMAP
On Wed, Jun 12, 2013 at 10:34:05PM +0200, Robert Schetterer wrote: Am 12.06.2013 20:56, schrieb Joel Becker: On Wed, Jun 12, 2013 at 02:27:05PM +0200, Robert Schetterer wrote: Hi @ll i have a question about feature Shared Writeable MMAP and its relation to concurent file use. Background, over years i am hosting dovecot imap servers behind loadbalancers on a drbd master/master setup shared formated storage ocfs2 1.4.3 maildir without problems. the dovecot wiki says http://wiki2.dovecot.org/Director Director can be used by Dovecot's IMAP/POP3/LMTP proxy to keep a temporary user - mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem so the director service is most installed on the loadbalancers mainly with nfs storage, proved to be a must have after online investigation and long lasting reality tests , i guess or a cluster filesystem isnt true for ocfs2, cause on drbd it works fine with concurent files use ( i.e mailbox format maildir ) by feature Shared Writeable MMAP. Can sombody confirm this, or teach in easy words how it works ( or not ) and why ? I'm not sure what you are asking. I *think* you are saying, ocfs2 is not a sufficient cluster filesystem for Dovecot no my meaning is reverse of that, i am running it fine since years , but others believe using a director service on the loadbalancer is a must have avoiding concurent use of shared maildir storage and dovecot indexes, which is definite true for nfs, but for my understanding its not needed using drbd/ocfs2 cause of feature i.e Shared writeable mmap or additional features. In using that setup over years without directors, i didnt noticed any errors nor i could produce one with file tests or with imap clients I believe you are correct. ocfs2 will make sure your files are kept in sync between the nodes. So directors are not needed. , because it does not have shared writeable mmap(2). If that's what you are saying, then you are incorrect. Shared writeable mmap(2) is a very standard feature of ocfs2; we can't be POSIX without it. I also don't understand how you are configured. Because it sounds like you are using ocfs2+drbd successfully, which conflicts with your statement that ocfs2 is not a sufficient cluster filesystem. sorry ,descriptions of what exact Shared writeable mmap is, is hard to understand for none kernel hackers which are not natural english speakers, so i asked for clarification mmap(2) is a system call where a program treats an open file as if it were just an area of memory. Rather than issuing read(2) or write(2) calls, the program just accesses the memory with pointers. The OS and filesystem handle reading and writing data when this happens. On a local filesystem, this mostly Just Works. Cluster filesystems have to make sure that modifying memory on node A will cause node B to see the changes. That is what is meant by shared writable mmap. ocfs2 supports this. Joel Joel Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users -- Life's Little Instruction Book #43 Never give up on somebody. Miracles happen every day. http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Shared Writeable MMAP
On Wed, Jun 12, 2013 at 02:27:05PM +0200, Robert Schetterer wrote: Hi @ll i have a question about feature Shared Writeable MMAP and its relation to concurent file use. Background, over years i am hosting dovecot imap servers behind loadbalancers on a drbd master/master setup shared formated storage ocfs2 1.4.3 maildir without problems. the dovecot wiki says http://wiki2.dovecot.org/Director Director can be used by Dovecot's IMAP/POP3/LMTP proxy to keep a temporary user - mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem so the director service is most installed on the loadbalancers mainly with nfs storage, proved to be a must have after online investigation and long lasting reality tests , i guess or a cluster filesystem isnt true for ocfs2, cause on drbd it works fine with concurent files use ( i.e mailbox format maildir ) by feature Shared Writeable MMAP. Can sombody confirm this, or teach in easy words how it works ( or not ) and why ? I'm not sure what you are asking. I *think* you are saying, ocfs2 is not a sufficient cluster filesystem for Dovecot, because it does not have shared writeable mmap(2). If that's what you are saying, then you are incorrect. Shared writeable mmap(2) is a very standard feature of ocfs2; we can't be POSIX without it. I also don't understand how you are configured. Because it sounds like you are using ocfs2+drbd successfully, which conflicts with your statement that ocfs2 is not a sufficient cluster filesystem. Joel Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users -- Life's Little Instruction Book #451 Don't be afraid to say, 'I'm sorry.' http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] How can I umount the OCFS2 file system?
On Tue, Apr 16, 2013 at 11:01:19AM +, Guozhonghua wrote: root@Server20:~# uname -a Linux Server20 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux root@Server20:~# root@Server20:~# mount | grep ocfs2 ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sdc on /vms/vepa-testlun1 type ocfs2 (rw,_netdev,heartbeat=local) /dev/sdb on /vms/vepa-testlun0 type ocfs2 (rw,_netdev,heartbeat=local) /dev/sdh on /vms/share type ocfs2 (rw,_netdev,heartbeat=local) root@Server20:~# umount /vms/vepa-testlun1 umount: /vms/vepa-testlun1: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) root@Server20:~# fuser -m /vms/vepa-testlun1 root@Server20:~# lsof +d /vms/vepa-testlun1 root@Server20:~# lsof /vms/vepa-testlun1 root@Server20:~# umount /vms/vepa-testlun1 umount: /vms/vepa-testlun1: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) root@Server20:~# umount /vms/vepa-testlun1 ;echo $? umount: /vms/vepa-testlun1: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) 1 Are there any NFS or CIFS exports? Joel - ??? This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users -- Nearly all men can stand adversity, but if you really want to test a man's character, give him power. - Abraham Lincoln http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ERROR: status = -17
On Tue, Apr 16, 2013 at 11:08:34AM -0600, Jerry Smith wrote: Good Morning all, Have started seeing this over the last few days on a couple of hosts. Does anyone have any familiarity with this, and or how I can debug what mount/filesystem/inode is having the issues? Apr 16 02:49:36 host1 kernel: (dtocatalogbr,5864,3):ocfs2_prepare_dir_for_insert:4391 ERROR: status = -17 Apr 16 02:49:36 host1 kernel: (dtocatalogbr,5864,3):__ocfs2_prepare_orphan_dir:1932 ERROR: status = -17 Apr 16 02:49:36 host1 kernel: (dtocatalogbr,5864,3):ocfs2_prepare_orphan_dir:1973 ERROR: status = -17 Apr 16 02:49:36 host1 kernel: (dtocatalogbr,5864,3):ocfs2_unlink:887 ERROR: status = -17 Somehow you are trying to unlink a file whose inode is already in the orphan dir. You can use debugfs.ocfs2 to determine what inodes are in the orphan dir, but it isn't easy to see which one triggered the error. You might try checking multiple times over the course of an hour. If the same inode number remains while all of the others clear out, that might be your culprit. Joel #uname -a Linux host1 2.6.32-300.27.1.el6uek.x86_64 #1 SMP Tue Jun 19 14:21:30 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux #cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago) #rpm -qa | grep ocfs ocfs2console-1.6.4-1.el6.x86_64 ocfs2-tools-devel-1.6.4-1.el6.x86_64 ocfs2-tools-1.6.4-1.el6.x86_64 I have a few boxes a bit further along at OEL 6.3 and newer ocfs2 bits, where I am not seeing these errors arise. ocfs2 versions of: $ rpm -qa |grep ocfs ocfs2-tools-1.8.0-10.el6.x86_64 ocfs2-tools-devel-1.8.0-10.el6.x86_64 Are there known issues/fixes between these kernels/ocfs2-tools that fix the ERROR -17 I am seeing? Thanks, --Jerry ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users -- You can use a screwdriver to screw in screws or to clean your ears, however, the latter needs real skill, determination and a lack of fear of injuring yourself. It is much the same with JavaScript. - Chris Heilmann http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Shutting down one node caused all the other nodes to shutdown aswell.
Did you power down nodes uncleanly? The message says that one node lost track of who was doing a particular recovery. If nodes are shut down cleanly, they should be communicating that information. Joel On Thu, Apr 11, 2013 at 12:10:22PM +0200, Kristiansen Morten wrote: I've had no response on my problem, is there anybody who can help me on this? Morten K. Tlf: +47 76 16 61 81 | Mob: +47 906 52 903 Kvalitet - Trygghet - Respekt From: ocfs2-users-boun...@oss.oracle.com [mailto:ocfs2-users-boun...@oss.oracle.com] On Behalf Of Kristiansen Morten Sent: 21. mars 2013 14:47 To: ocfs2-users@oss.oracle.com Subject: [Ocfs2-users] Shutting down one node caused all the other nodes to shutdown aswell. Hi, We are running a 8 nodes cluster on RHEL 2.6.18-128 64-bit. Yesterday the server/san guys exchanged the ocfs2 disks to another SAN, by mirroring and synchronizing the disks. When they rebooted the servers, one of the nodes, tos-dipsprod-07 wasn't able to start Oracle Grid Infrastructure, the voting disk was not found. Then we tried to reboot that node, causing all nodes to reboot. Time round about 02:25. When examine the /var/log/messages I discovered a BUG message on one of the node that rebooted unexpectedly, tos-dipsprod-02. I've tried to google it, but I couldn't find any solution. Is this a well known bug? Does any body have a solution to this problem? Below is a extract of o2net and ocfs2 messages from the /var/log/message file. /var/log/messages til tos-dipsprod-07: Mar 21 02:08:49 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105: has been idle for 10.0 seconds, shutting it down. Mar 21 02:25:25 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-01 (num 0) at 192.168.7.100: has been idle for 10.0 seconds, shutting it down. Mar 21 02:25:35 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101: has been idle for 10.0 seconds, shutting it down. Mar 21 02:25:40 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-03 (num 2) at 192.168.7.102: has been idle for 10.0 seconds, shutting it down. Mar 21 02:25:45 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105: has been idle for 10.0 seconds, shutting it down. Mar 21 02:25:54 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-04 (num 5) at 192.168.7.103: has been idle for 10.0 seconds, shutting it down. Mar 21 04:03:17 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105: has been idle for 10.0 seconds, shutting it down. Mar 21 04:06:32 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-01 (num 0) at 192.168.7.100: has been idle for 10.0 seconds, shutting it down. Mar 21 04:06:37 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101: has been idle for 10.0 seconds, shutting it down. Mar 21 04:06:47 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-03 (num 2) at 192.168.7.102: has been idle for 10.0 seconds, shutting it down. Mar 21 06:04:25 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101: has been idle for 10.0 seconds, shutting it down. Og her fra tos-dipsprod-02: 10474-Mar 21 02:25:15 tos-dipsprod-02 kernel: (o2net,7452,5):dlm_begin_reco_handler:2730 992D008CD522447C8333FC34BD46F8CD: dead_node previously set to 7, node 3 changing it to 7 10646-Mar 21 02:25:25 tos-dipsprod-02 kernel: (o2net,7452,5):dlm_finalize_reco_handler:2839 ERROR: node 6 sent recovery finalize msg, but node 3 is supposed to be the new master, dead=7 10826:Mar 21 02:25:25 tos-dipsprod-02 kernel: Kernel BUG at ...shran/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmrecovery.c:2840 10939-Mar 21 02:43:01 tos-dipsprod-02 syslogd 1.4.1: restart. 10995-Mar 21 02:43:02 tos-dipsprod-02 modprobe: FATAL: Module ocfs2_stackglue not found. -- 17537-Mar 21 04:06:19 tos-dipsprod-02 kernel: (o2net,7472,1):dlm_begin_reco_handler:2730 992D008CD522447C8333FC34BD46F8CD: dead_node previously set to 6, node 6 changing it to 7 17709-Mar 21 04:06:29 tos-dipsprod-02 kernel: (o2net,7472,1):dlm_finalize_reco_handler:2839 ERROR: node 6 sent recovery finalize msg, but node 255 is supposed to be the new master, dead=7 17891:Mar 21 04:06:29 tos-dipsprod-02 kernel: Kernel BUG at ...shran/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmrecovery.c:2840 18004-Mar 21 04:38:04 tos-dipsprod-02 syslogd 1.4.1: restart. 18060-Mar 21 04:41:33 tos-dipsprod-02 modprobe: FATAL: Module ocfs2_stackglue not found. Morten Kristiansen| Counsellor Helse Nord IKT | Departement of Serviceproduction Tlf: +47 76 16 61 81 | Mob: +47 906 52 903 Office address: Amtmann Worsøes gate 63, 8012 Bodø, Norway Quality - Safety - Respect
Re: [Ocfs2-users] Any ideas about ocfs2 fence senario,thanks
On Wed, Jul 04, 2012 at 09:40:26AM +, Guozhonghua wrote: Hi, Everyone, I have a scenario using ocfs2 with a question. If I configure an OCFS2 cluster with three hosts using two ocfs2 disks over iSCSI with different two targets, and they are in different networks. Host1 | ---/dev/sdb --ocfs2-iscsi target1, iscsi's ip address 192.168.10.1 | ---/dev/sdc --ocfs2-iscsi target2, iscsi's ip address 192.168.20.1 Host2 | ---/dev/sdb --ocfs2-iscsi target1, iscsi's ip address 192.168.10.1 | ---/dev/sdc --ocfs2-iscsi target2, iscsi's ip address 192.168.20.1 Host3 | ---/dev/sdb --ocfs2-iscsi target1, iscsi's ip address 192.168.10.1 | ---/dev/sdc --ocfs2-iscsi target2, iscsi's ip address 192.168.20.1 So host1, host2, host3 are the nodes of the ocfs2 cluster, and every one using two ocfs2 disks over iSCSI, and they constructed one cluster for HA. Once Host1 fences itself for network disconnection with other two nodes host2 and host3, or it is timeout of its heartbeat writing timestamp to one targets of the iSCSI sdb. The jobs such as virtual machine's data file on sdb will be restarted for the fence of Host1. At this time, some other jobs on Host1 are running with the disk sdc, such as database of mysql will be also be restarted. Is there any way to avoid this scenario? I mean the jobs running with different disks will be not affected for the host fence of ocfs2. Such as the mysql database on the disk sdc will not restart for the sdb's fence. And is there any way to unmount or dicard the sdb disk without restart the host? So the service running with the disk sdc will not be influenced. The short answer is 'no'. When sdb goes missing, the host has unclean state. If it is allowed to continue, it might later write its out-of-date data to sdb when the network is healthy again. This causes data corruption. Instead, we reboot the node. Joel -- But all my words come back to me In shades of mediocrity. Like emptiness in harmony I need someone to comfort me. http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] remove locks? or copy the whole file?
On Tue, Jul 03, 2012 at 06:57:53PM -0700, Aleks Clark wrote: well, by 'clean', it said it was clean. the locks persisted though. I seriously can't believe there's no way to force lock removal. is it just a file somewhere I can delete? There's no lock hanging around past a full restart. This looks like on-disk corruption. Did fsck.ocfs2 say that it run multiple passes, or just say clean and exit? Please try fsck.ocfs2 with the '-f' flag (obviously with the filesystem not mounted on ANY node). Joel On Tue, Jul 3, 2012 at 6:56 PM, Aleks Clark aleks.cl...@gmail.com wrote: yep, tried that, returned clean. On Tue, Jul 3, 2012 at 6:25 PM, herbert van.den.bergh herbert.van.den.be...@oracle.com wrote: One more thing: did you try running fsck.ocfs2 on it? Thanks, Herbert. On 7/3/2012 6:23 PM, herbert van.den.bergh wrote: Hmm doesn't mean much to me, but maybe to someone else on the list. But I bet their first suggestion will be to try a recent kernel... Thanks, Herbert. On 7/3/2012 6:19 PM, Aleks Clark wrote: Nick, I don't think so, it's a 2tb partition with only 300gb used. Herb, Jul 3 14:47:26 castor kernel: [3488036.578659] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: bug expression: path_leaf_bh(left_path) == path_leaf_bh(right_path) Jul 3 14:47:26 castor kernel: [3488036.578714] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: Owner 18319883: error during insert of 15761664 (left path cpos 20725762) results in two identical paths ending at 395267 Jul 3 14:47:26 castor kernel: [3488036.578800] [ cut here ] Jul 3 14:47:26 castor kernel: [3488036.578826] kernel BUG at /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/ocfs2/alloc.c:2483! Jul 3 14:47:26 castor kernel: [3488036.578881] invalid opcode: [#1] SMP Jul 3 14:47:26 castor kernel: [3488036.578909] last sysfs file: /sys/devices/virtual/net/lo/operstate Jul 3 14:47:26 castor kernel: [3488036.578937] CPU 0 Jul 3 14:47:26 castor kernel: [3488036.578960] Modules linked in: drbd tun ocfs2 jbd2 quota_tree raid0 ip6table_filter ip6_tables iptable_filter ip_tables sha1_generic ebtable_nat ebtables hmac x_tables lru_cache cn kvm_intel kvm ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bridge stp loop md_mod snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr processor button psmouse joydev evdev serio_raw usbhid hid ext3 jbd mbcache dm_mod sd_mod crc_t10dif ahci ehci_hcd libata usbcore scsi_mod e1000e nls_base thermal thermal_sys [last unloaded: drbd] Jul 3 14:47:26 castor kernel: [3488036.579279] Pid: 25326, comm: kvm Not tainted 2.6.32-5-amd64 #1 X9SCL/X9SCM Jul 3 14:47:26 castor kernel: [3488036.579309] RIP: 0010:[a041177b] [a041177b] ocfs2_do_insert_extent+0x5dc/0x1aaf [ocfs2] Jul 3 14:47:26 castor kernel: [3488036.579363] RSP: 0018:880014839688 EFLAGS: 00010292 Jul 3 14:47:26 castor kernel: [3488036.579390] RAX: 00bf RBX: 00060803 RCX: 1806 Jul 3 14:47:26 castor kernel: [3488036.579435] RDX: RSI: 0096 RDI: 0246 Jul 3 14:47:26 castor kernel: [3488036.579479] RBP: 8800148398a8 R08: 000209d0 R09: 000a Jul 3 14:47:26 castor kernel: [3488036.579524] R10: R11: 0001 R12: 013c4002 Jul 3 14:47:26 castor kernel: [3488036.579568] R13: 88002a1e4030 R14: 0001 R15: 88023c153c60 Jul 3 14:47:26 castor kernel: [3488036.579613] FS: 7f0cfef83700() GS:880008a0() knlGS: Jul 3 14:47:26 castor kernel: [3488036.579659] CS: 0010 DS: 002b ES: 002b CR0: 8005003b Jul 3 14:47:26 castor kernel: [3488036.579687] CR2: 7f0d25dbf000 CR3: 00023ccb6000 CR4: 000426e0 Jul 3 14:47:26 castor kernel: [3488036.579732] DR0: DR1: DR2: Jul 3 14:47:26 castor kernel: [3488036.579776] DR3: DR6: 0ff0 DR7: 0400 Jul 3 14:47:26 castor kernel: [3488036.579821] Process kvm (pid: 25326, threadinfo 880014838000, task 88023b999c40) Jul 3 14:47:26 castor kernel: [3488036.579867] Stack: Jul 3 14:47:26 castor kernel: [3488036.579887] 00f08100 013c4002 00060803 880014839718 Jul 3 14:47:26 castor kernel: [3488036.579923]0 880232abde80 88023b999c40 88023b999c40 8800148397a8 Jul 3 14:47:26 castor kernel: [3488036.579977]0 8800148397c8 8800148398a8 88023d8027f8 00f08100 Jul 3 14:47:26 castor kernel: [3488036.580047] Call Trace: Jul 3 14:47:26 castor kernel: [3488036.580074] [a04186b9] ? ocfs2_insert_extent+0x5fb/0x6e6 [ocfs2] Jul 3 14:47:26 castor kernel:
Re: [Ocfs2-users] remove locks? or copy the whole file?
Because it's unsafe to do any I/O at that point. We'd rather you have to reboot than scribble more bad data on your disk! Joel On Tue, Jul 03, 2012 at 11:35:32PM -0700, Aleks Clark wrote: it said 'clean' and exited. Working on bringing the cluster down. Is there a reason why, after the kernel panics, ocfs2 makes all i/o block? I can't even unmount the filesystem on any node, I have to actually reboot it. On Tue, Jul 3, 2012 at 11:17 PM, Joel Becker jl...@evilplan.org wrote: On Tue, Jul 03, 2012 at 06:57:53PM -0700, Aleks Clark wrote: well, by 'clean', it said it was clean. the locks persisted though. I seriously can't believe there's no way to force lock removal. is it just a file somewhere I can delete? There's no lock hanging around past a full restart. This looks like on-disk corruption. Did fsck.ocfs2 say that it run multiple passes, or just say clean and exit? Please try fsck.ocfs2 with the '-f' flag (obviously with the filesystem not mounted on ANY node). Joel On Tue, Jul 3, 2012 at 6:56 PM, Aleks Clark aleks.cl...@gmail.com wrote: yep, tried that, returned clean. On Tue, Jul 3, 2012 at 6:25 PM, herbert van.den.bergh herbert.van.den.be...@oracle.com wrote: One more thing: did you try running fsck.ocfs2 on it? Thanks, Herbert. On 7/3/2012 6:23 PM, herbert van.den.bergh wrote: Hmm doesn't mean much to me, but maybe to someone else on the list. But I bet their first suggestion will be to try a recent kernel... Thanks, Herbert. On 7/3/2012 6:19 PM, Aleks Clark wrote: Nick, I don't think so, it's a 2tb partition with only 300gb used. Herb, Jul 3 14:47:26 castor kernel: [3488036.578659] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: bug expression: path_leaf_bh(left_path) == path_leaf_bh(right_path) Jul 3 14:47:26 castor kernel: [3488036.578714] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: Owner 18319883: error during insert of 15761664 (left path cpos 20725762) results in two identical paths ending at 395267 Jul 3 14:47:26 castor kernel: [3488036.578800] [ cut here ] Jul 3 14:47:26 castor kernel: [3488036.578826] kernel BUG at /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/ocfs2/alloc.c:2483! Jul 3 14:47:26 castor kernel: [3488036.578881] invalid opcode: [#1] SMP Jul 3 14:47:26 castor kernel: [3488036.578909] last sysfs file: /sys/devices/virtual/net/lo/operstate Jul 3 14:47:26 castor kernel: [3488036.578937] CPU 0 Jul 3 14:47:26 castor kernel: [3488036.578960] Modules linked in: drbd tun ocfs2 jbd2 quota_tree raid0 ip6table_filter ip6_tables iptable_filter ip_tables sha1_generic ebtable_nat ebtables hmac x_tables lru_cache cn kvm_intel kvm ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bridge stp loop md_mod snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr processor button psmouse joydev evdev serio_raw usbhid hid ext3 jbd mbcache dm_mod sd_mod crc_t10dif ahci ehci_hcd libata usbcore scsi_mod e1000e nls_base thermal thermal_sys [last unloaded: drbd] Jul 3 14:47:26 castor kernel: [3488036.579279] Pid: 25326, comm: kvm Not tainted 2.6.32-5-amd64 #1 X9SCL/X9SCM Jul 3 14:47:26 castor kernel: [3488036.579309] RIP: 0010:[a041177b] [a041177b] ocfs2_do_insert_extent+0x5dc/0x1aaf [ocfs2] Jul 3 14:47:26 castor kernel: [3488036.579363] RSP: 0018:880014839688 EFLAGS: 00010292 Jul 3 14:47:26 castor kernel: [3488036.579390] RAX: 00bf RBX: 00060803 RCX: 1806 Jul 3 14:47:26 castor kernel: [3488036.579435] RDX: RSI: 0096 RDI: 0246 Jul 3 14:47:26 castor kernel: [3488036.579479] RBP: 8800148398a8 R08: 000209d0 R09: 000a Jul 3 14:47:26 castor kernel: [3488036.579524] R10: R11: 0001 R12: 013c4002 Jul 3 14:47:26 castor kernel: [3488036.579568] R13: 88002a1e4030 R14: 0001 R15: 88023c153c60 Jul 3 14:47:26 castor kernel: [3488036.579613] FS: 7f0cfef83700() GS:880008a0() knlGS: Jul 3 14:47:26 castor kernel: [3488036.579659] CS: 0010 DS: 002b ES: 002b CR0: 8005003b Jul 3 14:47:26 castor kernel: [3488036.579687] CR2: 7f0d25dbf000 CR3: 00023ccb6000 CR4: 000426e0 Jul 3 14:47:26 castor kernel: [3488036.579732] DR0: DR1: DR2: Jul 3 14:47:26 castor kernel: [3488036.579776] DR3: DR6: 0ff0 DR7: 0400 Jul 3 14:47:26 castor kernel: [3488036.579821] Process kvm (pid: 25326, threadinfo 880014838000, task 88023b999c40) Jul 3 14:47:26 castor kernel
Re: [Ocfs2-users] remove locks? or copy the whole file?
On Tue, Jul 03, 2012 at 11:46:08PM -0700, Aleks Clark wrote: any ideas how long this is going to take on a 2tb fs with ~400gb used? going on 10 hours of downtime, and it's been doing Pass 0a for the past 10 minutes. also, should all the nodes be up (but unmounted) for this? I don't have any good idea how long it will take. That's a lot of filesystem. However, it should take less than 10h! You don't need any other nodes up or down. All that matters is that there are no mounts. Joel On Tue, Jul 3, 2012 at 11:42 PM, Joel Becker jl...@evilplan.org wrote: Because it's unsafe to do any I/O at that point. We'd rather you have to reboot than scribble more bad data on your disk! Joel On Tue, Jul 03, 2012 at 11:35:32PM -0700, Aleks Clark wrote: it said 'clean' and exited. Working on bringing the cluster down. Is there a reason why, after the kernel panics, ocfs2 makes all i/o block? I can't even unmount the filesystem on any node, I have to actually reboot it. On Tue, Jul 3, 2012 at 11:17 PM, Joel Becker jl...@evilplan.org wrote: On Tue, Jul 03, 2012 at 06:57:53PM -0700, Aleks Clark wrote: well, by 'clean', it said it was clean. the locks persisted though. I seriously can't believe there's no way to force lock removal. is it just a file somewhere I can delete? There's no lock hanging around past a full restart. This looks like on-disk corruption. Did fsck.ocfs2 say that it run multiple passes, or just say clean and exit? Please try fsck.ocfs2 with the '-f' flag (obviously with the filesystem not mounted on ANY node). Joel On Tue, Jul 3, 2012 at 6:56 PM, Aleks Clark aleks.cl...@gmail.com wrote: yep, tried that, returned clean. On Tue, Jul 3, 2012 at 6:25 PM, herbert van.den.bergh herbert.van.den.be...@oracle.com wrote: One more thing: did you try running fsck.ocfs2 on it? Thanks, Herbert. On 7/3/2012 6:23 PM, herbert van.den.bergh wrote: Hmm doesn't mean much to me, but maybe to someone else on the list. But I bet their first suggestion will be to try a recent kernel... Thanks, Herbert. On 7/3/2012 6:19 PM, Aleks Clark wrote: Nick, I don't think so, it's a 2tb partition with only 300gb used. Herb, Jul 3 14:47:26 castor kernel: [3488036.578659] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: bug expression: path_leaf_bh(left_path) == path_leaf_bh(right_path) Jul 3 14:47:26 castor kernel: [3488036.578714] (25326,0):ocfs2_rotate_tree_right:2483 ERROR: Owner 18319883: error during insert of 15761664 (left path cpos 20725762) results in two identical paths ending at 395267 Jul 3 14:47:26 castor kernel: [3488036.578800] [ cut here ] Jul 3 14:47:26 castor kernel: [3488036.578826] kernel BUG at /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/ocfs2/alloc.c:2483! Jul 3 14:47:26 castor kernel: [3488036.578881] invalid opcode: [#1] SMP Jul 3 14:47:26 castor kernel: [3488036.578909] last sysfs file: /sys/devices/virtual/net/lo/operstate Jul 3 14:47:26 castor kernel: [3488036.578937] CPU 0 Jul 3 14:47:26 castor kernel: [3488036.578960] Modules linked in: drbd tun ocfs2 jbd2 quota_tree raid0 ip6table_filter ip6_tables iptable_filter ip_tables sha1_generic ebtable_nat ebtables hmac x_tables lru_cache cn kvm_intel kvm ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bridge stp loop md_mod snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr processor button psmouse joydev evdev serio_raw usbhid hid ext3 jbd mbcache dm_mod sd_mod crc_t10dif ahci ehci_hcd libata usbcore scsi_mod e1000e nls_base thermal thermal_sys [last unloaded: drbd] Jul 3 14:47:26 castor kernel: [3488036.579279] Pid: 25326, comm: kvm Not tainted 2.6.32-5-amd64 #1 X9SCL/X9SCM Jul 3 14:47:26 castor kernel: [3488036.579309] RIP: 0010:[a041177b] [a041177b] ocfs2_do_insert_extent+0x5dc/0x1aaf [ocfs2] Jul 3 14:47:26 castor kernel: [3488036.579363] RSP: 0018:880014839688 EFLAGS: 00010292 Jul 3 14:47:26 castor kernel: [3488036.579390] RAX: 00bf RBX: 00060803 RCX: 1806 Jul 3 14:47:26 castor kernel: [3488036.579435] RDX: RSI: 0096 RDI: 0246 Jul 3 14:47:26 castor kernel: [3488036.579479] RBP: 8800148398a8 R08: 000209d0 R09: 000a Jul 3 14:47:26 castor kernel: [3488036.579524] R10: R11: 0001 R12: 013c4002 Jul 3 14:47:26 castor kernel: [3488036.579568] R13: 88002a1e4030 R14: 0001 R15: 88023c153c60 Jul 3 14:47:26 castor kernel
Re: [Ocfs2-users] BUG at fs/ocfs2/extent_map.c:476
On Mon, Apr 02, 2012 at 10:36:53AM +0200, richard -rw- weinberger wrote: On Mon, Apr 2, 2012 at 10:25 AM, Joel Becker jl...@evilplan.org wrote: On Sat, Mar 24, 2012 at 09:50:49AM +0100, richard -rw- weinberger wrote: Hi! Today one of my kvm hosts ran into this BUG. The host is member of y small ocfs2-cluster (3 hosts). CentOS 6.2, Kernel 3.1.1 (vanilla). ... Is this a known/fixed bug? Not that I know of. Have you seen it more than once? No. All three cluster members had an uptime of 70 days. Only one showed the BUG. I've rebooted it and it seems to work fine now. I'm not an ocfs2 expert, what could cause this kind of BUG? Internal logic error? Filesystem corruption? No idea. That's the problem. More details would be nice, but I suspect you'd rather not crash again :-) Joel -- Soap and education are not as sudden as a massacre, but they are more deadly in the long run. - Mark Twain http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 performance and debug help
On Sun, Apr 08, 2012 at 10:54:52AM -0700, Jay V wrote: I have a similar blocking/hanging/stall issue. On Oracle 6.2/x64. We are running OCFS2 on 3 partitions about 14-15TB each. One of the partitions has been running extremely slowly too. They are running on the same hardware-- LSI HW Raid Cards and Enterprise Drives. I am running over drbd and nfsd. How do drbd and nfsd fit in your scheme? nfsd is on top of ocfs2, right? You're exporting ocfs2 volumes via NFS? Can you think of any differences between the slow partition and the fast ones? I am getting really slow performance in writes. The process [jbd2-drbd-18] seems to be stuck for a long length of time (about 2 minutes) before it finally commits. This jbd2-drbd2 prevents any other writes from happening which stalls the system. I'm worried that drbd is doing things slowly and getting in your way. But given the slowdown you are replying to, that might not be the only issue. This would be from ps auxr USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6374 0.0 0.0 0 0 ? D Apr03 0:28 [jbd2/drbd2-18] root 6876 0.0 0.0 0 0 ? D Apr06 0:11 [nfsd] root 6884 0.0 0.0 0 0 ? D Apr06 0:09 [nfsd] root 6999 0.0 0.0 0 0 ? D Apr06 0:45 [nfsd] root 7046 0.0 0.0 0 0 ? D Apr06 0:09 [nfsd] root 7053 0.0 0.0 0 0 ? D Apr06 0:09 [nfsd] root 7054 0.0 0.0 0 0 ? D Apr06 0:08 [nfsd] Running scan_locks2 shows nothing. Nothing is held up locking wise. It seems to happen more with files copying about 1MB or larger. It only happens for me on my second partition, but not the other 2. It seems to super slow in writes. Reads are fast. I hope to find a solution quickly too. I wonder it is because we have very large partitions. It shouldn't be because of large partitions. How long have your partitions been around? How full are they? Joel -- One look at the From: understanding has blossomed .procmailrc grows - Alexander Viro http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 performance and debug help
On Thu, Apr 05, 2012 at 05:11:59PM +0100, Adelino Monteiro wrote: Hello all, For 4 month now I'm using OCFS in an environment with 7 partitions each with 14 Tb running Oracle Linux 6.2 and until last week everything was fine. Now however we're running into severe performance problems when doing simple copies. I have one of the 7 partitions mounted as RW on one server and 4 servers with RO. I did a simple cp of various files on the RW server and during that copy the process got into D state and a simple df for instance blocked. It took minutes for something that should be immediate. This is happening on any of those partitions. Hey Adelino, I'd love to understand your problems. You say you've been running these systems for four months. Are the slowdowns new? Was anything happening on the RO servers at the time? Especially touching the same files or directories? How full are the filesystems? How much change to they have (that is, are the files long-lived or constantly being deleted and created)? Joel -- Also, all of life's big problems include the words 'indictment' or 'inoperable.' Everything else is small stuff. - Alton Brown http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] BUG at fs/ocfs2/extent_map.c:476
On Sat, Mar 24, 2012 at 09:50:49AM +0100, richard -rw- weinberger wrote: Hi! Today one of my kvm hosts ran into this BUG. The host is member of y small ocfs2-cluster (3 hosts). CentOS 6.2, Kernel 3.1.1 (vanilla). ... Is this a known/fixed bug? Not that I know of. Have you seen it more than once? Joel -- None of our men are experts. We have most unfortunately found it necessary to get rid of a man as soon as he thinks himself an expert -- because no one ever considers himself expert if he really knows his job. A man who knows a job sees so much more to be done than he has done, that he is always pressing forward and never gives up an instant of thought to how good and how efficient he is. Thinking always ahead, thinking always of trying to do more, brings a state of mind in which nothing is impossible. The moment one gets into the expert state of mind a great number of things become impossible. - From Henry Ford Sr., My Life and Work http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Slow OCFS2 on very high-end hardware
On Sun, Dec 04, 2011 at 10:48:40AM +0100, Marek Królikowski wrote: From: Joel Becker Sent: Sunday, December 04, 2011 2:10 AM To: MarekKrólikowski Cc: ocfs2-users@oss.oracle.com Subject: Re: [Ocfs2-users] Slow OCFS2 on very high-end hardware On Sat, Dec 03, 2011 at 08:32:30PM +0100, Marek Królikowski wrote: Hello Today i create a cluster with OCFS2. I name servers MAIL1 and MAIL2 Both connect via HBA card with 2 links 4Gbit/s and EMC storage with FC RAID10. Both connect to this same Cisco switch 1Gbit/s line. Hardware is awsome but ocfs2 work vey slow. I use Gentoo Linux with Kernel 3.0.6 and ocfs2-tools-1.6.4 that will be postfix/imap/pop3 cluster with maildir support so there will be many many directores and little files. I link /home to my ocfs2 and do few tests but work verry slow... When i write any file on server MAIL1 and try check mailbox from MAIL2 working amazing slow... I've gotta ask, what is amazingly slow to you? A cluster filesystem accessing the same files from two places necessarily is slower than local access. But if it is slow enough that you notice it by hand, it's probably something in configuration. Did you select the 'mail' filesystem type when creating the filesystem? This probably shouldn't affect your simple test, but it will absolutely help as your system grows. When i copy any file from/to MAIL1 and enter to home directory (using MC) where is 7000 users on MAIL2 i need wait 30+ sec at normal when i don`t copy write on ocfs2 i wait 3 sec. Yeah, that seems really long. Are you using indexed directories? I don't think your multipath setup is the problem. Joel -- There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle. - Albert Einstein http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 userspace tools
On Mon, Dec 05, 2011 at 09:09:50AM +0100, David Weber wrote: We are currently investigating to use OCFS2 in our virtualization servers and we are very interested in the reflink feature. We found the reflink userspace tool at http://oss.oracle.com/git/?p=jlbec/reflink.git;a=summary but it's bit outdated. Is there any newer version available somewhere like in Oracle Linux? Same question with the shared-du and defragmentation tool. The reflink tool will work just fine as-is. If you have problems, let me know. Joel -- Hey mister if you're gonna walk on water, Could you drop a line my way? http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] mkfs.ocfs2 optimal options for web server
On Wed, Jun 01, 2011 at 09:44:30AM -0700, Sunil Mushran wrote: On 06/01/2011 02:03 AM, Alex Sobrino wrote: We're planning a three web server cluster based on OCFS2. Basically, it will handle a huge CMS, with lots of PHP code, and some file uploads (but mainly file reads). Initially, I was thinking in: - Block size 4K - Cluster size 4K - Node slots 3 - FS type mail - Journal options - let mkfs.ocfs2 guess them Do you think that I should modify anything? Do any of you have any similar configuration? It makes sense to add additional slots upfront to allow you to grow later. It will be easier if you have slots available. So go with 4 atleast if not 6. The only downside is that it will reserve more journal space. But it's space you can get it back later. Also enable most features. The list depends on the version of the tools/fs you have. If you are on a kernel that is based off 2.6.32+, then you should be able to enable most features. For more, refer to man mkfs.ocfs2. For example, you'll really want indexed directories. Joel -- One of the symptoms of an approaching nervous breakdown is the belief that one's work is terribly important. - Bertrand Russell http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 1.6 for RHEL?
On Fri, May 13, 2011 at 09:53:37PM +0200, Kristian Jörg wrote: We are using CentOS (based on RHEL5). So this basically means we are stuck with ocfs2 1.4 forever when it comes to official releases? But I guess when you release the source code for 1.6 I could try and compile it myself for CentOS? 1.6 is still GPL right? Yes, you absolutely can compile it yourself. There is no guarantees that the specfiles will exist or work, but if you create working specfiles please sent a patch. We have been testing ocfs2 since we have been having huge problems with GFS with the kind of filesystems we have (lots and lots of small files and mostly read access). Ocfs2 seems to solve all these performance problems and we had high hopes for it. However the news that 1.6 and forward will not be available for RHEL/CentOS is bad news. I do hope you will reconsider. A big user base is the best way to make successful products IMHO. The latest ocfs2 is always in mainline. Any distro building a recent kernel an including ocfs2 will have it. As Sunil says, this includes Fedora, Debian, Ubuntu, and OpenSuSE. Whether ocfs2 is enabled in the CentOS kernel is up to CentOS. Joel -- Life's Little Instruction Book #232 Keep your promises. http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs or configfs bug ?
On Tue, Apr 19, 2011 at 04:54:32PM +0200, Welterlen Benoit wrote:
I have a bug with OCFS through configfs : to illustrate this, try :
while true ; do ls -l /sys/kernel/config/cluster/ocfs2/heartbeat ; done
while true ; do echo 31
/sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold ; done
Interesting!
RIP [a01fd214] configfs_readdir+0xf4/0x230 [configfs]
snip
#8 [880c6c8b3ee0] vfs_readdir at 8116c120
#9 [880c6c8b3f30] sys_getdents at 8116c2a9
I presume this is in the process that is ls(1)ing the directory?
I've looked into the source code, but I found that a lock is useless :
/* Only sets a new threshold if there are no active regions.
*
* No locking or otherwise interesting code is required for reading
* o2hb_dead_threshold as it can't change once regions are active and
* it's not interesting to anyone until then anyway. */
static void o2hb_dead_threshold_set(unsigned int threshold)
{
if (threshold O2HB_MIN_DEAD_THRESHOLD) {
spin_lock(o2hb_live_lock);
if (list_empty(o2hb_all_regions))
o2hb_dead_threshold = threshold;
spin_unlock(o2hb_live_lock);
}
}
You're too late here. This is in the echo process (bash,
really). getdents() isn't happening.
The problem is almost certainly in configfs. It's a race
between setup and teardown of the virtual attribute files. If anyone
else has a cycle to look at it, great, otherwise I'll try to get to it
later this week.
Joel
--
In a crisis, don't hide behind anything or anybody. They're going
to find you anyway.
- Paul Bear Bryant
http://www.jlbec.org/
jl...@evilplan.org
___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] o2cb stack and kernel = 2.6.37
On Mon, Apr 04, 2011 at 03:20:43PM +0200, Jacek Stępniewski wrote: Is there anybody who uses o2cb stack with mixed kernels (2.6.37 or more and less than 2.6.37)? I looks like locking protocol version problem (2.6.37 has 1.1. dlm protocol). Is there any patch for this? The change is rolling upgrade compatible. If the first machine to mount is the newer one, the newer protocol is selected. If an older machine mounts first, the newer machines will join at the older protocol. Joel -- Not everything that can be counted counts, and not everything that counts can be counted. - Albert Einstein http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Versions and features
On Mon, Feb 28, 2011 at 02:07:16PM +0100, Werner Flamme wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all, last Friday my colleague (using Oracle's Linux 5) und me tried to get access to a shared OCFS2 volume. Since he claimed that he now has OCFS2 1.6, and I only have 1.4.3 (list below), we decided that I create the filesystem with --fs-feature-level=max-features. If you want to share among differing versions, you want --fs-feature-level=max-compat. max-features means select the newest features, which will be incompatible with ocfs2 1.4. Joel -- Life's Little Instruction Book #407 Every once in a while, take the scenic route. http://www.jlbec.org/ jl...@evilplan.org ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] servers blocked on ocfs2
On Fri, Dec 10, 2010 at 08:42:19AM +0100, frank wrote: Hi Joel, thanks for your answer, usually when the cable is unplugged it appears messages related to link, and there isn't. I thought in a OpenVZ issue also, but I don't find any evidence of that. Yeah, I'm just wondering what could make traffic not pass over the link. There is a dedicated link as a mentioned, used for OCFS2 and heartbeat/Pacemaker; may be last one hampered the interface. I don't see how either one would have hampered the interface. If no other traffic (iSCSI, http, whatever) is over that interface, there just shouldn't be that much. Anyway, if there was a cut in the heartbeat or something similar, one of the nodes should have fenced itself, haven't it? Why did the nodes stall? Can we avoid that? If both nodes saw the network go down, but the disk heartbeat was still working, the higher node should have fenced. Was there no fencing? Was it just both nodes hung? How were they hung? All operation, or just ocfs2 operations? Joel -- War doesn't determine who's right; war determines who's left. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] servers blocked on ocfs2
On Thu, Dec 09, 2010 at 11:10:58AM +0100, frank wrote: we have recently started to use ocfs2 on some RHEL 5.5 servers (ocfs2-1.4.7) Some days ago, two servers sharing an ocfs2 filesystem, and with quite virtual services, stalled, in what it seems on ocfs2 issue. This are the lines in their messages files: =node heraclito (0) /Dec 4 09:15:06 heraclito kernel: o2net: connection to node parmenides (num 1) at 192.168.1.2: has been idle for 30.0 seconds, shutting it down. snip As you can see, it seems problem started at same time in both nodes. This nodes have a crossover cable on a dedicated eth interface in the private net 192.168.1.0; we use OpenVZ on these hosts, so kernel is OpenVZ patched and we recompiled ocfs2 sources to get apropiated ocfs2 modules. The network connection went down or stalled. The first question I would have is whether the cable was unplugged or otherwise hampered? Could OpenVZ have closed off that network connection for a bit, or could an admin have accidentally shut it down? Is there other traffic on the cable that might have caused o2cb messaging to be stalled? Joel -- Three o'clock is always too late or too early for anything you want to do. - Jean-Paul Sartre Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Reservation conflicts
On Thu, Dec 09, 2010 at 04:02:14PM -0600, brad hancock wrote: I have two vmware guests that share a OCFS2 partition through VMWare VMFS. It resides on a SAN. Let me see if I understand the configuration. You have a SAN. On that SAN is a LUN. You have one VMWare host in this configuration. The VMWare host has formatted that LUN for VMFS. There is a disk image on the VMFS that both guests see as sdb. Is this correct? Each host has the following error in the kernel log: [2037805.922718] end_request: I/O error, dev sdb, sector 1735 [2037805.922974] (0,0):o2hb_bio_end_io:225 ERROR: IO Error -5 [2037805.923370] (27506,0):o2hb_do_disk_heartbeat:753 ERROR: status = -5 We also see the following on both machines: (1888,0):o2hb_do_disk_heartbeat:753 ERROR: status = -5 [202381.822030] (1888,0):o2hb_do_disk_heartbeat:762 ERROR: Device sdb1: another node is heartbeating in our slot! I notice the sector is the same on both machines: 1735. Is this an issue with vmware? The read errors (another node is heartbeating in our slot) sound like VMWare is not allowing the nodes to see each other's activity. The write error is more surprising, but it could be part of the same issue. Joel -- Every day I get up and look through the Forbes list of the richest people in America. If I'm not there, I go to work. - Robert Orben Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Reservation conflicts
On Thu, Dec 09, 2010 at 04:45:25PM -0600, brad hancock wrote: Yeah both guest have the same Harddrive attached with the virtual scsi controller configured as Physical to set a policy to allow virtual disk to be used simultaneously by multi virtual machines. as /dev/sdb1 It sure seems like VMWare is caching some data somewhere. That's my best guess. These are on the same host, right? Joel -- Life's Little Instruction Book #510 Count your blessings. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Compiling ocfs2-tools-1.6.3 on slackware64-13.1
On Wed, Dec 01, 2010 at 02:15:24AM +, pica pica wrote: Any clues as to what could be wrong? What is breaking is that libcom_err only has a dynamic object, so it dynamically references sem_init, etc. When the linker tries to resolve that, it can't find it. Adding -lpthread directly specifies the location of sem_init. This all is not necessary when runtime linking is used (ld.so). Thus, we know you are getting a static build. Something about your build environment is causing a static build. Does Slackware have a build-everything-static mode? The compile line does not have -static, so I'm guessing it is in your toolchain. Would the las 2 flags not force a dynamic build? They force ocfs2-tools to build dynamically, ie, without -static. If your toolchain is doing something underneath, our configure script can't fix that. Joel -- I don't want to achieve immortality through my work; I want to achieve immortality through not dying. - Woody Allen Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] disabling self-fencing
On Tue, Nov 02, 2010 at 02:10:53PM +0100, Riccardo Villa wrote: on my nodes I have another cluster manager running that takes care of fencing the node, is it safe to disable ocfs2 self-fencing? Details: I'm using ocfs2 over drbd in master/master aka primary/primary mode. In case of loss of network connectivity I would like to disconnect the drbd device, invalidate it and unmount the filesystem but ocfs2 reboots the node Is it possible to disable ocfs2 self-fencing in other ways rather than commenting-out releveant code in ocfs2 kernel module? No, because there is no way for the ocfs2 software to be notified by your other code that it has been fenced. Since ocfs2 can't be notified, it can't know whether it made a decision that would result in a corruption before or after your fencing software did its job. Joel -- Life's Little Instruction Book #94 Make it a habit to do nice things for people who will never find out. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2_delete_inode kernel bug
On Thu, Oct 28, 2010 at 02:16:03AM -0200, Andre Nathan wrote: Hello Sunil The errors happened again, but now I think it may be completely fixed. I only got the -17 error for a single inode this time: Are you having more than one machine access the same disk without being in the same cluster? I would hope not, but something is weird here. Joel -- None of our men are experts. We have most unfortunately found it necessary to get rid of a man as soon as he thinks himself an expert -- because no one ever considers himself expert if he really knows his job. A man who knows a job sees so much more to be done than he has done, that he is always pressing forward and never gives up an instant of thought to how good and how efficient he is. Thinking always ahead, thinking always of trying to do more, brings a state of mind in which nothing is impossible. The moment one gets into the expert state of mind a great number of things become impossible. - From Henry Ford Sr., My Life and Work Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2_delete_inode kernel bug
On Tue, Oct 26, 2010 at 02:59:17PM -0200, Andre Nathan wrote: My setup is the following: I have two servers sharing OCFS2 filesystems through one dedicated 10Gbps interface. Where are the disks? I'm guessing they're on an iSCSI server at the other end of the 10Gbps interface, but what is the technology there? There is also a backup server which exports OCFS2-formatted devices via ATA-over-ethernet. This machine is connected to each server through standard gigabit ethernet interfaces. The two servers mount their respective volumes and run the backup script. Let me see if I understand this. The backup server has disks. Those disks are formatted for ocfs2. The backup server does NOT mount these disks, it merely exports them via AoE. Each server mounts its own AoE disk as ocfs2 and writes backup data to the ocfs2 filesystems there. Is this correct? Are those filesystems on the AoE disks clustered or in local mode? Does the backup server ever mount those filesystems? Are the errors you see on the live disks (the 10Gbps iSCSI ones) or on the backup disks (AoE)? What machine sees the errors, the two servers or the backup server? -- It is not the function of our government to keep the citizen from falling into error; it is the function of the citizen to keep the government from falling into error. - Robert H. Jackson Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No space left on device error, older kernels?
On Thu, Oct 28, 2010 at 12:09:59AM +0300, Antonis Kopsaftis wrote: Even if 2.6.18 is a too old kernel, its then kernel thats its been used by the current production running versions (5.x) of redhat enterprise distros (and all his branches: centos, SL , ...). You can easily get the Unbreakable kernel on those distros. We understand your concern, as there are a lot of people running into this issue, but we feel that running the Unbreakable kernel is far less risky than backporting features of this size. If it was a simple fix, we wouldn't be having this conversation. ;-) Joel -- Where are my angels? Where's my golden one? And where is my hope Now that my heroes are gone? Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2_delete_inode kernel bug
On Thu, Oct 28, 2010 at 12:19:14PM -0200, an...@digirati.com.br wrote: Em 28/10/2010, Joel Becker lt;joel.bec...@oracle.comgt; escreveu: gt; Where are the disks?nbsp; I'm guessing they're on an iSCSI server at gt; the other end of the 10Gbps interface, but what is the technology there? There are two servers, each with 16 SATA local disks. The servers areconnected to each other through the 10Gbps interface. The disks arearrayed in RAID1 pairs done via hardware, and therefore the OS sees 8disks. These 8 disks are configured in an active-active DRBD setupbetween the two machines. The DRBD devices are formatted as OCFS2.The OCFS2 cluster configuration is done using the same 10Gbpsinterface used by DRBD. snip The backup server never mounts its filesystems. The errors I reportedalways happen on the two servers, never on the backup server. I havealso run fsck on all backup filesystems, but never found any errors tobe corrected. I'm starting to think that DRBD isn't keeping a consistent view of the devices between your servers. Joel -- Life's Little Instruction Book #157 Take time to smell the roses. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No space left on device error, older kernels?
On Thu, Oct 28, 2010 at 04:02:47PM +0300, Antonis Kopsaftis wrote: I search for info about the unbreakable kernel, and by the info that i found i came up to this conclusions: 1. To use unbreakable kernel you have to upgrade your distro to oracle linux first. This upgrade is only available for Redhat linux and not the free branches(px centos). 2. To upgrade to oracle linux you have to BUY a support contract. As Herbert pointed out, you can get the packages from public-yum.oracle.com right now. The Unbreakable kernel should install on your EL5-based system with perhaps a couple dependent package upgrades. If you want us to answer phone calls about it, you need a support contract, but given that you are running SL and not RHEL/OL, I don't think you care about that. Joel -- Life's Little Instruction Book #267 Lie on your back and look at the stars. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 + iscsi: another node is heartbeating in our slot (over scst)
On Wed, Oct 20, 2010 at 10:17:49AM +0200, Adam Skowronek wrote: I'm building a cluster containing two nodes with seperate common storage server. On storage server i have volume with ocfs2 fs which is sharing this volume via iscsi target. When node connected to the target i can local mount volume on node and using it. I'm not sure what you mean here. I think you are saying: - The host storage_machine has a disk exported via scst. - The hosts node1 and node2 see that disk via iscsi. - The host nod1 can mount the ocfs2 volume and use it just fine. Unfortunately. on storage server ocfs2 logged to dmesg: Oct 19 22:21:02 storage kernel: [ 1510.424144] (o2hb-2283B3335E,4427,0):o2hb_do_disk_heartbeat:770 ERROR: Device sda1: another node is heartbeating in our slot! Oct 19 22:21:02 storage kernel: [ 1510.428600] o2net: connection to node node-2 (num 0) at 192.168.1.69: shutdown, state 7 It looks like you have also mounted the ocfs2 volume on the storage_machine host via its regular disk. You cannot do this, because the iscsi software caches data that the ocfs2 driver won't see in /dev/sda1. If you want to mount the volume on storage_machine as well, you will have to export the disk back to itself via iscsi. Joel -- Not everything that can be counted counts, and not everything that counts can be counted. - Albert Einstein Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] kernel BUG at fs/dlm/lowcomms.c:647!
/lib/debug/lib/modules/2.6.32-100.0.19.el5/vmlinux DUMPFILE: /var/var/crash/127.0.0.1-2010-10-18-16:42:07/vmcore [PARTIAL DUMP] CPUS: 64 DATE: Mon Oct 18 16:41:48 2010 UPTIME: 00:15:00 LOAD AVERAGE: 1.06, 1.22, 1.65 TASKS: 1594 NODENAME: chili0 RELEASE: 2.6.32-100.0.19.el5 VERSION: #1 SMP Fri Sep 17 17:51:41 EDT 2010 MACHINE: x86_64 (1999 Mhz) MEMORY: 64 GB PANIC: kernel BUG at fs/dlm/lowcomms.c:647! PID: 27062 COMMAND: dlm_recv/34 TASK: 880c7caa00c0 [THREAD_INFO: 880c77c6a000] CPU: 34 STATE: TASK_RUNNING (PANIC) crash bt PID: 27062 TASK: 880c7caa00c0 CPU: 34 COMMAND: dlm_recv/34 #0 [880c77c6b910] machine_kexec at 8102cc9b #1 [880c77c6b990] crash_kexec at 810964d4 #2 [880c77c6ba60] oops_end at 81439bd9 #3 [880c77c6ba90] die at 81015639 #4 [880c77c6bac0] do_trap at 8143952c #5 [880c77c6bb10] do_invalid_op at 81013902 #6 [880c77c6bbb0] invalid_op at 81012b7b [exception RIP: receive_from_sock+1364] RIP: a02406c3 RSP: 880c77c6bc60 RFLAGS: 00010246 RAX: 0030 RBX: 8810774b8d30 RCX: 88087c4548f8 RDX: 0030 RSI: 880876dce000 RDI: 81398045 RBP: 880c77c6be50 R8: R9: 880c77c6b900 R10: 880c77c6b8f0 R11: 0030 R12: 0030 R13: 8810774b8d20 R14: 880c7caa00c0 R15: a023ecca ORIG_RAX: CS: 0010 SS: 0018 #7 [880c77c6be58] process_recv_sockets at a023ecea #8 [880c77c6be78] worker_thread at 81071802 #9 [880c77c6bee8] kthread at 810756d3 #10 [880c77c6bf48] kernel_thread at 81012dea ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users -- Every new beginning comes from some other beginning's end. Joel Becker Senior Development Manager Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] partition offset/alignment on SAN devices.
On Fri, Jul 09, 2010 at 04:29:59PM +0200, thomas.zimol...@bmi.bund.de wrote: parted -- /dev/sdc mklabel gpt mkpart ext2 2048s -34s Cool, I'll have to put that one somewhere into my Library :-) This gives you a partition aligned on a 1MiB boundary, which I figured was enough for any reasonable current or future alignment need. Hm, as far as I understood, it's not a matter of reserving enough space for alignment, but the alignment has to exactly match the stripe size of your shared disk, in our case, an EMC CLARiion LUN. Note that some of the storage topology stuff appearing in sysfs will allow partitioners and mkfs tools to smartly align their disks going forward. Joel -- Life's Little Instruction Book #173 Be kinder than necessary. Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] partition offset/alignment on SAN devices.
On Thu, Jul 08, 2010 at 01:18:30PM -0400, Enrique Sanchez wrote: I've been getting requests from the users that they want the OCFS2 filesystems aligned to 64k boundaries, however, when I began the project, the dummy of me, created the OCFS2 filesystem on top of the whole disk device (no partition) for example, one of the SAN devices would show up as /dev/dm-10, I created the filesystem using the following command: mkfs.ocfs2 -L SOMELABEL /dev/dm-10 is this supported, am I safe or did I just shoot myself in the foot, mouth brain? It will certainly work. The big danger is that you might fdisk the drive without thinking, stomping on the ocfs2 volume. Joel -- Copy from one, it's plagiarism; copy from two, it's research. - Wilson Mizner Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 'No space left on device' error with plenty of space.
On Fri, Jun 25, 2010 at 01:49:28PM -0400, Jason Price wrote: I've updated the bug #1263. I am still periodically getting ENOSPC errors out of this file system. Apparently bugzilla isn't accepting attachments at this moment, so I'll attach the current stat_sysdir.sh output. At this point, I can't continue having these errors in production. Next week, I'll begin migrating away from OCFS2. If you have any ideas for tests or remedy's, I'll be happy to run some tests before I begin the migration. The solution is in the mainline Linux kernel now. We're working on bringing it to production releases. Joel -- People with narrow minds usually have broad tongues. Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 'No space left on device' error with plenty of space.
On Fri, Jun 25, 2010 at 02:03:59PM -0400, Eric Raskin wrote: In the meantime, I'm planning on moving data off the file system, re-creating it, then moving the data back on again. I'm hoping that will defragment it enough to allow it to continue working for a while. Does anyone know if I'm wasting my time? Will it just have the same problem when I put the data back? That will help for a little while, but you will run into the problem as you get nearer full. The solution is in mainline, we're working on bringing it to production releases. Joel -- Joel's Second Law: If a code change requires additional user setup, it is wrong. Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2-tools Debian packaging
On Fri, Jun 25, 2010 at 11:07:30AM +0200, Jeremy Lainé wrote: Just a heads up to let you know I uploaded ocfs2-tools 1.4.4 to Debian yesterday. It will take a couple of days to make its way into Debian/unstable because I introduced two new packages following a user's request: - ocfs2-tools-pacemaker : package containing ocfs2_controld.pcmk - ocfs2-tools-cman : package containing ocfs2_controld.cman Thank you! You are absolutely correct to separate out the controld packages. We intended them to work that way. I hope that you've kept the base ocfs2-tools package from requiring the complicated dependencies of cman and pacemaker. = some time ago I suggested the debian directory be dropped from the ocfs2-tools source tree, as it is very much out of date. Proskurin Kirill's recent email on this mailing list seems to confirm this is confusing to users. You're probably right about this. Would you be ok with a vestigial debian/rules that specifies the URL to get the current debian/ directory? B/ ocfs2-tools's source code does not compile with GCC 4.5, we had to apply the following patch: http://svn.debian.org/wsvn/collab-maint/deb-maint/ocfs2-tools/trunk/debian/patches/gcc45-ftbfs.patch Well, that's less fun, but I suppose I can see the problem. C/ the manpages for mkfs.ocfs2 and o2image contain lines which are too long. I apply the following patch to address this: http://svn.debian.org/wsvn/collab-maint/deb-maint/ocfs2-tools/trunk/debian/patches/shorten-manpage-lines.patch Gonna NAK this one, because the r...@node is inconsistent with the rest of the examples. Better to keep the hostname and wrap the line. D/ Debian has switched to a dependency-based init system, so the LSB tags in the init scripts are of vital importance now. While packaging ocfs2-tools 1.4.4 I encountered some issues: - the ocfs2 init script does not list $local_fs in it's Required-Start and Required-Stop. I think this is wrong because the init script makes use of /var, which requires the local FS to be mounted. The hard part about the LSB tags is that many distros break when you fill certain things in. That said, I can't see why $local_fs would be a problem. - the ocfs2 and o2cb init scripts state that they should be started in runlevels 2, 3, 5 and don't specify when they should be stopped at all. In Debian, we have always started these two scripts in the S runlevel, and stopped them in the 0 and 6 runlevels (halt and reboot), so I have adjusted the LSB tags to reflect this. What is your opinion on this? The problem is that other distros do not have the network and block devices up until the middle of the regular runlevels. EL5, for example, starts the network at S10 in runlevel 3, and iscsi at S13. At least they start mdadm in rc.sysinit ;-) I don't have any problems with Debian choosing runlevel S, but our scripts need to work on most systems. While the $local_fs change probably wants to come upstream, I think the runlevel S patch should remain in your vendor patches. E/ I have recently closed a crash report against ocfs2-tools in Debian's bug tracking system because it is triggered by an erroneous use of the cluster.conf config file, but it might still be worth investigating. If two clusters are defined in the cluster.conf file, a crash will occur due to a double free: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=424922 We definitely support more than one configuration in cluster.conf, even though we only support one _active_ configuration at a time. I think this bug has been fixed, either in the master branch or in a queued patch. Joel -- Viro's Razor: Any race condition, no matter how unlikely, will occur just often enough to bite you. Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] New Install
On Thu, Jun 24, 2010 at 05:11:36PM -0500, Bryan McGuire wrote: I am trying to setup OCFS2 on CentOS. I have downloaded the correct rpms. When trying to install rpm like rpm -Uvh ocfs2- tools-1.4.4-1.el5.x86_64.rpm I get the following. This is the RPM for 64-bit intel machines. My server is up to date with Package glibc-2.5-49.el5_5.2.i686 already installed and latest version But this is the glibc RPM for 32-bit intel machines. Is your machine 64-bit? If so, make user that glibc-2.5-49.el5_5.2.x86_64 or the like is installed. If your machine is 32-bit, you want the ocfs2-tools 32-bit RPM. Joel -- When I am working on a problem I never think about beauty. I only think about how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong. - Buckminster Fuller Joel Becker Consulting Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Kernel independent OCFS2 packages for RHEL, Scientific Linux and CentOS
On Fri, Jun 18, 2010 at 10:12:43AM +0200, Dag Wieers wrote: I would like to announce a set of OCFS2 kABI-tracking kernel module packages for RHEL5, Scientific Linux 5 and CentOS-5 and kernels. These packages have been introduced into the ELRepo testing repository (http://elrepo.org/). Hey Dag, Thank you for taking an interest in ocfs2. You've always been a great resource for folks working on el-like distributions and their limited package archives. I have a couple of questions, if you don't mind. First, do you have any provisions in your package dependencies to limit packages to certain update releases? What I mean is, can one package be installed and used against EL5GA, EL5U1, ..., EL5U5? Or do you have some mechanism to isolate a package to only a particular update release? I bring this up because there are often ABI changes that are not detectable via modversions or any other automatic method we currently have. A specific example is ocfs2 1.4, which will crash on EL5GA or EL5U1. They do not contain mainline commit 349457ccf2592c14bdf13b6706170ae2e94931b1 Allow file systems to manually d_move() inside of -rename(), which ocfs2 1.4 requires. This was introduced in EL5U2. No function signature is changed, so modversions will not catch the difference. On another note, I was wondering if you have any language mentioning that these are not the officially supported packages? I want to be clear here. We develop ocfs2 to be used as widely as possible, and we will never shortchange mainline or a packager like yourself when it comes to help on ocfs2-users and other community forums. ocfs2 is something we're very proud of, and we back that to the best of our ability. I'm just want to have a clear story for our customers. Thanks again, Joel -- But all my words come back to me In shades of mediocrity. Like emptiness in harmony I need someone to comfort me. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Kernel independent OCFS2 packages for RHEL, Scientific Linux and CentOS
On Fri, Jun 18, 2010 at 10:35:50PM +0200, Dag Wieers wrote: The only thing we could do is prevent the package to be installed if an older kernel is available, however we thought that was one bridge too far. The best solution is to make the mechanism stronger in such a way that this cannot happen, but we are only using what is available to us. I completely agree that you shouldn't prevent installation just because an older kernel is available. I think the mechanism should instead have, at a minimum, a way to delineate versions to link to. I realize I'm talking about a capability it currently does not have. In fact there is a Driver Backport WorkGroup at the Linux Foundation lead by Jon Masers of Red Hat, and we follow his recommendations. If you think we can or should improve the infrastructure in future updates, I guess it's best to bring those issues up. I think that the mechanism needs to be improved to take into account the myriad ABI and behavior changes that don't affect function signatures. I've never managed to completely solve them myself (otherwise you'd see ocfs2 packaged differently), but I know there are measures that are more stringent that what exists now. Of course, I am sympathetic with corporate support issues, so please provide me with the specifics and rationale for any changes you desire and I will discuss it with the rest of the team. Well, I don't have a specific answer right now. I guess I'll have to look at your packages and page directly to see what I can suggest. Joel -- print STDOUT q Just another Perl hacker, unless $spring - Larry Wall Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Kernel independent OCFS2 packages for RHEL, Scientific Linux and CentOS
On Fri, Jun 18, 2010 at 03:08:34PM -0700, Joel Becker wrote: On Fri, Jun 18, 2010 at 10:35:50PM +0200, Dag Wieers wrote: Of course, I am sympathetic with corporate support issues, so please provide me with the specifics and rationale for any changes you desire and I will discuss it with the rest of the team. Well, I don't have a specific answer right now. I guess I'll have to look at your packages and page directly to see what I can suggest. Oh, my. You provide 'ocfs2' and directly override the existing modules as delivered. Am I correct that, should a person have our ocfs2 package for 2.6.18-92.el5 installed and your kmod package, they will actually load your kmod module instead? Is there any way to prevent this? Joel -- When ideas fail, words come in very handy. - Goethe Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Kernel independent OCFS2 packages for RHEL, Scientific Linux and CentOS
On Sat, Jun 19, 2010 at 02:11:33AM +0200, Dag Wieers wrote: On Fri, 18 Jun 2010, Joel Becker wrote: Oh, my. You provide 'ocfs2' and directly override the existing modules as delivered. Am I correct that, should a person have our ocfs2 package for 2.6.18-92.el5 installed and your kmod package, they will actually load your kmod module instead? Is there any way to prevent this? Well, our aim is to prevent that both packages are installed at the same time, because we don't want the user to not know for sure which package is being used. Unfortunately since the Oracle packages contain a kernel-version in the package name, we would have to Conflict or Obsolete every kernel released by Red Hat (and for each new released kernel, release our own kmod package), which obviously goes directly against our goals. I agree that preventing the combination is the least confusing solution. I also agree that specifying the conflict for each possible version is a non-starter. If you see a way around that, I am happy to implement that. But at the moment I don't see an easy way to do that, unless the official kernel module packages has a set of (kernel-agnostic) virtual provides, honestly I haven't checked that. It provides 'ocfs2'. But so do you. The reason for overriding the ocfs2 module the way we do is to ensure that if an ELRepo package is installed, it has the priority. This is the standardized way to do it with kmod packages, another kmod package would conflict on that file preventing a 'conflict'. The fact that you want to have priority over what shipped in the kernel rpm makes perfect sense. Unfortunately, you're overriding what's in a separate rpm. Beware though that the virtual provides of ocfs2 is just a mechanism to assist users to install ocfs2, there is no other reason. I am open to improvements. I've been trying to think of ways just to know which is loaded (eg, having your ocfs2.ko append -elrepo in the string ocfs2 prints at module load time). But I've got nothing that shines. If I can think of anything, I'll let you know. Joel -- A narcissist is someone better looking than you are. - Gore Vidal Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Adding node with storage to existing ocfs2 FS?
On Sun, Jun 13, 2010 at 04:12:03PM -0400, Jeffrey Layton wrote: I hate to be a total noob but I've been experimenting and I'm a bit stuck. I've got a node with a couple of 500GB drives that I've used to create a logical volume. I formatted the LV with ocfs2 and mounted it and things look correct so far. snip Now I want to take a second node that has some internal storage and add the storage to the ocfs2 pool. Here's where I struggle. Are the basic steps the following? ocfs2 is a shared-storage cluster filesystem. The disks need to be connected to all nodes that want to share them. ocfs2 does not share internal disks over the network by itself. Joel -- Heav'n hath no rage like love to hatred turn'd, nor Hell a fury, like a woman scorn'd. - William Congreve Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] fsck.ocfs2 using huge amount of memory?
On Thu, May 20, 2010 at 06:00:19PM -0700, Ulf Zimmermann wrote: We are setting up 2 new EL5 U4 machines to replace our current database servers running our demo environment. We use 3Par SANs and their snap clone options. The current production system we snap clone from is EL4 U5 with ocfs2 1.2.9, the new servers have ocfs2 1.4.3 installed. Part of the refresh process is to run fsck.ocfs2 on the volume to recover, but right now as I am trying to run it on our 700GB volume it shows a virtual memory size of 21.9GB, resident of 10GB and it is killing the machine with swapping (24GB physical memory). Can anyone enlighten what is going on? How big are your filesystems? Can we get the output of debugfs.ocfs2 -R 'stats' /dev/xxx? Recent fsck.ocfs2 knows how to build its own I/O cache for significant speed improvements. It only tries to get as much cache as the filesystem actually needs, and no more than half of system memory. That's why I'm asking for your filesystem size - I'm guessing you have more than 12GB of used space on the filesystem, so fsck.ocfs2 is trying to grab that much cache. Joel -- In the room the women come and go Talking of Michaelangelo. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] memory leak
On Thu, Apr 15, 2010 at 12:31:02PM +0200, Kristiansen Morten wrote: I discovered our four node cluster running on RedHat EL5, Ocfs2 1.2.6 and Oracle 10.2.0.3 have memory leak. I suspect ocfs2, but I could be wrong. I suspect ocfs2 because when we run RMAN backup the free memory goes from 8 GB down to 200 MB. When I umount the ocfs2 backupdisk after the backup is finished, the memory is released again. You don't have a memory leak. Your backup is reading every file into cache in order to process it. This is a normal behavior of filesystem cache. If other processes need memory, the file data will be evicted from cache. I want to test it some more and found a script to test writing to the disk. This script contains a command saying echo 3 /proc/sys/vm/drop_caches. Is this a safe command to run in production? Meaning the cluster and oracle database is running. Or should I run the sync command pre to this command? Or should I never run this command in a production environment? I'm afraid that this command will free up memory not written to disk yet. And therefore I would get into trouble in my production environment. The script I want to test looks like this and is captured from this mailing list: The command echo 3 /proc/sys/vm/drop_caches is safe to run in a production environment. However, I don't think you want to do so. What it does is evicts all file data from cache. Don't worry, you won't lose any data. But you will not only evict the data from the backup volume, you will also evict data from any other file. This may slow down some processes as they have to re-read their data. Why do you feel there is a problem? Is it just that the free memory number shrinks? That's not a problem, as stated above. Is there some other affect on the system? Joel -- Here's something to think about: How come you never see a headline like ``Psychic Wins Lottery''? - Jay Leno Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] POSIX locks supported?
On Wed, Mar 10, 2010 at 01:21:32PM -0800, Sunil Mushran wrote: For Enterprise kernels, currently only SLES11 (HAE) satisfies both requirements. (RH)EL6, when it ships, should also offer the same. For non-enterprise distros, you will have to see the state of the clusterstack in that distro. For Fedora, the upcoming FC13 will be your best bet. For ubuntu, the upcoming 10.04 looks promising. https://wiki.ubuntu.com/ClusterStack/LucidTesting#Pacemaker,%20drbd8%20and%20OCFS2%20or%20GFS2 Sunil gives a good summary. I believe the latest opensuse also has all the support needed. I think FC12 has the right cluster bits, but I'm unsure if it has a new enough ocfs2-tools. Joel -- Life's Little Instruction Book #43 Never give up on somebody. Miracles happen every day. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No Space left on the device.
On Fri, Mar 05, 2010 at 08:33:34AM -0600, Brian Kroth wrote: As mentioned in the bug (didn't think it was a proper place for discussion) I'm also curious more generally about backporting these fixes to the 2.6.32 kernel since it's been designated long term stable. Is that responsibility just on the individual distro's kernel maintainer or are the OCFS2 devs planning on submitting fixes to the mainline 2.6.32 tree? Who 'designated' it long term stable? I'm just wondering who we should send our patches to ;-) Joel -- Life's Little Instruction Book #232 Keep your promises. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No Space left on the device.
On Fri, Mar 05, 2010 at 10:23:19AM +0530, Aravind Divakaran wrote: My ocfs filesystem has 270gb free space. What version of ocfs2 do you have? What kernel is it running on? Joel -- I'm so tired of being tired, Sure as night will follow day. Most things I worry about Never happen anyway. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No Space left on the device.
On Fri, Mar 05, 2010 at 04:12:51PM -0600, Brian Kroth wrote: Fair enough. Here's the most authoritative source [1] [2] I can find, though a quick google on long term stable kernel produces a number of other results [3]. [1] http://lwn.net/Articles/370236/ [2] http://www.kroah.com/log/linux/stable-status-01-2010.html [3] http://www.fabian-fingerle.de/2010-02-23.233 Ok, its sta...@kernel.org. We'll see what comes up. Joel -- If at first you don't succeed, cover all traces that you tried. -Unknown Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] High load average on Apache Cluster with drbd + ocfs2
On Wed, Mar 03, 2010 at 11:04:48AM +0100, Andreas Kossmann wrote: I have an enviroment with 2 Debian 5.0 servers. Kernel is 2.6.26-2-amd64. I have installed drbd-8.0.14 and ocfs2-tools 1.4.1. It is an Active/Active WebCluster with Apache. The 2 servers write to the same log files. It's always better if they can write to separate log files, but I'm going to proceed as if that won't change, because we should handle both. First I thought the problem may be drbd, but I have read many problemes with ocfs2 and apache load average. Doesn't mean it's not drbd ;-) Let's see what I can find out. The curios thing is that the load is often very high at times where request are very small ( eg. 11:00 PM ) The problem may or may not be related to apache's volume of requests. I've disconnected the second webserver from the network and checked the filesystem. A few bitmap errors occured and i repaired them. Then I changed the drbd config so, that only webserver 1 is primary and the webserver 2 is secondary. So webserver 2 cannot write to the device. Wait, you checked the filesystem while webserver 1 was still running? The errors you found are dirty state from webserver 1. They would not occur if webserver 1 was cleanly unmounted. After I connect the webserver 2 to the network again and the sync from the primary starts. The load on webserver 1 is going 100. What does it look like after the sync is done? Does performance get better? Is there any solution for the ocfs2 load problem with apache? I don't know if this is drbd keeping the volumes in sync or an ocfs2 issue. Are you sure it is the log files that cause the problem? Can you get me debugfs.ocfs2 output for those files? For example, if your filesystem is on /dev/drbd1, is mounted at /web, and your log file is /web/logs/error_log, then we want the output of debugfs.ocfs2 -R 'stat /logs/error_log' /dev/drbd1. Please do this for each log file and send it along to me. Joel -- Can any of you seriously say the Bill of Rights could get through Congress today? It wouldn't even get out of committee. - F. Lee Bailey Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] High load average on Apache Cluster with drbd + ocfs2
On Wed, Mar 03, 2010 at 09:47:08PM +1100, Brad Plant wrote: Turns out I'd hit a free space fragmentation problem. While df reported I had heaps of free space (50% from memory!), I couldn't write (echo ) to the log files on the problem web server. Note that you'll find you can still create small files and append to small files, but not the larger apache log files. The fact that it happens late at night was very confusing, but eventually made sense. As the day goes on, the log files get bigger and bigger pieces of contiguous free space are required to extend the file. Eventually, a contiguous piece of free space cannot be found and your writes will start to fail. This is my assumption, which is why I asked for the debugfs output. This will give us layout information. There are actually two fragmentation issues here. First, free space is fragmented. This prevents us from allocating metadata blocks. These metadata blocks are used in the housekeeping of files with many data extents. This fits your description of lots of free space, yet files are unable to grow. The second issue is fragmentation of the actual file. Log files are written in small hunks. As each node takes turns extending the log file, the hunks come from disparate parts of the disk. This means the log files have lots of small extents instead of a few large ones. This is why we recommend separate log files for each node. Separate log files would grow in a much more contiguous fashion, allowing fewer, larger extents. With fewer extents, they would need fewer metadata blocks to grow. A *partial* fix went into 2.6.33. It's partial because it doesn't fix the free space fragmentation issue but rather allows the problem node to steal some free space from the node that is still ok. All it does is prolong the problem a little such that writes will start to fail on both nodes at the same time. This alleviates the first problem some, as you point out. It does nothing for the second problem. We have code coming that will help files like these allocate larger extents, reducing their fragmentation. However, it will never solve the alternating extend problem that comes from having both nodes extend the same log file. Joel -- Reality is merely an illusion, albeit a very persistent one. - Albert Einstien Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] High load average on Apache Cluster with drbd + ocfs2
On Wed, Mar 03, 2010 at 11:24:57PM +1100, Brad Plant wrote: I forgot to mention that I was actually using a separate log file per web server. Then it's free space fragmentation for you. Like Sunil said, we are working on it. Joel -- The only way to get rid of a temptation is to yield to it. - Oscar Wilde Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] I have questions regarding Fencing
On Tue, Mar 02, 2010 at 05:09:21AM -0500, Enrique Sanchez wrote: During my test (take Node0 down cold turkey) Node1 hung pretty badly, is this something expected?? What did you do to take it down? Power off? Node1 should take around 90 seconds to notice (depending on your heartbeat timeout settings), and then it should start recovery. I flip the power off, on almost any test Node1 crashes as well. I would expect node1 to survive, but perhaps I'm being too hopeful? Maybe someone else knows? I don;t understand why you don't have plans to add a referential IP address to find who's on the network and who isn't, while you got a Once you're off the disk, who is on the network is irrelevant. The only thing that is interesting is which nodes can access the filesystem. Quorum is interesting when you are still talking to the disk but cannot be reached via network. Quorum is deciding which group of nodes can stay on the disk and which group must leave. It definitely takes into account who a node can see via the network. In your case above, where you powered off node0 and node1 still crashed, I would expect node1 to notice node0 is off the disk as well as off the network and determine that it is the only surviving node. That's why I'm wondering if any of my colleagues have some input. Joel -- Life's Little Instruction Book #173 Be kinder than necessary. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] more than one cluster in cluster.conf
On Wed, Feb 24, 2010 at 10:19:56AM +0100, Werner Flamme wrote: I wonder how to configure more than one OCFS2 cluster in /etc/ocfs2/cluster.conf. Is this possible? You can define more than one cluster, but only one cluster may be running at a time. The reason for my question is that I have one OCFS2 cluster shared between 3 hosts and now have to integrate these 3 nodes into another cluster with (up to now) 5 nodes, so they will become 8 nodes total. Do I have to drop the cluster I use up to now in order to join the new one? You need to have the same configuration, including cluster name, on all eight nodes. Joel -- Life's Little Instruction Book #80 Slow dance Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] renaming a OCFS2 cluster
On Wed, Feb 24, 2010 at 10:31:15AM +0100, Werner Flamme wrote: I ran o2cb configure and configured a cluster named ocfs2. Lazy me, I did not invent a new name. That's the normal case, so don't worry about being lazy ;-) I changed /etc/ocfs2/cluster.conf to the new name and ran o2cb configure again, but to no avail: - ---snip--- Starting cluster sapcluster: Failed o2cb_ctl: Memory allocation failed while setting cluster name Stopping cluster sapcluster: OK - ---pins--- Did you stop o2cb before renaming? The o2cb cluster cannot be changed on the fly. Joel -- f/8 and be there. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Any chance using OCFS2 as a distributed file system (without a shared disk)?
On Tue, Jan 26, 2010 at 07:19:01AM +0100, Christian Baun wrote: is it possible to use OCFS2 as a distributed file system? I don't have a shared disk in my cluster. :-( Nope. ocfs2 is a shared disk filesystem. You can have one machine provide iSCSI targets and all the other machines use those iSCSI disks. Joel -- Life's Little Instruction Book #30 Never buy a house without a fireplace. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] More questions: Max partition size? Throughput?
On Wed, Dec 23, 2009 at 02:31:32PM -0800, Patrick J. LoPresti wrote: 1) According to the FAQ (http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#LIMITS), the size limit for a single OCFS2 partition is 16 TB. The FAQ goes on to say, This block addressing limit will be relaxed in future software. At that point the limit becomes addressing clusters of 1MB each with 32 bits which leads to a 4PB file system. Has this future software been released? If so, what version of OCFS2 and/or the Linux kernel do I need? Any version where ocfs2 uses jbd2 instead of jbd for journaling. This first appeared in Linux 2.6.27. You also need a recent enough ocfs2-tools to enable large journals (-J block64). Joel -- Win95 file and print sharing are for relatively friendly nets. - Paul Leach, Microsoft Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Combining OCFS2 with Linux software RAID-0?
On Thu, Dec 10, 2009 at 11:40:43AM -0800, Sunil Mushran wrote: It should work. All ocfs2 needs is a stable shared storage that is consistent across all nodes all the time. Patrick, What Sunil says is correct; your solution should work. The big issue with software RAID is that many forms of RAID can change the storage topology automatically. Think of failing one disk and adding a hot-spare in its place. If you had such a configuration, one node would be running on the hot spare while all the other nodes would still be using the original LUN. Corruption! So RAID 5, RAID 6, even RAID 1 - anything where the RAID software can make an automatic decision that is seen by one node and not the others - is ripe for data loss. RAID 0 and linear concatenation don't have this automatic problem. But they still can cause data loss if a fat-fingered admin accidentally changes the topology on one node while another node is still using it. This is an easier mistake to make than you think. You have to be absolutely sure your procedures never let this happen. This is why our documents say No Software RAID. Hardware RAID doesn't have this problem because it is invisible to the nodes. If you change the hardware, all nodes see the change at the same time. It looks like the same LUN as always. If somehow the software RAID code became cluster aware and coordinated topology changes across the cluster, then ocfs2 would be able to support it. Until that time, we generally have to say No Software RAID; when folks like you come along we say You can do RAID 0, but caveat emptor! Joel -- Life's Little Instruction Book #139 Never deprive someone of hope; it might be all they have. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 2 node OCFS2 clusters
On Mon, Nov 16, 2009 at 12:57:26PM -0800, Luis Freitas wrote: For CRS there is no need for a dedicated switch, only a need for using switches instead of cross cables. Although it is not recommended you can use the same switch for the public and private networks, using different vlans. The network status can be checked by the link status, that is what CRS does, and also by pinging the router. This information could be used as part of the heuristics to decide which node should survive. Of course it doesn't cover all network topologies, but it is sure better than node 0 always survive when network is down. How does node1 know the link status of node0? That's the fundamental problem of self-fencing. You have to assume the other guy is going to make a predictable decision. node1 has no way of knowing that node0 is going to reboot. What if the switch chip between node0 and node1 is down? Both see their links as up? Which link status do you check? Do you consider your link status down if the interconnect link is down or all links are down? What if you have a separate public and private network, node1 has lost public network and node0 has lost private? In the current scheme, node1 resets and node0 continues talking on the public. The web service is working. In your scheme node0 resets and node1 can't talk to the public. The web service is down. Self-fencing is hard and never perfect. The two node case is the worst because there is no difference between a majority of nodes and all nodes. The easiest way to alleviate it is to add a third node. Now you have a majority and much easier decisions. I see this as a problem in a RAC implementation, since there are two different cluster stacks running (O2CB and CRS), they are not integrated and take decisions with a different heuristic. For me it would make more sense if they were integrated and one of the cluster stacks was in control, in the same way that happens when you use RAC with Veritas/HP ServiceGuard/Sun Cluster Suite, or OCFS2 with heartbeat2, for example. The standard install documentation makes sure that o2cb and crs behave well together. crs won't make a decision before o2cb does, thus giving o2cb precedence. Joel -- Joel's First Law: Nature abhors a GUI. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 2 node OCFS2 clusters
On Mon, Nov 16, 2009 at 08:55:38AM -0800, Luis Freitas wrote: If this is true then I would sugest OCFS2 is not taking the best decision in this scenario. It's doing the best it can. See, each node knows the other is alive because of the disk heartbeat. Thus they know the error is network related. But with only two nodes, there is no way to determine who has the better network. The node that still has network connectivity should survive instead of the lowest node number. Oracle CRS has heuristics to detect if the network is down and in this scenario the node that lost network conectivy is evicted. That is why it is required to use a switch between the two nodes, instead of a cross cable. ocfs2 is a general purpose filesystem that allows a myriad of network configurations. It has never required a dedicated network switch, nor will it, so relying on that isn't possible. Joel -- The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts. - Bertrand Russell Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 vs Oracle ASM CFS
On Thu, Sep 03, 2009 at 07:21:55AM -0500, Jeremy Schneider wrote: 3) ACFS shapshots -- or are those a feature of DVM? Seems like a great feature - isn't xfs the only other currently released filesystem to support this? Any plans to add snapshots to the OCFS2 roadmap? ocfs2 will be able to snap inodes via the reflink(2) system call with the addition of the refcount tree feature. The work is done and is scheduled to go into 2.6.32. Joel -- When ideas fail, words come in very handy. - Goethe Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] git checkout on an ocfs2 filesystem
On Mon, Aug 31, 2009 at 09:18:29PM -0700, James Harrell wrote: -- Warning - threadjack in progress. But I think it might be related. -- Interesting. Could this be similar to the case that I'm seeing with no space left on device? Here's my uneducated assumption: - at this point I believe some form of I/O error or interrupt causes ocfs2 to error out - the remount read only effect silently kicks in (no log message though) If you remount read only, you really should see an error message. Are you getting Read-only filesystem errors at this point? - now file operations return no space left on device, but my device is showing 2% use See, if it were readonly, you'd get read-only filesystem. The rm -rf on a very large dir would do presumably do this. I also hit this when I do a local rsync or untar of very large directory trees. EMC also recommends the qlogic HBA be set to Interrupt after every I/O completion. Could this cause a race condition? All of these have interrupts in common. Think setting nointr as a mount option would help here? If you have, say, some large file you're removing and an interrupt prevents it from getting cleaned up, then yes, that could be a problem. However, we block signals around that file deletion to prevent this very problem. Maybe there's another spot we missed. If you can reproduce this, testing with nointr would help a lot. Joel -- Life's Little Instruction Book #450 Don't be afraid to say, 'I need help.' Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] git checkout on an ocfs2 filesystem
On Sun, Aug 30, 2009 at 08:19:08PM -0500, Nathaniel Griswold wrote: Has anyone here had problems with git checkouts on ocfs2? Oh, boy, this is wacky. On a freshly created filesystem, i'm getting a bunch of weird errors as below, but only if i mount the filesystem on multiple nodes: [r...@node-2 tmp]# mkfs.ocfs2 -L testvol /dev/sdm [r...@node-2 tmp]# mount /dev/sdm /tmp/m [r...@node-1 tmp]# mount /dev/sdm /tmp/m [r...@node-1 tmp]# cd m [r...@node-1 m]# git clone git://git.kernel.org/pub/scm/git/git.git Initialized empty Git repository in /tmp/m/git/.git/ remote: Counting objects: 104239, done. remote: Compressing objects: 100% (29045/29045), done. remote: Total 104239 (delta 75710), reused 101527 (delta 73491) Receiving objects: 100% (104239/104239), 25.43 MiB | 1299 KiB/s, done. Resolving deltas: 100% (75710/75710), done. error: git checkout-index: unable to create file contrib/remotes2config.sh (File exists) error: git checkout-index: unable to create file t/t4013/diff.diff_--dirstat_master~1_master~2 (File exists) I got it on this particular file myself. I was using 2.6.18-128.el5 on a ppc64. 1.4.2 of ocfs2 as well. Can you file a bug at http://oss.oracle.com/bugzilla? Include all the info you have in your emails. Thanks! Joel -- This is the end, beautiful friend. This is the end, my only friend the end Of our elaborate plans, the end Of everything that stands, the end No safety or surprise, the end I'll never look into your eyes again. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] git checkout on an ocfs2 filesystem
On Mon, Aug 31, 2009 at 12:16:36PM -0700, Joel Becker wrote:
On Sun, Aug 30, 2009 at 08:19:08PM -0500, Nathaniel Griswold wrote:
Has anyone here had problems with git checkouts on ocfs2?
Oh, boy, this is wacky.
No, it's extra wacky:
5441 lstat64(t/t6015-rev-list-show-all-parents.sh, 0xffc2e318) = -1 ENOENT
(No such file or directory)
5441 lstat64(t, {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
5441 open(t/t6015-rev-list-show-all-parents.sh,
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0777) = ? ERESTARTSYS (To be restarted)
5441 --- SIGALRM (Alarm clock) @ 0 (0) ---
5441 sigreturn() = ? (mask now [])
5441 open(t/t6015-rev-list-show-all-parents.sh,
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0777) = -1 EEXIST (File exists)
5441 write(2, error: git-checkout-index: unabl..., 100) = 100
How on earth is userspace seeing ERESTARTSYS? Did someone forget to
-ive it?
Joel
--
Reality is merely an illusion, albeit a very persistent one.
- Albert Einstien
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.bec...@oracle.com
Phone: (650) 506-8127
___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] git checkout on an ocfs2 filesystem
On Mon, Aug 31, 2009 at 12:39:02PM -0700, Joel Becker wrote: On Mon, Aug 31, 2009 at 12:16:36PM -0700, Joel Becker wrote: 5441 open(t/t6015-rev-list-show-all-parents.sh, O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0777) = ? ERESTARTSYS (To be restarted) The workaround is to mount with the 'nointr' option. Joel -- Vote early and vote often. - Al Capone Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] git checkout on an ocfs2 filesystem
On Tue, Sep 01, 2009 at 09:32:14AM +0800, Tao Ma wrote: 5441 open(t/t6015-rev-list-show-all-parents.sh, O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0777) = ? ERESTARTSYS (To be restarted) 5441 --- SIGALRM (Alarm clock) @ 0 (0) --- 5441 sigreturn() = ? (mask now []) 5441 open(t/t6015-rev-list-show-all-parents.sh, O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0777) = -1 EEXIST (File exists) 5441 write(2, error: git-checkout-index: unabl..., 100) = 100 How on earth is userspace seeing ERESTARTSYS? Did someone forget to -ive it? The ERESTARTSYS may happen when we get interrupted from ocfs2_cluster_lock. I met with it when I rm -rf a very large dir and use ctrl+c to stop it when I tested bug 1162. Yeah, I got there. In the git case, we do ocfs2_add_entry(), but then a signal interrupts ocfs2_dentry_lock(). So ERESTARTSYS is returned, but the file has been created. When entry.S goes to retry the open(O_EXCL), it gets EEXIST. I've added code to block signals around that dentry lock call, but there's another place that the git code is triggering. I'm hunting that down. In general, we can't return ERESTARTSYS once we've done something that isn't idempotent. I think we need to audit our code a bit. Joel -- Life's Little Instruction Book #173 Be kinder than necessary. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] How to mount OCFS2 file systems using the EMC Power Path multipath device
On Wed, Jun 10, 2009 at 12:22:03PM -0500, Sridhar Avantsa wrote: OEL 5, OCFS 1.4, using EMC Power Path for multi pathing. I want t mount the OCFS2 file system on the emc power path device. I can mount by UUID, and not have to worry about persistent bindings across nodes. But how do I make sure when it mounts by UUID , that the pwoer path device is used. Any help will be much appreciated. There are two things you can do. The first is to not worry, because PP takes over all devices. That is, if 'sda' and 'sdb' are two paths of 'emcpowera', it doesn't matter whether you open /dev/sda1, /dev/sdb1, or /dev/emcpowera1. PP has hijacked the devices and is used in all cases. Now, a lot of people consider this to be unkosher and wish that PP didn't hijack the devices. Maybe in the future EMC will change this. You can, of course, not worry about it until they do. The other alternative is to use udev to ensure you have a device name that points to the right device. Then you just mount by the device name. For example, a udev rule that only triggers on 'emcpower*' devices, checks that they are ocfs2 volumes, and then creates /dev/mydisks/uuid for them. Joel -- Life's Little Instruction Book #396 Never give anyone a fruitcake. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
On Wed, Apr 29, 2009 at 05:24:11PM +1000, Andrew (Anything) wrote: I just checked max-features, it doesn't include local which means that you still need to create dlm lock in your local node which will cost some delay. You can check whether your volume enable local by command echo 'stats'|debugfs.ocfs2 /dev/sdX|grep Incompat If the volume is mkfsed to used as local disk, you should see local. Otherwise you need to add --fs-features=local to your mkfs. Yeah, 'local' is a feature, but it's not a feature in any default list because we generally expect to be used in a cluster :-) Thanks for your replies Tao and Thomas I still intend on using the filesystem in a 3 node cluster, but when I found it to be 25x slower than a single ext3 part I thought id start troubleshooting at the start. I just did a format of a small partition to give it a quick test anyway, and found that it performs just as well as ext3, in some test even better. Except if I understand correctly, local disables the ability for clustering completely? There are a lot of safety concerns we have to make just in case there are other nodes in the cluster. When in 'local' mode, those checks are all disabled, and that's why we can run as fast as any other local filesystem. In a cluster mode, we have to have those checks enabled, and it does cause a speed penalty. But remember, you're gaining the concurrent access on other nodes. That's the tradeoff. And I think we compare favorably in speed to other cluster filesystems. Joel -- We'd better get back, `cause it'll be dark soon, and they mostly come at night. Mostly. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] O2CB heartbeat: Not active
On Wed, Apr 29, 2009 at 04:39:23PM -0400, McKinley, Reid wrote: In the /var/log/messages, I see this at the time the mount fails: Apr 29 12:01:13 nyclx2 kernel: (12430,0):o2net_check_handshake:1163 node nyclx1 (num 0) at 192.168.0.218: advertised net protocol version 11 but 103 is required, disconnecting Apr 29 12:01:17 nyclx2 kernel: (16953,0):ocfs2_initialize_super:1454 ERROR: couldn't mount because of unsupported optional features (10). Apr 29 12:01:17 nyclx2 kernel: (16953,0):ocfs2_fill_super:578 ERROR: status = -22 Apr 29 12:01:17 nyclx2 kernel: ocfs2: Unmounting device (8,0) on (node 255) It looks like one node is running 1.2 and the other 1.4. You cannot mount the same filesystem with different versions of the driver at the same time. Both versions may understand the disk format, but they cannot coordinate with each other. Specifically, you're seeing the network protocol version mismatch. Upgrade the other node to 1.4.1 and you should be able to mount. Joel -- A good programming language should have features that make the kind of people who use the phrase software engineering shake their heads disapprovingly. - Paul Graham Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 1.4 and openais
On Fri, Apr 24, 2009 at 10:33:56AM -0500, Karl Katzke wrote: I have ocfs2 set up on a cluster running pacemaker and openais. When setting up ocfs2 using service o2cb configure, do I need to specify a different cluster stack besides o2cb? Yes, you need to specify pcmk. Joel -- Drake! We're LEAVING! Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 32000 subdir limit
On Tue, Apr 21, 2009 at 11:00:13AM +0100, Gavin Hamill wrote: Hi, we've just encountered this in a production system - my bad for not thinking about such things ahead of time. We're using ocfs2 version 1.3.3 as it comes packaged with Ubuntu 8.04. I've just noticed http://lkml.org/lkml/2009/3/17/390 - what steps would be required to move a 1.3.3 two-node system to a system capable of addressing more than 32000 subdirs? The short answer is get a version capable of indexed directories, and turn them on. The long answer is that indexed directories were just pushed to mainline this cycle - they're going to be in 2.6.30 - and the tools support isn't complete. So you'd need to get 2.6.30-rc or 2.6.30 when it is released, and you also need a tunefs.ocfs2 that knows how to turn on indexed directories. I don't know when Ubuntu will have a version of the kernel with this capability, and while I know the tools work will be completed soon, the release of the capability will take time and testing. Joel -- I think it would be a good idea. - Mahatma Ghandi, when asked what he thought of Western civilization Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 32000 subdir limit
On Tue, Apr 21, 2009 at 01:03:01PM +0100, Gavin Hamill wrote: On Tue, 2009-04-21 at 03:54 -0700, Joel Becker wrote: The short answer is get a version capable of indexed directories, and turn them on. The long answer is that indexed directories were just pushed to mainline this cycle - they're going to be in 2.6.30 - and the tools support isn't complete. So you'd need to get 2.6.30-rc or 2.6.30 when it is released, and you also need a tunefs.ocfs2 that knows how to turn on indexed directories. Thanks for the quick response - OK looks like we'll pursue an interim route of mod_rewrite and focus on splitting the dir-structure for a future code release :) ocfs2 works really well for us in its present state (the 32000-dir limit aside), and I'm hesistant to start tinkering around with bleeding-edge code! I figured you were more into stable, and that's why I laid out the newness of the feature. It's going to go through the usual heavy testing we do, of course, but it's not there yet. I'm glad ocfs2 is working well for you. Feel free to email with any more issues, we want to keep people like you happy! Joel -- Life's Little Instruction Book #497 Go down swinging. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2 Error: Group Descriptor Mismatch
On Wed, Mar 18, 2009 at 02:05:09PM +, Jari Takkala wrote: Thanks for your response. My comments are inline below. - Joel Becker joel.bec...@oracle.com wrote: First and foremost, can you file a bugzilla bug? This is great detail, and it should be captured there. More comments below. Done, bug 1090 opened, http://oss.oracle.com/bugzilla/show_bug.cgi?id=1090. Thanks. I'm guessing this was a global bitmap cluster group based on the function call chain, but I'd like to verify. Is it possible to get an o2image of the volume for us to look at? o2image should create an image without data so that its safe to send to us. I've run o2image against the snapshot. I can email that directly to you, or if there is a private FTP server you want me to upload it to please let me know. It's 5.2MB compressed, 4.2GB uncompressed. Even though it's metadata, I don't think I'll be able to attach it to the bug report for security reasons. Email it to me and copy srinivas.e...@oracle.com. Don't copy ocfs2-users@oss.oracle.com :-) Joel -- Life's Little Instruction Book #497 Go down swinging. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No space left on device
On Sat, Feb 28, 2009 at 12:09:37PM +, Nuno Fernandes wrote: That's rather odd. What is the blocksize and cluster size of this filesystem? Can you send me the output of 'debugfs.ocfs2 -R stat /.zbr /dev/hda3'? Joel Hi, Well.. we've deleted over 275 000 files and the problem disappeared... at least for now. We'll just have to wait for that amount of files to be added. I wanted to see what .zbr looked like, because my hunch was that you were very fragmented and couldn't allocate enough contiguous space to create more extent blocks. Deleting files would perhaps have freed some in-use extent blocks, thus making it work again. If you run into the problem again, we'll revisit. Joel -- I don't know anything about music. In my line you don't have to. - Elvis Presley Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
On Tue, Feb 24, 2009 at 07:08:00PM +, Tim Hughes wrote: I am looking for a little help with some orphaned files that are taking up diskspace. I deleted a approximately 30 x 1GB mysql-bin-X.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. Others have pointed you to 'fsck -f'. What I want to know is whether you had any nodes die or killed while this filesystem was running? Eg an 'xm destroy' of a vm that had the filesystem mounted. That sort of thing. Joel -- Life's Little Instruction Book #407 Every once in a while, take the scenic route. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] mounting mpath devices
On Tue, Feb 24, 2009 at 02:21:03PM -0700, Masaryk Kevin D wrote: I'm seeing some strange behavior from OCFS2 while trying to mount mpath/dm-multipath devices under RHEL5. Sometimes I can mount the EMC-connected, dual-pathed volumes just fine and sometimes I get device busy errors. I've tried mounting by label and also by explicit /dev/mapper/mpathXpY name with the same unpredictable behavior. I've also noticed that sometimes when a device is successfully mounted on all nodes, each node may return different output from the mount command regarding the device mounted; e.g. /dev/mapper/mpath0p1 vs. /dev/dm-17. I'm unsure as to why you get the 'device busy' errors. That's the weird thing. Mounting by label makes you vulnerable to the whims of blkid - whichever /dev name it finds first is what you'll see in the output of mount(8). But in the end, it should be the same device (that is, /dev/mapper/mpath0p1 is probably the same thing as /dev/dm-17). One consistent aspect I have noticed whenever I receive the device busy error is that the /dev/dm-X names don't match up on each node. I also see that ocfs2console refers to each device by the /dev/dm-X name instead of the /dev/mapper/XX name. The /dev/dm-X names are not intended to match up. They appear in the order the system discovers them. The multipath daemon is responsible for matching the /dev/dm-X name to the consistent /dev/mapper/mpathXpY name. So if you want to see a consistent name, use that one. I guess my question is simply: Are dm-multipath devices supported under OCFS2? Are multipathed devices not recommended with OCFS2? Any documentation available on this? They should work, and they are absolutely recommended. Joel -- If at first you don't succeed, cover all traces that you tried. -Unknown Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] No space left on device
On Wed, Feb 25, 2009 at 02:11:53PM +, Nuno Fernandes wrote: We are using the latest ocfs with 11 nodes: OCFS2 Node Manager 1.4.1 Tue Dec 16 19:18:05 PST 2008 (build 0f78045c75c0174e50e4cf0934bf9eae) OCFS2 DLM 1.4.1 Tue Dec 16 19:18:05 PST 2008 (build 4ce8fae327880c466761f40fb7619490) OCFS2 DLMFS 1.4.1 Tue Dec 16 19:18:05 PST 2008 (build 4ce8fae327880c466761f40fb7619490) OCFS2 User DLM kernel interface loaded But we are unable to create files bigger that 972 KB. Here is the output: [r...@t1 root]# df -h FilesystemSize Used Avail Use% Mounted on /dev/hda1 16G 8.7G 6.3G 58% / none 1.0G 0 1.0G 0% /dev/shm /dev/hda3 299G 252G 48G 85% /home/nfs [r...@t1 root]# mount /dev/hda1 on / type ext3 (rw) none on /proc type proc (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) configfs on /sys/kernel/config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /dev/hda3 on /home/nfs type ocfs2 (rw,_netdev,noatime,heartbeat=local) [r...@t1 root]# dd if=/dev/zero of=/home/nfs/.zbr bs=1M count=10 dd: writing `/home/nfs/.zbr': No space left on device 1+0 records in 0+0 records out [r...@t1 root]# ls -lah /home/nfs/.zbr -rw-r--r--1 root root 972K Feb 25 14:09 /home/nfs/.zbr But we are able to create other files smaller than that. That's rather odd. What is the blocksize and cluster size of this filesystem? Can you send me the output of 'debugfs.ocfs2 -R stat /.zbr /dev/hda3'? Joel -- Practice random acts of kindness and senseless acts of beauty. Oh, and don't forget where your towel is. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 16TB limitation
On Thu, Feb 26, 2009 at 12:21:54PM -0800, Adam Schrotenboer wrote: Is the 16TB limitation per-node, or for the aggregate filesystem? Also, is the 16TB limit only in 1.2, or also in 1.4? The limit is for the entire filesystem, but it doesn't come from ocfs2 itself. ocfs2 can address 16TB with 4K clusters up to 4PB with 1M clusters. The limitation comes from the Linux Journaled Block Device (JBD). This is what ocfs2 uses for its journaling in 1.2 and 1.4. Linux now has a new version of JBD, called JBD2. This version relaxes the limit, allowing ocfs2 to support larger volumes. ocfs2 in the mainline Linux kernel is already doing this. However, JBD2 support is not yet in 1.4. We hope to provide that support later in the year. Joel -- Life's Little Instruction Book #450 Don't be afraid to say, 'I need help.' Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] quota/acl support
On Fri, Feb 27, 2009 at 03:46:42PM -0600, Brian Kroth wrote: Attempting this again since I got a DSN earlier. Sorry about that, the mail servers were acting up. Brian Kroth bpkr...@gmail.com 2009-02-25 10:25: I'm doing some research on the possibility of using OCFS2 to serve users' home directories and other shared space. I noticed that quota and posix acl support was added in 2.6.29 but the tools are not there yet. When can we expect that? The tools for quota support are coming. We're probably looking at the second half of this year. We have a number of new features coming down the pipe. Also, are the quotas implemented on a directory or volume level? Volume level, I believe. They are pretty similar to ext3's quotas. Joel -- Life's Little Instruction Book #232 Keep your promises. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] OCFS2: ERROR (device sdh1): ocfs2_direct_IO_get_blocks
On Fri, Feb 27, 2009 at 06:40:38PM -0600, Daniel Keisling wrote: I'm am getting the following error when writing to an OCF2 filesystem: Feb 27 19:06:37 wilracdbdr01 kernel: OCFS2: ERROR (device sdh1): ocfs2_direct_IO_get_blocks: Inode 23693699 has a hole at block 6 Feb 27 19:06:37 wilracdbdr01 kernel: File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted. This basically says that your filesystem does not support sparse files, but it does have a hole in an inode - which shouldn't happen if sparse isn't supported. Can you send the output of debugfs.ocfs2 -R 'stat 23693699' /dev/sdh1 and debugfs.ocfs2 -R stats /dev/sdh1? I saw a patch that was released in September 2008. How do I get this? What patch? Do you have a link? Without knowing the patch I can't tell you whether that patch affects you. This a production system and we are currently unable to start the DB. If you have the appropriate support, you should call support and file that way. Support will also want the information I requested above. Joel -- Three o'clock is always too late or too early for anything you want to do. - Jean-Paul Sartre Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ASM over OCFS2 vs. Standard locally managed tablespaces
On Mon, Feb 09, 2009 at 04:16:23AM -0800, Luis Freitas wrote: About the performance, ASM is said to have similar performance to raw devices in a SAME layout, being tightly integrated to Oracle. OCFS2 has some overheads that are inherent to a file system, like cache management, locking, context switching, so it is likely to use more CPU power than ASM. But I dont remember any specific benchmark comparing those. ocfs2 has performance equivalent to raw devices when using O_DIRECT, which the database will do for its datafiles. We worked hard at that from the beginning. You won't see filesystem overhead for the O_DIRECT access. You only see the overhead of cache management, etc, for cached (non-O_DIRECT) files, which isn't what you're worried about for database performance. Also, keep in mind that when you use a filesystem you are using part of the memory for the filesystem cache. When using RAW or ASM you would need to allocate this memory to the block buffer in order to compare results. Again, Oracle uses O_DIRECT for datafiles. This keeps the data out of the filesystem cache. A single-node (non-RAC) database can use the filesystem cache, and that can cause benchmark discrepancies, but we're talking about RAC here. Joel -- To spot the expert, pick the one who predicts the job will take the longest and cost the most. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] future of ocfs2
On Thu, Feb 05, 2009 at 06:28:51PM -0600, Jeremy Schneider wrote: We're internally debating whether or not we should include OCFS2 in this design right now, and I'm curious if anyone has arguments one way or the other to share. Our standard design on Solaris does utilize a cluster filesystem and we would welcome a similar design, but there are some concerns about the readiness, stability and future of OCFS2. Short answer: It's been ready for years, it's been stable for years, and it has a bright future. OCFS2 is being considered for these four use cases: - database binaries (vs local files or NFS) - diag top (11g) or admin tree (10g) (vs local files or NFS) - archived logs - backups These are the most standard and basic of ocfs2 uses outside of data files. Almost all customers running ocfs2 are using it for most or all of these files. I have seen mention in blogs such as http://bigdaveroberts.wordpress.com/ of something called ASMFS in 11gR2 and I'm wondering - will this feature (if included) have any impact on Oracle's commitment to OCFS2 development? Could Oracle conceivably develop a whole new cluster filesystem and put their full weight behind it as they did for ASM storage, leaving OCFS2 as a lower priority for new features and improvements? Has Oracle demonstrated significant commitment to OCFS2 development and support in the past, and is this a mature enough technology for wide-scale deployment? Oracle continually works to improve the power and usability of of its products. Sometimes features overlap, but that doesn't mean they must exclude each other. For example, you are planning to use ASM storage for datafiles, but a filesystem of some type (ocfs2, NFS, whatever) for other files. This works just fine. Those of us who work on ocfs2 are not stopping. Let's revisit your questions at the top. First up is readiness. ocfs2 1.2, the older of the two currently supported production releases, was released in February 2006. That's three years ago. Customers have been running it for the use cases you describe ever since. Next is stability. Oracle does extensive testing on each release of ocfs2 to provide the best stability we can. The 1.2 release ironed out most of its bugs long ago. The new 1.4 release, released in August 2008, provides some significant performance improvements and is already being adopted by our customers. What about the future? We just released the 1.4 version, and we're not done yet. As Sunil pointed out in his email, we have a number of significant features landing in the mainline Linux kernel; this is where we do the development of ocfs2. These features will make their way into ocfs2 1.4 or a future release as they become stable. I hope that covers Oracle's commitment to the development of ocfs2. For support, let's look at http://oss.oracle.com/projects/ocfs2/ (you can go there for all sorts of information about ocfs2): -8 SUPPORT Oracle provides full support for the OCFS2 file system for Oracle's Unbreakable Linux Network subscribers. Oracle also extends support for the OCFS2 file system to Red Hat Enterprise Linux users for use with Oracle's database product. Novell provides full support for the OCFS2 file system to SUSE Linux Enterprise Server users. -8 I hope this answers your questions and helps you in making your decision. Joel -- Life's Little Instruction Book #313 Never underestimate the power of love. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Shared writable mmap
On Wed, Jan 07, 2009 at 01:32:14PM +0200, Eran Maman wrote: I would like to know more about the ocfs2 implementation for shared writable mmap. Does every read go to the disk? Or is it cached to RAM memory? Like all mmap operations, the data is cached in RAM as long as it is unmodified. Basically, all local node operations behave like an un-clustered filesystem. The only change is when another node needs something. If the local node has a cached page and a remote node wants to change that page, the local node will drop the page and re-read it later. Conversely, if the local node has a modified page and the remote node wants to read that page, the local node will write the changes to disk. I hope that answers your question. Joel -- Viro's Razor: Any race condition, no matter how unlikely, will occur just often enough to bite you. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Feature Question: Inotify over OCFS2
On Sun, Dec 28, 2008 at 07:45:11PM +1100, Brett Worth wrote: Raz Ben-Yehuda wrote: Is Inotify **fully supported** by OCFS2? I just did some grep'ing through the source tree and my guess is that inotify is not supported at all by OCFS2. I'd be happy to be wrong but there's no reference to it anywhere. There's no results for grepping 'inofity' (or 'fsnotify', the generic bit) in ext2 or ext3 either. To my knowledge, filesystems don't have to do anything special. Now, how does inotify play in the world of a cluster filesystem like ocfs2? That's a different story. My guess is that actions taking place on the local node will be seen as expected, but actions happening on another node will not. ocfs2 makes no provision to tell one node about something that happens on another node unless it affects the first node's open files. I bet this fails Raz's test of fully supported, if he expects to have inotify watches on node 1 see actions that occur on node 2. Conversely, I would bet that an un-clustered (local) ocfs2 filesystem would behave identically to other local filesystems like ext3. Joel -- There is a country in Europe where multiple-choice tests are illegal. - Sigfried Hulzer Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Unable to access cluster service
On Tue, Oct 28, 2008 at 02:15:07PM -0600, Paul Mathews wrote: On Tuesday 28 October 2008 1:44:53 pm Joel Becker wrote: On Tue, Oct 28, 2008 at 01:10:46PM -0600, Paul Mathews wrote: Currently, ocfs2 is compiled into the kernel, and the o2cb init script does not run on startup: [EMAIL PROTECTED]:~# /etc/init.d/o2cb start Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting Oracle cluster ocfs2: Failed o2cb_ctl: Unable to access cluster service Cannot initialize cluster Hmm, it doesn't say 'mounting configfs'. Is configfs mounted (at /sys/kernel/config)? Is sysfs mounted? it doesn't appear that configfs, but sysfs is: If configfs is *not* mounted, can you run 'sh -x /etc/init.d/o2cb start' and send us the output? That way we know why it isn't mounted. OTOH, if it is mounted, we want the strace(1) of o2cb_ctl. debian1:~# sh -x /etc/init.d/o2cb start + export LC_ALL=C + LC_ALL=C + CLUSTERCONF=/etc/ocfs2/cluster.conf + '[' -f /etc/sysconfig/o2cb ']' + '[' -f /etc/default/o2cb ']' + CONFIGURATION=/etc/default/o2cb + '[' -f /etc/default/o2cb ']' + . /etc/default/o2cb ++ O2CB_ENABLED=true ++ O2CB_BOOTCLUSTER=ocfs2 ++ O2CB_HEARTBEAT_THRESHOLD=7 ++ O2CB_IDLE_TIMEOUT_MS=1 ++ O2CB_KEEPALIVE_DELAY_MS=5000 ++ O2CB_RECONNECT_DELAY_MS=2000 + LOAD_ACTIONS=(mount_fs ocfs2_dlmfs /dlm) LOAD_ACTIONS is not part of ocfs2-tools 1.4. Can you verify you have ocfs2-tools 1.4 installed and that your /etc/init.d/o2cb comes from it? I wonder if it was marked as a 'configuration' file and as such not overwritten by a newer version. That being said, if you are using ocfs2-tools 1.2, LOAD_ACTIONS should contain a number of other actions. Did you hand-edit /etc/init.d/o2cb to remove some actions when you compiled the modules in? Joel -- Life's Little Instruction Book #43 Never give up on somebody. Miracles happen every day. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Unable to access cluster service
On Tue, Oct 28, 2008 at 02:25:42PM -0600, Paul Mathews wrote: Sorry. I found the problem - my error. I accidentally removed the configfs mount load action when I was commenting out the ocfs2 modules. Adding it in fixed the problem :) I'm glad you found it. :-) Joel -- Hell is oneself, hell is alone, the other figures in it, merely projections. - T. S. Eliot Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 problem
[Cc'd ocfs2-users@oss.oracle.com, the right forum] On Fri, Oct 17, 2008 at 09:12:35AM -0400, Burs, Lee wrote: I am having a problem with the ocfs2 stack loading on a Red Hat Linux 2.6.18-92.1.13.el5xen system. I have installed the following components of ocfs2 for this x86_64 system type: ocfs2-2.6.18-92.1.13.el5xen-1.2.9-1.el5.x86_64.rpm ocfs2console-1.4.1-1.el5.x86_64.rpm ocfs2-tools-1.4.1-1.el5.x86_64.rpm The package versions look right. When I check the status of ocfs2 as follows /etc/init.d/o2cb status Have you confiugred o2cb? Started it? Configured cluster.conf? Joel -- But then she looks me in the eye And says, 'We're going to last forever,' And man you know I can't begin to doubt it. Cause it just feels so good and so free and so right, I know we ain't never going to change our minds about it, Hey! Here comes my girl. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] why does mkfs.ocfs2 take so long?
On Mon, Jul 28, 2008 at 10:23:14AM -0700, Sunil Mushran wrote: Two inits take time. 1. Cluster group init. 2. Journal init. Also, joining the cluster takes time. You are overwriting an existing volume. mkfs.ocfs2 will start up the cluster software to ensure another node is not using the volume. This takes time. Joel -- Life's Little Instruction Book #313 Never underestimate the power of love. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Different size with du and ls
On Fri, Jul 11, 2008 at 12:06:48AM +0200, Markus Meyer wrote: Jep, 4 TB it is ;) 64k!? I had to act fast so I didn't have the time to check all mkfs options. As I read about the -T mail option I thought it would choose sensible values for creating the FS. All the files are around 12-15 kB in size and the blocksize was 4 kB as in the XFS filesystem we use. If all files are 12-15k, perhaps you want a clustersize of 8k or 16k. If your files are always written in one go, though, 4K will do you just fine. Blocksize I understand but what means cluster size in this context (I read the man page but couldn't wrap my head around it). ocfs2's unit of allocation is a 'cluster'. The smallest file is one cluster in size. An ocfs2 filesystem can have at most UINT32_MAX clusters (2^32 - 1, or 4294967295). Thus, a filesystem with 4K clusters can be at most 16TB (which you fit within). Larger cluster sizes allow larger filesystems (up to 4PB with 1M clusters). The other benefit of larger cluster sizes is reduced fragmentation. A 16K file requires 4 4K clusters but only 1 16K cluster. If the filesystem is fragmented, those 4 clusters can be in different places on the disk, increasing seek times and reducing throughput. Thus, there are two reasons to choose a larger cluster size. Your disk is so bit that it is needed, or you have larger files that benefit from improved contiguousness. The downside, as you know, is wasted space from small files. mkfs.ocfs2 has to try and guess what the filesystem is going to be used for. Larger filesystems are often used for large files like database data, so larger cluster sizes are chosen. It's not a perfect science, which is why we provide the -T option as well as explicit sizes. Block sizes are units of metadata. A file inode. An allocation tree element. These are sub-cluster entities. You can have blocks of 512, 1K, 2K, or 4K bytes. The larger the blocksize, the more metadata can be held in it - this is especially useful for files with lots of fragmented data extents, among other things. We really recommend 4K blocks for all users - there isn't much to be gained from going smaller, and there is much to be lost. Still it seems that reformatting is my only choice since I don't have enough space with the current setup. *argh* How do you not have enough space? You have a *lot* of files? Joel -- Here's something to think about: How come you never see a headline like ``Psychic Wins Lottery''? - Jay Leno Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Query regarding OCFS2 and RHEL4 LVM
On Tue, Jul 08, 2008 at 05:36:48PM +0100, Pedro Figueira wrote: Why do you consider the LVM2 not cluster safe? Is it just because you should not (can't) modify any LVM configuration with the volume groups activated in more than one server? Not only can you not modify it, there are also configurations where the LVM2 code will do modifications behind your back - and the other nodes don't know about it. For example, dm-mirror keeps track of changes that have only hit one copy so far. Another node might read the un-updated copy before it has been updated by the writing node. That's not safe. We consider data integrity of paramount importance. Trying to configure LVM2 (non-clustered) to make sure nothing will change is not trivial. We won't recommend it, knowing that it just takes one admin who isn't paying attention to wipe out your entire volume. Joel -- Only a life lived for others is a life worth while. -Albert Einstein Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Query regarding OCFS2 and RHEL4 LVM
On Mon, Jul 07, 2008 at 05:45:30PM +0530, Vinayak Malap wrote: Hi, How can I configure LVM in linux(RHEL4 u5 x86_64) to use with OCFS2 file system ? Is there any special configuration steps we need to follow ? Is there any document to guide on this ? This belongs on the ocfs2-users@oss.oracle.com mailing list. The short answer is that LVM is not cluster-safe, so you would have to create an LVM configuration that makes no changes online. What do I mean? A RAID-5 setup modifies data outside the block being written, and so is not cluster safe. A setup that swaps in a hot-spare device is not safe. About the only thing that could work is a concatenation of disks, and you cannot change it while ocfs2 is mounted. You can try the clutsered LVM (CLVM), but that uses a different cluster technology than ocfs2 does, and as such the two cluster stacks may conflict when a node goes down. There are plans to make ocfs2 better work with CLVM in the future. Joel -- Life's Little Instruction Book #407 Every once in a while, take the scenic route. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Query regarding OCFS2 and RHEL4 LVM
On Tue, Jul 08, 2008 at 07:54:38AM +0530, Vinayak Malap wrote: Pnbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; We are usingnbsp; systems with FONT face=Courier Newlinux version(RHEL4 u5 x86_64). We have 18 * 20GB LUNs from storage accesssible to both nodes. Our requirement is 250GB(5*50GB),150GB etc. for data .Can we create logical volumes (linear/concatenate type )innbsp;RHEL4 LVM2 to use with ocfs2 for Oracle RAC and Data files. /FONT/P It is possible if done carefully, but it is not cluster safe and not recommeneded for a production environment. Joel -- If the human brain were so simple we could understand it, we would be so simple that we could not. - W. A. Clouston Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Is an SMP version available for 1.2.8-2?
On Tue, May 13, 2008 at 02:06:00PM -0700, Diane Petersen wrote: I'm installing OCFS2 on the following RH release and architecture: 2.6.18-53.1.14.el5 #1 SMP Wed Mar 5 11:37:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux But I couldn't find an smp version of the latest 1.2.8-2 OCFS2 RH rpm. I downloaded this one, did I miss the smp version somewhere? ocfs2-2.6.18-53.1.14.el5-1.2.8-2.el5.x86_64.rpm That is the SMP version. The default 2.6.18-53.1.14.el5 kernel is an SMP kernel, and the ocfs2 package matches it. Joel -- I have never let my schooling interfere with my education. - Mark Twain Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] restore deleted partition
On Fri, Apr 25, 2008 at 12:40:46PM -0700, Joel Becker wrote: On Fri, Apr 25, 2008 at 06:30:13PM +0200, Christian Lox wrote: Am 25.04.2008 um 18:28 schrieb Sunil Mushran: That's the first backup. That looks bad... not a good sign. What about the second backup? It is at offset 4G. We tried with options -r 1 up to -r 6. Same result. :( Wouldn't the new mkfs have written new backup superblocks? They're not going to help. The only thing I can think of is Ok, I assumed the overwrite was with ocfs2, which may not be correct. What kind of filesystem was written? Joel -- Reader, suppose you were and idiot. And suppose you were a member of Congress. But I repeat myself. - Mark Twain Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Did anything substantial change between 1.2.4 and 1.3.9?
On Mon, Apr 21, 2008 at 05:02:33PM +0800, Tao Ma wrote: Then there is only one thing maybe. Have you modify /etc/sysconfig/o2cb(This is the place for RHEL, not sure the place in ubuntu)? I have checked the rpm package for RHEL, it will update /etc/sysconfig/o2cb and this file has some timeouts defined in it. It is probably /etc/default/o2cb for Ubuntu. Joel -- I am working for the time when unqualified blacks, browns, and women join the unqualified men in running our overnment. - Sissy Farenthold Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] Symbolic Link issues in ocfs
On Tue, Apr 15, 2008 at 03:59:19PM +1000, Brendan Beveridge wrote: if i change the owner of the link using chmod: node1:# chown -h user1 /opt/link node1:# ls -lah link # ls -lah link lrwxrwxrwx 1 user1 root 9 2008-04-15 15:35 link - /opt/file and on node2: node2:# ls -lah link # ls -lah link lrwxrwxrwx 1 user1 root 9 2008-04-15 15:35 link - /opt/file So now i unmount the ocfs parition on node1, then remount it and check the file again: node1# umount /opt node1# mount /opt node1# ls -lah /opt/link lrwxrwxrwx 1 root root 9 2008-04-15 15:35 /opt/link - /opt/file I just checked, and this does indeed happen. My best guess? We don't provide -setattr() for symlinks. Most filesystems don't. They let the regular inode_setattr() handle it, which calls mark_inode_dirty(). When that dirty inode is written, it hits disk. I'm guessing that our dirty handling, which is slightly different, doesn't work here. Mark might have a quick idea, or I'll look more when I get back from vacation. Note that the owner has changed back to root? Note that standard files do not have this issue Out of curiosity, why do you want to do this? Symlinks are always 777. Joel -- Senator let's be sincere, As much as you can. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] SuSe Hangs when /etc/init.d/o2cb online
On Tue, Apr 01, 2008 at 05:05:59PM -0700, Sunil Mushran wrote: Do a mount and you should see configfs mounted: configfs on /sys/kernel/config type configfs (rw) If so, the hang is puzzling. Email opensuse help list if anyone else has encountered the same. Joel, Any ideas? Not off the top of my head. Does the ocfs2 directory already exist? Joel -- There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle. - Albert Einstein Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] user-space heartbeat ptaches from suse
On Sat, Jan 05, 2008 at 07:52:46PM +0100, Christian Lox wrote: Does anyone know why the user-space heartbeat (linux-ha) patches from SuSE / Novell are not commited to the mainline ocfs2 source tree? Or are they? Is there an easy way (which does not break the distros package management) to apply these? We looked at those patches and determined they weren't a flexible way to go forward. We've worked on it, and we think we have a scheme that will work for all userspace stacks. You can see the design documents at http://oss.oracle.com/osswiki/OCFS2/DesignDocs/UserspaceClustering. The attempt to use the SuSE patches with mainline is documented at http://oss.oracle.com/osswiki/OCFS2/CManUnderO2CB. The new scheme won't make mainline for a little while - we have some details to work out and some testing to do. You're welcome to try it from our GIT trees before then, of course. Regarding distro packages, the new scheme won't be in a distro package any time soon. Unfortunately, you cannot apply patches on top of a binary distro package (which is what I think you were asking). Joel -- Life's Little Instruction Book #314 Never underestimate the power of forgiveness. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] 6 node cluster with unexplained reboots
On Thu, Aug 16, 2007 at 02:29:43AM -0700, Ulf Zimmermann wrote: Ok, we had now 4 reboots, plus 2 more by my own action, which were by OCFS2 fencing. As said in previous emails we were seeing some SCSI errors and although device-mapper-multipath seems to take care of it, sometimes the 10 second configured in multipath.conf and the default timings of o2cb are colliding. I can certainly see that happening. Now I am still concerned about the timing of device-mapper-multipath and o2cb. O2cb is currently set to the default of: Specify heartbeat dead threshold (=7) [7]: Specify network idle timeout in ms (=5000) [1]: Specify network keepalive delay in ms (=1000) [5000]: Specify network reconnect delay in ms (=2000) [2000]: I would certainly bump up the timeouts as in the FAQ (see Marcos' reply) So the timeout I seem to hit is the 10,000 of network idle timeout? Even this timeout occurs on the disk? What values would you recommend I should set this to? I think you're hitting heartbeat dead threshold. Bump that to something larger (eg, 31) like in the FAQ. This is a multiplier of the region check interval. Better way to put it: Every N seconds the system checks for a heartbeat on disk. If it checks threshold times without seeing the other node (that is, N * threshold seconds), it will consider that node dead. What it looks like you are seeing is that the one node gets to (N * threshold seconds) before multipath has a chance to fix the I/O. That's why we suggest bumping the timeout. Mark/Marcos, correct me if I'm wrong here. Another question in case someone can answer this. If I get a syslog entries like: Aug 16 00:44:33 dbprd01 kernel: SCSI error : 1 0 0 1 return code = 0x2 Aug 16 00:44:33 dbprd01 kernel: end_request: I/O error, dev sdj, sector 346452448 Aug 16 00:44:33 dbprd01 kernel: device-mapper: dm-multipath: Failing path 8:144. ...snip Aug 16 00:44:33 dbprd01 multipathd: 8:144: mark as failed Aug 16 00:44:33 dbprd01 multipathd: u01: remaining active paths: 3 Does this actually errors out all the way or does the request still go to one of the remaining paths? If this request doesn't error out, because it was able to still fulfill it via the 2 remaining paths, then it is really just the timing between device-mapper-multipath recovering this request through the remain paths and our o2cb settings. If not, we might still have another problem. We have seen many such errors but only had like 8 reboots, all I think attributed to fencing now. That error log looks like multipath is correcting the I/O. The error isn't coming all the way up, it's being handled by multipath. If the error came all the way back up, you should see errors printed by the heartbeat process. If you don't see them, it never got an error, which is how multipath should behave. Joel -- Life's Little Instruction Book #314 Never underestimate the power of forgiveness. Joel Becker Principal Software Developer Oracle E-mail: [EMAIL PROTECTED] Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
