Re: [Ocfs2-users] one node kernel panic

[Sunil Mushran]

uek is a different kernel entirely. It is hard to say whether you
will or will not hit it with uek mainly because the underlying code
is different.

On 10/06/2011 10:33 PM, Hideyasu Kojima wrote:
> Thank you for responding.
>
> I think UEK5 is based on RHEL5 kernel.
> Does the problem same as UEK5 arise?
>
> (2011/10/05 1:45), Sunil Mushran wrote:
>> int sigprocmask(int how, sigset_t *set, sigset_t *oldset)
>> {
>> int error;
>>
>> spin_lock_irq(¤t->sighand->siglock); < CRASH
>> if (oldset)
>> *oldset = current->blocked;
>> ...
>> }
>>
>> current->sighand is NULL. So definitely a race. Generic kernel issue.
>> Ping your kernel vendor.
>>
>> On 10/03/2011 07:49 PM, Hideyasu Kojima wrote:
>>> Hi,
>>>
>>> I run ocfs2/drbd active-active 2node cluster.
>>>
>>> ocfs2 version is 1.4.7-1
>>> ocfs2-tool version is 1.4.4
>>> Linux version is RHEL 5.4 (2.6.18-164.el5 x86_64)
>>>
>>> 1 node crash with kernel panic once.
>>>
>>> What is the cause?
>>>
>>> The bottom is the analysis of vmcore.
>>>
>>> 
>>>
>>> Unable to handle kernel NULL pointer dereference at 0808 RIP:
>>> [] _spin_lock_irq+0x1/0xb
>>> PGD 187e15067 PUD 187e16067 PMD 0
>>> Oops: 0002 [1] SMP
>>> last sysfs file:
>>> /devices/pci:00/:00:09.0/:06:00.0/:07:00.0/irq
>>> CPU 1
>>> Modules linked in: mptctl mptbase softdog autofs4 ipmi_devintf ipmi_si
>>> ipmi_msghandler ocfs2(U) ocfs2_dlmfs(U) ocfs2_dlm(U)
>>> ocfs2_nodemanager(U) configfs drbd(U) bonding ipv6 xfrm_nalgo crypto_api
>>> bnx2i(U) libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi cnic(U)
>>> dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core
>>> button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev
>>> sr_mod cdrom sg pcspkr serio_raw hpilo bnx2(U) dm_raid45 dm_message
>>> dm_region_hash dm_log dm_mod dm_mem_cache hpahcisr(PU) ata_piix libata
>>> shpchp cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
>>> Pid: 21924, comm: res Tainted: P 2.6.18-164.el5 #1
>>> RIP: 0010:[] []
>>> _spin_lock_irq+0x1/0xb
>>> RSP: 0018:81008b1cfae0 EFLAGS: 00010002
>>> RAX: 810187af4040 RBX:  RCX: 8101342b7b80
>>> RDX: 81008b1cfb98 RSI: 81008b1cfba8 RDI: 0808
>>> RBP: 81008b1cfb98 R08:  R09: 
>>> R10: 810075463090 R11: 88595b95 R12: 81008b1cfba8
>>> R13: 81007f070520 R14: 0001 R15: 81008b1cfce8
>>> FS: () GS:810105d51840()
>>> knlGS:
>>> CS: 0010 DS:  ES:  CR0: 8005003b
>>> CR2: 0808 CR3: 000187e14000 CR4: 06e0
>>> Process res (pid: 21924, threadinfo 81008b1ce000, task
>>> 810187af4040)
>>> Stack: 8001db30 81007f070520 885961f3
>>> 810105d39400
>>> 88596323 06ff813231393234 810075463018 810075463018
>>> 0297 81007f070520 810075463028 0246
>>> Call Trace:
>>> [] sigprocmask+0x28/0xdb
>>> [] :ocfs2:ocfs2_delete_inode+0x0/0x1691
>>> [] :ocfs2:ocfs2_delete_inode+0x130/0x1691
>>> [] :ocfs2:ocfs2_drop_lock+0x67a/0x77b
>>> [] :ocfs2:ocfs2_remove_lockres_tracking+0x10/0x45
>>> [] :ocfs2:ocfs2_delete_inode+0x0/0x1691
>>> [] generic_delete_inode+0xc6/0x143
>>> [] :ocfs2:ocfs2_drop_inode+0xf0/0x161
>>> [] dput+0xf6/0x114
>>> [] prune_one_dentry+0x66/0x76
>>> [] prune_dcache+0x10f/0x149
>>> [] shrink_dcache_parent+0x1c/0xe1
>>> [] proc_flush_task+0x17c/0x1f6
>>> [] sched_exit+0x27/0xb5
>>> [] release_task+0x387/0x3cb
>>> [] do_exit+0x865/0x911
>>> [] cpuset_exit+0x0/0x88
>>> [] get_signal_to_deliver+0x42c/0x45a
>>> [] do_notify_resume+0x9c/0x7af
>>> [] deactivate_task+0x28/0x5f
>>> [] __up_read+0x19/0x7f
>>> [] do_page_fault+0x4fe/0x830
>>> [] audit_syscall_exit+0x336/0x362
>>> [] int_signal+0x12/0x17
>>>
>>>
>>> Code: f0 ff 0f 0f 88 f3 00 00 00 c3 53 48 89 fb e8 33 f5 02 00 f0
>>> RIP [] _spin_lock_irq+0x1/0xb
>>> RSP
>>> crash> bt
>>> PID: 21924 TASK: 810187af4040 CPU: 1 COMMAND: "res"
>>> #0 [81008b1cf840] crash_kexec

Re: [Ocfs2-users] Kernel Panic / Fencing

[Sunil Mushran]

I am unclear. What happens when a server is rebooted (or crashes).
Crash the network? Can you expand on this?

On 10/06/2011 05:52 PM, Tony Rios wrote:
> Hey all,
>
> I'm running a current version of Ubuntu and we are using OCFS2 across
> a cluster of 9 web servers.
> Everything works perfectly, so long as none of the servers need to be
> rebooted (or crash).
>
> I've done several web searches and one of the items that I've found to
> be suggested was to double the Heartbeat threshold.
> I increased ours from 31 to 61 and it doesn't appear to have helped at all.
>
> I can't imagine that if a server becomes unreachable that by design it
> is intended to crash the entire network.
>
> I'm hoping that someone will have some feedback here because I'm at a loss.
>
> Thanks so much,
> Tony
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Fwd: OCFS drives not syncing

[Sunil Mushran]

On 10/05/2011 08:46 AM, Bradlee Landis wrote:
> Sorry Sunil, my email replied to you instead of the list.
>
> On Wed, Oct 5, 2011 at 10:09 AM, Sunil Mushran  
> wrote:
>> ocfs2 is a shared disk cluster file system. It requires a shared disk.
>>
>> However, if you are only going to use 2 nodes, you could use drbd,
>> a replicating block device. To ocfs2, it appears as a shared disk.
>> Google drbd and ocfs2 for more.
>>
> So I've been confused about this the whole time I guess. So how is the
> OCFS drive shared? Is it done through OCFS, or does it require NFS?
>
> How do I access the filesystem from the other node?

The drives need to be physically shared. As in, all nodes need
to be able to concurrently read and write directly to the disk.

Two popular solutions are fiber channel and iscsi.

A fiber channel solution could be EMC disk array + FC switch +
hbas on all nodes hooked up to the switch.

An iscsi solution could be a iscsi target running on one server
with the disks. The nodes would use an iscsi initiator to access
the target. The devices will show up as regular devices (/dev/sdX)
on all nodes.

The cheapest solution would be to use drbd.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS drives not syncing

[Sunil Mushran]

ocfs2 is a shared disk cluster file system. It requires a shared disk.

However, if you are only going to use 2 nodes, you could use drbd,
a replicating block device. To ocfs2, it appears as a shared disk.
Google drbd and ocfs2 for more.

On 10/05/2011 07:15 AM, Bradlee Landis wrote:
> I have asked a question at
> http://serverfault.com/questions/318526/ocfs-configuration , but I
> thought this might be a better place to ask.
>
> I installed Oracle Linux 3.0 on my machine, and I created two ocfs2
> partitions, /cluster1 and /cluster2. I just created it with default
> options, so maybe that's where I went wrong. I configured
> /etc/ocfs2/cluster.conf and generated /etc/sysconfig/o2cb. It looks to
> be fine, but it's not actually syncing the filesystems.
>
> What am I doing wrong? Do I need to specify certain flags in /etc/fstab?
>
> I put a filesystem name "vmcluster" on sda3 to see if it might help.
> Also, on one server, it's using sda3 and sdb3, and the other is sda2
> and sdb2. They also may be different sizes, so if that would be
> causing my problem, let me know.
>
> Here is my configuration below.
>
> == /etc/mtab ==
> /dev/sda3 /cluster1 ocfs2 rw,_netdev,heartbeat=local 0 0
> /dev/sdb3 /cluster2 ocfs2 rw,heartbeat=none 0 0
>
> == /etc/ocfs2/cluster.conf ==
>
> cluster:
>  node_count = 2
>  name = vmcluster
>
> node:
>  ip_port = 
>  ip_address = 198.18.0.25
>  number = 0
>  name = vm1
>  cluster = vmcluster
>
> node:
>  ip_port = 
>  ip_address = 198.18.0.26
>  number = 1
>  name = vm2
>  cluster = vmcluster
>
> == /etc/sysconfig/o2cb (Generated) ==
>
> O2CB_ENABLED=true
> O2CB_STACK=o2cb
> O2CB_BOOTCLUSTER=vmcluster
> O2CB_HEARTBEAT_THRESHOLD=
> O2CB_IDLE_TIMEOUT_MS=
> O2CB_KEEPALIVE_DELAY_MS=
> O2CB_RECONNECT_DELAY_MS=
>
> 
>
>


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] one node kernel panic

[Sunil Mushran]

int sigprocmask(int how, sigset_t *set, sigset_t *oldset)
{
 int error;

 spin_lock_irq(¤t->sighand->siglock); < CRASH
 if (oldset)
 *oldset = current->blocked;
...
}

current->sighand is NULL. So definitely a race. Generic kernel issue.
Ping your kernel vendor.

On 10/03/2011 07:49 PM, Hideyasu Kojima wrote:
> Hi,
>
> I run ocfs2/drbd active-active 2node cluster.
>
> ocfs2 version is 1.4.7-1
> ocfs2-tool version is 1.4.4
> Linux version is RHEL 5.4 (2.6.18-164.el5 x86_64)
>
> 1 node crash with kernel panic once.
>
> What is the cause?
>
> The bottom is the analysis of vmcore.
>
> 
>
> Unable to handle kernel NULL pointer dereference at 0808 RIP:
>   [] _spin_lock_irq+0x1/0xb
> PGD 187e15067 PUD 187e16067 PMD 0
> Oops: 0002 [1] SMP
> last sysfs file:
> /devices/pci:00/:00:09.0/:06:00.0/:07:00.0/irq
> CPU 1
> Modules linked in: mptctl mptbase softdog autofs4 ipmi_devintf ipmi_si
> ipmi_msghandler ocfs2(U) ocfs2_dlmfs(U) ocfs2_dlm(U)
> ocfs2_nodemanager(U) configfs drbd(U) bonding ipv6 xfrm_nalgo crypto_api
> bnx2i(U) libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi cnic(U)
> dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core
> button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev
> sr_mod cdrom sg pcspkr serio_raw hpilo bnx2(U) dm_raid45 dm_message
> dm_region_hash dm_log dm_mod dm_mem_cache hpahcisr(PU) ata_piix libata
> shpchp cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
> Pid: 21924, comm: res Tainted: P  2.6.18-164.el5 #1
> RIP: 0010:[]  [] _spin_lock_irq+0x1/0xb
> RSP: 0018:81008b1cfae0  EFLAGS: 00010002
> RAX: 810187af4040 RBX:  RCX: 8101342b7b80
> RDX: 81008b1cfb98 RSI: 81008b1cfba8 RDI: 0808
> RBP: 81008b1cfb98 R08:  R09: 
> R10: 810075463090 R11: 88595b95 R12: 81008b1cfba8
> R13: 81007f070520 R14: 0001 R15: 81008b1cfce8
> FS:  () GS:810105d51840() knlGS:
> CS:  0010 DS:  ES:  CR0: 8005003b
> CR2: 0808 CR3: 000187e14000 CR4: 06e0
> Process res (pid: 21924, threadinfo 81008b1ce000, task 810187af4040)
> Stack:  8001db30 81007f070520 885961f3 810105d39400
>   88596323 06ff813231393234 810075463018 810075463018
>   0297 81007f070520 810075463028 0246
> Call Trace:
>   [] sigprocmask+0x28/0xdb
>   [] :ocfs2:ocfs2_delete_inode+0x0/0x1691
>   [] :ocfs2:ocfs2_delete_inode+0x130/0x1691
>   [] :ocfs2:ocfs2_drop_lock+0x67a/0x77b
>   [] :ocfs2:ocfs2_remove_lockres_tracking+0x10/0x45
>   [] :ocfs2:ocfs2_delete_inode+0x0/0x1691
>   [] generic_delete_inode+0xc6/0x143
>   [] :ocfs2:ocfs2_drop_inode+0xf0/0x161
>   [] dput+0xf6/0x114
>   [] prune_one_dentry+0x66/0x76
>   [] prune_dcache+0x10f/0x149
>   [] shrink_dcache_parent+0x1c/0xe1
>   [] proc_flush_task+0x17c/0x1f6
>   [] sched_exit+0x27/0xb5
>   [] release_task+0x387/0x3cb
>   [] do_exit+0x865/0x911
>   [] cpuset_exit+0x0/0x88
>   [] get_signal_to_deliver+0x42c/0x45a
>   [] do_notify_resume+0x9c/0x7af
>   [] deactivate_task+0x28/0x5f
>   [] __up_read+0x19/0x7f
>   [] do_page_fault+0x4fe/0x830
>   [] audit_syscall_exit+0x336/0x362
>   [] int_signal+0x12/0x17
>
>
> Code: f0 ff 0f 0f 88 f3 00 00 00 c3 53 48 89 fb e8 33 f5 02 00 f0
> RIP  [] _spin_lock_irq+0x1/0xb
>   RSP
> crash>  bt
> PID: 21924  TASK: 810187af4040  CPU: 1   COMMAND: "res"
>   #0 [81008b1cf840] crash_kexec at 800ac5b9
>   #1 [81008b1cf900] __die at 80065127
>   #2 [81008b1cf940] do_page_fault at 80066da7
>   #3 [81008b1cfa30] error_exit at 8005dde9
>  [exception RIP: _spin_lock_irq+1]
>  RIP: 80064ae6  RSP: 81008b1cfae0  RFLAGS: 00010002
>  RAX: 810187af4040  RBX:   RCX: 8101342b7b80
>  RDX: 81008b1cfb98  RSI: 81008b1cfba8  RDI: 0808
>  RBP: 81008b1cfb98   R8:    R9: 
>  R10: 810075463090  R11: 88595b95  R12: 81008b1cfba8
>  R13: 81007f070520  R14: 0001  R15: 81008b1cfce8
>  ORIG_RAX:   CS: 0010  SS: 0018
>   #4 [81008b1cfae0] sigprocmask at 8001db30
>   #5 [81008b1cfb00] ocfs2_delete_inode at 88596323
>   #6 [81008b1cfbf0] generic_delete_inode at 8002f49e
>   #7 [81008b1cfc10] ocfs2_drop_inode at 88595c85
>   #8 [81008b1cfc30] dput at 8000d46e
>   #9 [81008b1cfc50] prune_one_dentry at 800e9c44
> #10 [81008b1cfc70] prune_dcache at 8002e958
> #11 [81008b1cfca0] shrink_dcache_parent at 8004d66e
> #12 [81008b1cfcd0] proc_flush_task at 80104f8b
> #13 [81008b1cfd30] release_task at 80018024

Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource W0000000000000001b027d69b591f15 not on the Tracking list

[Sunil Mushran]

On 09/30/2011 06:49 AM, Herman L wrote:
> On Thursday, September 29, 2011 2:04 PM Sunil Mushran wrote:
>> On 09/29/2011 08:56 AM, Herman L wrote:
>>>> On Wednesday, September 21, 2011 4:00 PM, Sunil Mushran wrote:
>>>> On 09/21/2011 12:37 PM, Herman L wrote:
>>>>>>> On 09/19/2011 08:35 AM, Herman L wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Got a couple of these messages recently, but I don't know what they 
>>>>>>> mean.  Can anyone let me know if I need to panic?  I'm using OCFS2 
>>>>>>> compiled from the kernel source of RHEL 6.0's 2.6.32-71.18.2.el6.x86_64.
>>>>>>>
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.40] 
>>>>>>> (10387,12):dlm_lockres_release:507 ERROR: Resource 
>>>>>>> W0001b027d69b591f15 not on the Tracking list
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398194] lockres: 
>>>>>>> W0001b027d69b591f1, owner=1, state=0
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398195]  last used: 
>>>>>>> 8197071325, refcnt: 0, on purge list: no
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398197]  on dirty list: no, 
>>>>>>> on reco list: no, migrating pending: no
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398198]  inflight locks: 0, 
>>>>>>> asts reserved: 0
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398199]  refmap nodes: [ ], 
>>>>>>> inflight=0
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]  granted queue:
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]  converting queue:
>>>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398201]  blocked queue:
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Herman
>>>>>> From: Sunil Mushran
>>>>>> To: Herman L
>>>>>> Sent: Monday, September 19, 2011 12:57 PM
>>>>>> Subject: Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource 
>>>>>> W0001b027d69b591f15 not on the Tracking list
>>>>>>
>>>>>>I've no idea of the state of the source that you are using. The 
>>>>>> message
>>>>>>is a warning indicating a race. While it probably did not affect 
>>>>>> the functioning,
>>>>>>there is no guarantee that that would be the case the next time 
>>>>>> around.
>>>>>>
>>>>>>The closest relevant patch is over 2 years old.
>>>>>> http://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=b0d4f817ba5de8adb875ace594554a96d7737710
>>>>> Thanks Sunil for responding.  I know you can't easily support my setup, 
>>>>> but anyways I checked the sources.
>>>>>
>>>>> Looks like the patch you mention is in the sources I compiled from ( 
>>>>> RHEL6.0 kernel-2.6.32-71.24.1.el6.src.rpm ), so I guess the source of the 
>>>>> problem is elsewhere.
>>>>>
>>>>> The fs/ocfs2 directory from the RHEL6 sources I compiled from is almost 
>>>>> exactly the same as the mainline 2.6.32 kernel, except
>>>>> 1) It looks like they implemented the changes in aops.c from the "cleanup 
>>>>> blockdev_direct_IO locking" patch that's in 2.6.33.
>>>>> 2) In journal.c, they rename ocfs2_commit_trigger to 
>>>>> ocfs2_frozen_trigger, which seems to be from 2.6.35.
>>>>> 3) In cluster/masklog.c they add a "const" to the mlog_attr_ops 
>>>>> declaration
>>>>> 4) And in quota.h, they are missing "#define QFMT_OCFS2 3"
>>>>>
>>>>> Not sure if that helps any, but thanks in any case!
>>>> All those changes are ok. And unrelated. This is a new one.
>>> Sorry, I think I accidentally wrote a message with only the quoted block... 
>>> oops.  Sorry.
>>>
>>>
>>> Sunil, are you able to and interested in looking at this issue?  If so, is 
>>> there any information that I can provide that might help?  Fortunately, 
>>> after those few initial days of daily errors, it seems to have stopped for 
>>> now.  But of course, I'm still worried about this.
>>>
>> http://oss.oracle.com/~smushran/0001-ocfs2-dlm-Use-dlm-track_lock-when-adding-resource-to.patch
>>
>> This should fix it. But do note that the patch is untested.
> Thanks for the quick reply and patch!  I'll try to test it out when I get a 
> chance.  Also, is there any way to force this error so that I can know if 
> that patch is working?  Also, now that you have a fix for this, can you make 
> any kind of guess as to how likely or what circumstances that the unpatched 
> OCFS2 will cause  dangerous problems?

Well, the first goal is always to see nothing else is breaking. That's the most
important bit. As far as fixing the issue goes, only time will tell. There is no
way I can think of that will definitely prove that the issue is resolved. Also, 
even
if it does reproduce, it does not mean that this patch is bad. It could be there
is another race that we have to plug.

Depends on the definition of dangerous. If it means cluster-wide corruption, or
cluster-wide outage, then no. But if it means a node crashing, then yes. Though
the chance of that is fairly low.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource W0000000000000001b027d69b591f15 not on the Tracking list

[Sunil Mushran]

On 09/29/2011 08:56 AM, Herman L wrote:
>> On Wednesday, September 21, 2011 4:00 PM, Sunil Mushran wrote:
>> On 09/21/2011 12:37 PM, Herman L wrote:
>>>>> On 09/19/2011 08:35 AM, Herman L wrote:
>>>>> Hi all,
>>>>>
>>>>> Got a couple of these messages recently, but I don't know what they mean. 
>>>>>  Can anyone let me know if I need to panic?  I'm using OCFS2 compiled 
>>>>> from the kernel source of RHEL 6.0's 2.6.32-71.18.2.el6.x86_64.
>>>>>
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.40] 
>>>>> (10387,12):dlm_lockres_release:507 ERROR: Resource 
>>>>> W0001b027d69b591f15 not on the Tracking list
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398194] lockres: 
>>>>> W0001b027d69b591f1, owner=1, state=0
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398195]  last used: 8197071325, 
>>>>> refcnt: 0, on purge list: no
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398197]  on dirty list: no, on 
>>>>> reco list: no, migrating pending: no
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398198]  inflight locks: 0, 
>>>>> asts reserved: 0
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398199]  refmap nodes: [ ], 
>>>>> inflight=0
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]  granted queue:
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]  converting queue:
>>>>> Sep 19 08:07:15 server-1 kernel: [3892420.398201]  blocked queue:
>>>>>
>>>>> Thanks!
>>>>> Herman
>>>> From: Sunil Mushran
>>>> To: Herman L
>>>> Sent: Monday, September 19, 2011 12:57 PM
>>>> Subject: Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource 
>>>> W0001b027d69b591f15 not on the Tracking list
>>>>
>>>>  I've no idea of the state of the source that you are using. The 
>>>> message
>>>>  is a warning indicating a race. While it probably did not affect the 
>>>> functioning,
>>>>  there is no guarantee that that would be the case the next time 
>>>> around.
>>>>
>>>>  The closest relevant patch is over 2 years old.
>>>> http://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=b0d4f817ba5de8adb875ace594554a96d7737710
>>> Thanks Sunil for responding.  I know you can't easily support my setup, but 
>>> anyways I checked the sources.
>>>
>>> Looks like the patch you mention is in the sources I compiled from ( 
>>> RHEL6.0 kernel-2.6.32-71.24.1.el6.src.rpm ), so I guess the source of the 
>>> problem is elsewhere.
>>>
>>> The fs/ocfs2 directory from the RHEL6 sources I compiled from is almost 
>>> exactly the same as the mainline 2.6.32 kernel, except
>>> 1) It looks like they implemented the changes in aops.c from the "cleanup 
>>> blockdev_direct_IO locking" patch that's in 2.6.33.
>>> 2) In journal.c, they rename ocfs2_commit_trigger to ocfs2_frozen_trigger, 
>>> which seems to be from 2.6.35.
>>> 3) In cluster/masklog.c they add a "const" to the mlog_attr_ops declaration
>>> 4) And in quota.h, they are missing "#define QFMT_OCFS2 3"
>>>
>>> Not sure if that helps any, but thanks in any case!
>> All those changes are ok. And unrelated. This is a new one.
>
> Sorry, I think I accidentally wrote a message with only the quoted block... 
> oops.  Sorry.
>
>
> Sunil, are you able to and interested in looking at this issue?  If so, is 
> there any information that I can provide that might help?  Fortunately, after 
> those few initial days of daily errors, it seems to have stopped for now.  
> But of course, I'm still worried about this.
>

http://oss.oracle.com/~smushran/0001-ocfs2-dlm-Use-dlm-track_lock-when-adding-resource-to.patch

This should fix it. But do note that the patch is untested.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Problem with tunefs.ocfs2, similar to fsck.ocfs2 on EL5

[Sunil Mushran]

On 09/27/2011 09:12 AM, Ulf Zimmermann wrote:
> - -Original Message-
>> From: Sunil Mushran [mailto:sunil.mush...@oracle.com]
>> Sent: Monday, September 26, 2011 10:09 AM
>> To: Ulf Zimmermann
>> Cc: ocfs2-users@oss.oracle.com
>> Subject: Re: [Ocfs2-users] Problem with tunefs.ocfs2, similar to fsck.ocfs2 
>> on
>> EL5
>>
>> I'll look at the tunefs issue. But the other one does not make sense.
>> strict_jbd is a compat flag. Mount should work. What is the mount
>> error? As in, in dmesg.
> I don't see any dmesg or /var/log/messages, but the error I saw was from 
> tunefs:
>
> demodb01 root /home/ulf # /usr/bin/yes | /sbin/tunefs.ocfs2 -U -L /export/u07 
> /dev/mapper/u07
> tunefs.ocfs2 1.2.7
> tunefs.ocfs2: Filesystem has unsupported feature(s) while opening device 
> /dev/mapper/u07
>

So that is correct. In short that flag was added to allow us to use the
jbd(2) features. We use this to create volumes > 16TB.

I guess if you want to use with 1.2, format it with 1.2 tools.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Problem with tunefs.ocfs2, similar to fsck.ocfs2 on EL5

[Sunil Mushran]

I'll look at the tunefs issue. But the other one does not make sense.
strict_jbd is a compat flag. Mount should work. What is the mount
error? As in, in dmesg.

On 09/25/2011 04:43 AM, Ulf Zimmermann wrote:
> As tunefs.ocfs2 wasn't working for us, I tried to mkfs.ocfs2 the volumes 
> again with --fs-feature-level=max-compat. This still turns on 
> strict-journal-super and there seems no way around this? This makes the 
> volume not compatible with OCFS 1.2.9
>
>> -Original Message-
>> From: ocfs2-users-boun...@oss.oracle.com [mailto:ocfs2-users-
>> boun...@oss.oracle.com] On Behalf Of Ulf Zimmermann
>> Sent: Sunday, September 25, 2011 1:43 AM
>> To: ocfs2-users@oss.oracle.com
>> Subject: [Ocfs2-users] Problem with tunefs.ocfs2, similar to fsck.ocfs2 on 
>> EL5
>>
>> We are running into a problem which looks like the same we had with
>> fsck.ocfs2 a while back. This is with ocfs2-tools 1.4.4. I am trying to use
>> tunefs.ocfs2 to turn off some features. The program starts up but then starts
>> eating all available memory and more and the system starts to swap like crazy
>> in and out. This is exactly the same behavior as the fsck.ocfs2 for which we
>> were given a patched binary.
>>
>> I tried to compile the tunefs.ocfs2 from 1.6.x but the same problem with that
>> binary.
>>
>> Ulf.
>>
>>
>> ___
>> Ocfs2-users mailing list
>> Ocfs2-users@oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] ΑΠ: Linux kernel crash due to ocfs2

[Sunil Mushran]

#x27;\002', fpr = 0 '\0', reserved = 0,
>  overflow_arg_area = 0xffaaf478, reg_save_area = 0xffaaf3e0}}
>  fd = 4
>  list = (struct str_list *) 0xffaaedf0
>  nlist =
>  cp =
>  written =
> #3  0x0feb1ab4 in _int_free (av=0xffb14a8, mem=)
>  at malloc.c:5768
>  buf = "10045000"
>  cp =
>  p = (mchunkptr) 0x10045000
>  size = 8192
>  nextchunk = (mchunkptr) 0x10047000
>  nextsize = 8192
>  prevsize =
>  bck =
>  fwd =
>  errstr =
> #4  0x0feb5b68 in *__GI___libc_free (mem=0x10046000) at malloc.c:3545
>  __futex =
>  ar_ptr = (mstate) 0xffb14a8
>  p =
>  hook =
> #5  0x10007bb0 in ocfs2_free (ptr=0xffaaf530) at memory.c:65
> No locals.
> #6  0x10002748 in traverse_inode ()
> No symbol table info available.
> #7  0x10001f50 in traverse_group_desc ()
> No symbol table info available.
> #8  0x10002334 in traverse_chains ()
> No symbol table info available.
> #9  0x100026a0 in traverse_inode ()
> No symbol table info available.
> #10 0x10001f50 in traverse_group_desc ()
> No symbol table info available.
> #11 0x10002334 in traverse_chains ()
> No symbol table info available.
> #12 0x100026a0 in traverse_inode ()
> No symbol table info available.
> #13 0x1000358c in scan_raw_disk ()
> No symbol table info available.
> #14 0x10003e28 in main ()
> No symbol table info available.
> #15 0x0fe4dc60 in generic_start_main (main=0x10003a88, argc=3,
>  ubp_av=0xffaafa74, auxvec=0xffaafaec, init=,
>  fini=, rtld_fini=,
>  stack_end=) at ../csu/libc-start.c:231
>  self = (struct pthread *) 0x0
>  result =
>  unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-8358199, 0,
> 265182513, 0,
>  0, 0, 0, 0, 0, 0, 0, 0, 0, 268234624, -5572000, -5571980, 3, 0,
>  268367568, 268107764, 0, 570426402, 0,
> 268372748,
>  -5572296, -5572120, 0, 267932808, -5572100, 0, -1173947391,
> -5572288,
>  0, 268372748, 268369884, -5572320, -5572128, 268219208, 0, 0, 0,
>  265177660, 174420993, 10485760, 0, 0, -2147483648, 1,
>  0, 570426402, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
>mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0xffafff4,
> 0xffaafaec},
>  data = {prev = 0x0, cleanup = 0x0, canceltype = 268107764}}}
> #16 0x0fe4dea0 in __libc_start_main (argc=3, ubp_av=0xffaafa74,
>  ubp_ev=, auxvec=0xffaafaec,
>  rtld_fini=0xffcef80<_dl_fini>, stinfo=0x1001a0c8,
>  stack_on_entry=0xffaafa60)
>  at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:122
> No locals.
> #17 0x in ?? ()
> No symbol table info available.
>
> However, I don't know the package in which the traverse_* functions
> calls can be found. If you know, please let me know so I can install it.
>
> Also, the last time the servers crashed we fsck'ed the filesystem
> without any problem. But, we have added/deleted a lot of space and files
> since then.
>
> Thanks,
>
> George
>
>
>
> On Fri, 2011-09-16 at 11:00 -0700, Sunil Mushran wrote:
>> I got it. But I still don't see the symbols. Maybe we are corrupting the 
>> stack.
>> Maybe this is ppc specific. Do you have a x86/x86_64 box that can access
>> the same volume? If so I could give you a drop of the same for that arch.
>>
>> Also, have to run fsck on this volume before? One reason o2image could
>> fail is if there is a bad block pointer. While it is supposed to handle all 
>> such
>> cases, it is known to miss some cases.
>>
>> On 09/16/2011 12:06 AM, Betzos Giorgos wrote:
>>> Please try http://portal-md.glk.gr/ocfs2/core.32578.bz2
>>>
>>> Please let me know, in case you have any problem downloading it.
>>>
>>> Thanks,
>>>
>>> George
>>>
>>> On Thu, 2011-09-15 at 09:45 -0700, Sunil Mushran wrote:
>>>> I was hoping to get a readable stack. Please could you provide a link to
>>>> the coredump.
>>>>
>>>> On 09/15/2011 02:51 AM, Betzos Giorgos wrote:
>>>>> Hello,
>>>>>
>>>>> I am sorry for the delay in responding. Unfortunately, if faulted again.
>>>>>
>>>>> Here is the log. Although my email client folds the Memory Map lines.
>>>>> The core file is available.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> George
>>>>>
>>>>> # ./o2image.ppc.dbg /dev/mapper/mpath0 /files_shared/u02.o2image
>>>>> *** glibc detected *** 

Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource W0000000000000001b027d69b591f15 not on the Tracking list

[Sunil Mushran]

On 09/21/2011 12:37 PM, Herman L wrote:
>>> On 09/19/2011 08:35 AM, Herman L wrote:
>>> Hi all,
>>>
>>> Got a couple of these messages recently, but I don't know what they mean.  
>>> Can anyone let me know if I need to panic?  I'm using OCFS2 compiled from 
>>> the kernel source of RHEL 6.0's 2.6.32-71.18.2.el6.x86_64.
>>>
>>> Sep 19 08:07:15 server-1 kernel: [3892420.40] 
>>> (10387,12):dlm_lockres_release:507 ERROR: Resource 
>>> W0001b027d69b591f15 not on the Tracking list
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398194] lockres: 
>>> W0001b027d69b591f1, owner=1, state=0
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398195]   last used: 8197071325, 
>>> refcnt: 0, on purge list: no
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398197]   on dirty list: no, on 
>>> reco list: no, migrating pending: no
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398198]   inflight locks: 0, asts 
>>> reserved: 0
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398199]   refmap nodes: [ ], 
>>> inflight=0
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]   granted queue:
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398200]   converting queue:
>>> Sep 19 08:07:15 server-1 kernel: [3892420.398201]   blocked queue:
>>>
>>> Thanks!
>>> Herman
>> From: Sunil Mushran
>> To: Herman L
>> Sent: Monday, September 19, 2011 12:57 PM
>> Subject: Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource 
>> W0001b027d69b591f15 not on the Tracking list
>>
>>I've no idea of the state of the source that you are using. The message
>>is a warning indicating a race. While it probably did not affect the 
>> functioning,
>>there is no guarantee that that would be the case the next time around.
>>
>>The closest relevant patch is over 2 years old.
>> http://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=b0d4f817ba5de8adb875ace594554a96d7737710
> Thanks Sunil for responding.  I know you can't easily support my setup, but 
> anyways I checked the sources.
>
> Looks like the patch you mention is in the sources I compiled from ( RHEL6.0 
> kernel-2.6.32-71.24.1.el6.src.rpm ), so I guess the source of the problem is 
> elsewhere.
>
> The fs/ocfs2 directory from the RHEL6 sources I compiled from is almost 
> exactly the same as the mainline 2.6.32 kernel, except
> 1) It looks like they implemented the changes in aops.c from the "cleanup 
> blockdev_direct_IO locking" patch that's in 2.6.33.
> 2) In journal.c, they rename ocfs2_commit_trigger to ocfs2_frozen_trigger, 
> which seems to be from 2.6.35.
> 3) In cluster/masklog.c they add a "const" to the mlog_attr_ops declaration
> 4) And in quota.h, they are missing "#define QFMT_OCFS2 3"
>
> Not sure if that helps any, but thanks in any case!

All those changes are ok. And unrelated. This is a new one.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] 11gr1 RAC + ocfs2 node2 is down and not able to mount the ocfs2 FS on node1

[Sunil Mushran]

The connect is failing. One of the main reason is a firewall.
See if iptables is running. Check on both nodes. If so, shutdown
it down or add a rule to allow traffic on the o2cb port.

On 09/18/2011 08:57 PM, veeraa bose wrote:

Hi All,


we are having two node 11gr1 RAC (we have used ocfs2 for CRS and ASM for DB 
data), now the node2 is down and node1 got rebooted and after node1 is the 
ocfs2 Fs used for CRS is not getting mounted and the error is.

 #/etc/init.d/ocfs2 start
Starting Oracle Cluster File System (OCFS2) mount.ocfs2: Transport endpoint is 
not connected while mounting /dev/mapper/vg_oracle_shared-
RAC--DG--CLUS--01 on /u02/ocfs2/RAC-DG-CLUS-01. Check 'dmesg' for more 
information on this error.
mount.ocfs2: Transport endpoint is not connected while mounting 
/dev/mapper/vg_oracle_shared-RAC--DG--CLUS--02 on /u02/ocfs2/RAC-DG-CLUS-02. 
Check 'dmesg' for more information on this error.
mount.ocfs2: Transport endpoint is not connected while mounting 
/dev/mapper/vg_oracle_shared-global_backup on /global/backup. Check 'dmesg' for 
more information on this error.
   [FAILED]

And below is the log from Dmesg.

(o2net,6121,4):o2net_connect_expired:1664 ERROR: no connection established with 
node 2 after 60.0 seconds, giving up and returning errors.
(mount.ocfs2,7327,12):dlm_request_join:1036 ERROR: status = -107
(mount.ocfs2,7327,12):dlm_try_to_join_domain:1210 ERROR: status = -107
(mount.ocfs2,7327,12):dlm_join_domain:1488 ERROR: status = -107
(mount.ocfs2,7327,12):dlm_register_domain:1754 ERROR: status = -107
(mount.ocfs2,7327,12):ocfs2_dlm_init:2808 ERROR: status = -107
(mount.ocfs2,7327,12):ocfs2_mount_volume:1447 ERROR: status = -107
ocfs2: Unmounting device (253,19) on (node 1)

please guideme,  how to mount the ocfs2 FS on the node1 and bring the cluster 
up.

Thanks
Veera.


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] dlm_lockres_release:507 ERROR: Resource W0000000000000001b027d69b591f15 not on the Tracking list

[Sunil Mushran]

I've no idea of the state of the source that you are using. The message
is a warning indicating a race. While it probably did not affect the 
functioning,
there is no guarantee that that would be the case the next time around.

The closest relevant patch is over 2 years old.
http://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=b0d4f817ba5de8adb875ace594554a96d7737710

On 09/19/2011 08:35 AM, Herman L wrote:

Hi all,

Got a couple of these messages recently, but I don't know what they mean.  Can 
anyone let me know if I need to panic?  I'm using OCFS2 compiled from the 
kernel source of RHEL 6.0's 2.6.32-71.18.2.el6.x86_64.

Sep 19 08:07:15 server-1 kernel: [3892420.40] 
(10387,12):dlm_lockres_release:507 ERROR: Resource 
W0001b027d69b591f15 not on the Tracking list
Sep 19 08:07:15 server-1 kernel: [3892420.398194] lockres: 
W0001b027d69b591f1, owner=1, state=0
Sep 19 08:07:15 server-1 kernel: [3892420.398195]   last used: 8197071325, 
refcnt: 0, on purge list: no
Sep 19 08:07:15 server-1 kernel: [3892420.398197]   on dirty list: no, on reco 
list: no, migrating pending: no
Sep 19 08:07:15 server-1 kernel: [3892420.398198]   inflight locks: 0, asts 
reserved: 0
Sep 19 08:07:15 server-1 kernel: [3892420.398199]   refmap nodes: [ ], 
inflight=0
Sep 19 08:07:15 server-1 kernel: [3892420.398200]   granted queue:
Sep 19 08:07:15 server-1 kernel: [3892420.398200]   converting queue:
Sep 19 08:07:15 server-1 kernel: [3892420.398201]   blocked queue:

Thanks!
Herman


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] fsck doesn't fix "bad chain"

[Sunil Mushran]

Can you save the o2image of the volume when it is in that state.
We'll need that for analysis.

On 09/16/2011 05:41 AM, Andre Nathan wrote:
> Hello
>
> For a while I had seen errors like this in the kernel logs:
>
>OCFS2: ERROR (device drbd5): ocfs2_validate_gd_parent: Group
>descriptor #69084874 has bad chain 126
>File system is now read-only due to the potential of on-disk
>corruption. Please run fsck.ocfs2 once the file system is unmounted.
>
> This always happened in the same device, and whenever it happened I ran
> fsck.ocfs2 -fy /dev/drbd5, which showed messages like these:
>
>[GROUP_FREE_BITS] Group descriptor at block 201309696 claims to have
>9893 free bits which is more than 9886 bits indicated by the bitmap.
>Drop its free bit count down to the total? y
>[CHAIN_BITS] Chain 166 in allocator inode 11 has 1264713 bits
>marked free out of 1516032 total bits but the block groups in the
>chain have 1264706 free out of 1516032 total.  Fix this by updating
>the chain record? y
>[CHAIN_GROUP_BITS] Allocator inode 11 has 79407510 bits marked used
>out of 365955414 total bits but the chains have 79407911 used out of
>365955414 total.  Fix this by updating the inode counts? y
>[INODE_COUNT] Inode 69085510 has a link count of 0 on disk but
>directory entry references come to 1. Update the count on disk to
>match? y
>
> As time passed, the frequency of these issues started to increase, and
> the last time it happened, I decided to run fsck twice in a row, and was
> surprised to see it showed the same messages in both runs. It seems it
> was unable to fix the problem.
>
> I identified the files corresponding to the inodes using debugfs.ocfs2
> and copied them to a new place, and then moved the copy over the
> original file, in order to recreate the inodes. Whenever I did that for
> one inode, the error above happened and the filesystem became read-only,
> so I had to umount/mount the volume again in order to be able to write
> to it again.
>
> After doing this, I ran fsck.ocfs2 -fy again twice, and no errors were
> reported. Since then I haven't seen this problem again.
>
> I'm running kernel 2.6.35 and ocfs2-tools 1.6.4.
>
> Has anyone else seen an issue like that?
>
> Thanks
> Andre
>
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Linux kernel crash due to ocfs2

[Sunil Mushran]

I got it. But I still don't see the symbols. Maybe we are corrupting the stack.
Maybe this is ppc specific. Do you have a x86/x86_64 box that can access
the same volume? If so I could give you a drop of the same for that arch.

Also, have to run fsck on this volume before? One reason o2image could
fail is if there is a bad block pointer. While it is supposed to handle all such
cases, it is known to miss some cases.

On 09/16/2011 12:06 AM, Betzos Giorgos wrote:
> Please try http://portal-md.glk.gr/ocfs2/core.32578.bz2
>
> Please let me know, in case you have any problem downloading it.
>
> Thanks,
>
> George
>
> On Thu, 2011-09-15 at 09:45 -0700, Sunil Mushran wrote:
>> I was hoping to get a readable stack. Please could you provide a link to
>> the coredump.
>>
>> On 09/15/2011 02:51 AM, Betzos Giorgos wrote:
>>> Hello,
>>>
>>> I am sorry for the delay in responding. Unfortunately, if faulted again.
>>>
>>> Here is the log. Although my email client folds the Memory Map lines.
>>> The core file is available.
>>>
>>> Thanks,
>>>
>>> George
>>>
>>> # ./o2image.ppc.dbg /dev/mapper/mpath0 /files_shared/u02.o2image
>>> *** glibc detected *** ./o2image.ppc.dbg: corrupted double-linked list:
>>> 0x10075000 ***
>>> === Backtrace: =
>>> /lib/libc.so.6[0xfeb1ab4]
>>> /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
>>> ./o2image.ppc.dbg[0x1000d098]
>>> ./o2image.ppc.dbg[0x1000297c]
>>> ./o2image.ppc.dbg[0x10001eb8]
>>> ./o2image.ppc.dbg[0x1000228c]
>>> ./o2image.ppc.dbg[0x10002804]
>>> ./o2image.ppc.dbg[0x10001eb8]
>>> ./o2image.ppc.dbg[0x1000228c]
>>> ./o2image.ppc.dbg[0x10002804]
>>> ./o2image.ppc.dbg[0x10003bbc]
>>> ./o2image.ppc.dbg[0x10004480]
>>> /lib/libc.so.6[0xfe4dc60]
>>> /lib/libc.so.6[0xfe4dea0]
>>> === Memory map: 
>>> 0010-0012 r-xp 0010 00:00 0
>>> [vdso]
>>> 0f43-0f44 r-xp  08:13
>>> 180307 /lib/libcom_err.so.2.1
>>> 0f44-0f45 rw-p  08:13
>>> 180307 /lib/libcom_err.so.2.1
>>> 0f90-0f9c r-xp  08:13
>>> 180293 /lib/libglib-2.0.so.0.1200.3
>>> 0f9c-0f9d rw-p 000b 08:13
>>> 180293 /lib/libglib-2.0.so.0.1200.3
>>> 0fa4-0fa5 r-xp  08:13
>>> 180292 /lib/librt-2.5.so
>>> 0fa5-0fa6 r--p  08:13
>>> 180292 /lib/librt-2.5.so
>>> 0fa6-0fa7 rw-p 0001 08:13
>>> 180292 /lib/librt-2.5.so
>>> 0fce-0fd0 r-xp  08:13
>>> 180291 /lib/libpthread-2.5.so
>>> 0fd0-0fd1 r--p 0001 08:13
>>> 180291 /lib/libpthread-2.5.so
>>> 0fd1-0fd2 rw-p 0002 08:13
>>> 180291 /lib/libpthread-2.5.so
>>> 0fe3-0ffa r-xp  08:13
>>> 180288 /lib/libc-2.5.so
>>> 0ffa-0ffb r--p 0016 08:13
>>> 180288 /lib/libc-2.5.so
>>> 0ffb-0ffc rw-p 0017 08:13
>>> 180288 /lib/libc-2.5.so
>>> 0ffc-0ffe r-xp  08:13
>>> 180287 /lib/ld-2.5.so
>>> 0ffe-0fff r--p 00010000 08:13
>>> 180287 /lib/ld-2.5.so
>>> 0fff-1000 rw-p 0002 08:13
>>> 180287 /lib/ld-2.5.so
>>> 1000-1005 r-xp  08:13
>>> 7487795/root/o2image.ppc.dbg
>>> 1005-1006 rw-p 0004 08:13
>>> 7487795    /root/o2image.ppc.dbg
>>> 1006-1009 rwxp 1006 00:00 0
>>> [heap]
>>> f768-f7ff rw-p f768 00:00 0
>>> ff9a-ffaf rw-p ff9a 00:00 0
>>> [stack]
>>> Aborted (core dumped)
>>>
>>>
>>> On Thu, 2011-09-08 at 12:10 -0700, Sunil Mushran wrote:
>>>> http://oss.oracle.com/~smushran/o2image.ppc.dbg
>>>>
>>>> Use the above executable. Hoping it won't fault. But if it does
>>>> email me the backtrace. That trace will be readable as the exec
>>>> has debugging symbols enabled.
>>>>
>>>> On 09/07/2011 

Re: [Ocfs2-users] Trouble getting node to re-join two node cluster (OCFS2/DRBD Primary/Primary)

[Sunil Mushran]

-F does not run the full fsck. -f does.

But I would not recommend running fsck as this corruption is not
normal. The inodes in the system directory have been overwritten.
That typically means a storage issue. The fs does not create/remove
inodes in sysdir. Only the tools do that.

You may want to shutdown drbd and access the devices directly on
the two machines. See if they are ok. If so, then select one as the
master and copy it to the other.

On 09/15/2011 04:20 PM, Mike Reid wrote:

I may have made some progress on my OCFS2 error:

See the following output from "dmesg"

[88740.345617] OCFS2: ERROR (device drbd0): ocfs2_validate_inode_block: Invalid 
dinode #11: fs_generation is 376662488
[88740.345664] File system is now read-only due to the potential of on-disk 
corruption. Please run fsck.ocfs2 once the file system is unmounted.
[88740.345710] (mount.ocfs2,26394,5):ocfs2_read_locked_inode:499 ERROR: status 
= -22
[88740.345743] (mount.ocfs2,26394,5):_ocfs2_get_system_file_inode:120 ERROR: 
status = -116
[88740.345807] (mount.ocfs2,26394,5):ocfs2_init_global_system_inodes:466 ERROR: 
status = -22
[88740.345890] (mount.ocfs2,26394,5):ocfs2_init_global_system_inodes:469 ERROR: 
Unable to load system inode 4, possibly corrupt fs?
[88740.345958] (mount.ocfs2,26394,5):ocfs2_initialize_super:2261 ERROR: status 
= -22
[88740.346067] (mount.ocfs2,26394,5):ocfs2_fill_super:1023 ERROR: status = -22
[88740.346124] ocfs2: Unmounting device (147,0) on (node 0)


I decided to run "fsck.ocfs2 -F /dev/drbd0":

Checking OCFS2 filesystem in /dev/drbd0:
  label: 
  uuid:   fe 42 73 e1 f8 66 45 41 bb cf 66 c5 df d4 96 d6
  number of blocks:   2436
  bytes per block:4096
  number of clusters: 2436
  bytes per cluster:  4096
  max slots:  8

/dev/drbd0 wasn't cleanly unmounted by all nodes.  Attempting to replay the 
journals for nodes that didn't unmount cleanly
Checking each slot's journal.
Replaying slot 0's journal.
Slot 0's journal replayed successfully.
Slot 0's local alloc replayed successfully
/dev/drbd0 is clean.  It will be checked after 20 additional mounts.
Slot 0's journal dirty flag removed


Unfortunately, I still cannot mount the fs

> mount -t ocfs2 /dev/drbd/by-res/repdata /data

(see attached strace) 


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Trouble getting node to re-join two node cluster (OCFS2/DRBD Primary/Primary)

[Sunil Mushran]

open("/dev/drbd0", O_RDONLY|O_DIRECT) = -1 EMEDIUMTYPE (Wrong medium type)

drbd_open()
...
if (mdev->state.role != R_PRIMARY) {
if (mode & FMODE_WRITE)
rv = -EROFS;
else if (!allow_oos)
rv = -EMEDIUMTYPE;
}
...

So the failure appears to be emanating from drbd. There seems
to be a allow_oos module param that is not 0. I have no idea
what this param does. Also, am reading current mainline. 2.6.35 may
be different.

On 09/15/2011 01:26 PM, Mike Reid wrote:

Hello all,

** I have also posted this in the pacemaker list, but I have a feeling it's
more OCFS2 specific **

We have a two-node cluster still in development that has been running fine
for weeks (little to no traffic). I made some updates to our CIB recently,
and everything seemed just fine.

Yesterday I attempted to untar ~1.5GB to the OCFS2/DRBD volume, and once it
was complete one of the nodes had become completely disconnected and I
haven't been able to reconnect since.

DRBD is working fine, everything is UpToDate and I can get both nodes in
Primary/Primary, but when it comes down to starting OCFS2 and mounting the
volume, I'm left with:


resFS:0_start_0 (node=node1, call=21, rc=1, status=complete): unknown error

I am using "pcmk" as the cluster_stack, and letting Pacemaker control
everything...

The last time this happened the only way I was able to resolve it was to
reformat the device (via mkfs.ocfs2 -F). I don't think I should have to do
this, underlying blocks seem fine, and one of the nodes is running just
fine. The (currently) unmounted node is staying in sync as far as DRBD is
concerned.

Here's some detail that hopefully will help, please let me know if there's
anything else I can provide to help know the best way to get this node back
"online":


Ubuntu 10.10 / Kernel 2.6.35

Pacemaker 1.0.9.1
Corosync 1.2.1
Cluster Agents 1.0.3 (Heartbeat)
Cluster Glue 1.0.6
OpenAIS 1.1.2

DRBD 8.3.10
OCFS2 1.5.0

cat /sys/fs/ocfs2/cluster_stack = pcmk

node1: mounted.ocfs2 -d

DeviceFS UUID  Label
/dev/sda3 ocfs2  fe4273e1-f866-4541-bbcf-66c5dfd496d6

node2: mounted.ocfs2 -d

DeviceFS UUID  Label
/dev/sda3 ocfs2  d6f7cc6d-21d1-46d3-9792-bc650736a5ef
/dev/drbd0ocfs2  d6f7cc6d-21d1-46d3-9792-bc650736a5ef

* NOTES:
- Both nodes are identical, in fact one node is a direct mirror (hdd clone)
- I have attached the CIB (crm configure edit contents) and mount trace



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] The mounting of too many OCFS2 volumes (i.e. 50 or more) per cluster

[Sunil Mushran]

That's very old. We have users having 50+ mounts. The one disadvantage
is that the o2cb stack heartbeats on all mounts. That problem will be addressed
in 1.8 (the tools will be released soon), with global heartbeat (hb volumes
are user-configurable).

Having said that, the number of volumes depends on the hardware capability.
It is hard to provide simple rules for this. The best solution is to test and 
figure
out the perf bottleneck.

On 09/15/2011 01:00 AM, Marko Sutic wrote:
> Hi list,
>
> I have a question concerning number of OCFS2 volumes per cluster.
>
> From our storage vendor we received recommendations how to configure mount 
> volumes per database to gain the best possible performance.
> Basically, we should separate redo logs,archive logs, temp, data, etc.
>
> This is not hard to configure but I'm concerned about this line that I've 
> found on Oracle support site:
>
> Linux OCFS2 - Best Practices [ID 603080.1]
> Number of volumes
> The mounting of too many OCFS2 volumes (i.e. 50 or more) per cluster is 
> likely to create a performance (process) bottleneck - this is not 
> specifically related to OCFS2. Ideally, it is desirable to have no more than 
> around 20 OCFS2 partitions per system.
> See also http://oss.oracle.com/bugzilla/show_bug.cgi?id=992
>
>
>
> In our configuration we would need more than 60 OCFS2 mount volumes per 
> cluster so I don't know should we expect any performance problems due to the 
> number of OCFS2 volumes?
> What is your recommendation about number of OCFS2 volumes per cluster 
> regarding performance and stability?
>
>
> Our kernel and ocfs2 version:
> # uname -rvp
> 2.6.18-274.0.0.0.1.el5 #1 SMP Mon Jul 25 14:33:14 EDT 2011 x86_64
>
> # rpm -qa|grep ocfs2
> ocfs2-tools-1.6.3-2.el5
> ocfs2-2.6.18-274.0.0.0.1.el5-1.4.8-2.el5
> ocfs2console-1.6.3-2.el5
>
>
>
> Thank you very much for your help.
>
> Regards,
> Marko Sutic


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Linux kernel crash due to ocfs2

[Sunil Mushran]

I was hoping to get a readable stack. Please could you provide a link to
the coredump.

On 09/15/2011 02:51 AM, Betzos Giorgos wrote:
> Hello,
>
> I am sorry for the delay in responding. Unfortunately, if faulted again.
>
> Here is the log. Although my email client folds the Memory Map lines.
> The core file is available.
>
> Thanks,
>
> George
>
> # ./o2image.ppc.dbg /dev/mapper/mpath0 /files_shared/u02.o2image
> *** glibc detected *** ./o2image.ppc.dbg: corrupted double-linked list:
> 0x10075000 ***
> === Backtrace: =
> /lib/libc.so.6[0xfeb1ab4]
> /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
> ./o2image.ppc.dbg[0x1000d098]
> ./o2image.ppc.dbg[0x1000297c]
> ./o2image.ppc.dbg[0x10001eb8]
> ./o2image.ppc.dbg[0x1000228c]
> ./o2image.ppc.dbg[0x10002804]
> ./o2image.ppc.dbg[0x10001eb8]
> ./o2image.ppc.dbg[0x1000228c]
> ./o2image.ppc.dbg[0x10002804]
> ./o2image.ppc.dbg[0x10003bbc]
> ./o2image.ppc.dbg[0x10004480]
> /lib/libc.so.6[0xfe4dc60]
> /lib/libc.so.6[0xfe4dea0]
> === Memory map: 
> 0010-0012 r-xp 0010 00:00 0
> [vdso]
> 0f43-0f44 r-xp  08:13
> 180307 /lib/libcom_err.so.2.1
> 0f44-0f45 rw-p  08:13
> 180307 /lib/libcom_err.so.2.1
> 0f90-0f9c r-xp  08:13
> 180293 /lib/libglib-2.0.so.0.1200.3
> 0f9c-0f9d rw-p 000b 08:13
> 180293 /lib/libglib-2.0.so.0.1200.3
> 0fa4-0fa5 r-xp  08:13
> 180292 /lib/librt-2.5.so
> 0fa5-0fa6 r--p  08:13
> 180292 /lib/librt-2.5.so
> 0fa6-0fa7 rw-p 0001 08:13
> 180292 /lib/librt-2.5.so
> 0fce-0fd0 r-xp  08:13
> 180291 /lib/libpthread-2.5.so
> 0fd0-0fd1 r--p 0001 08:13
> 180291 /lib/libpthread-2.5.so
> 0fd1-0fd2 rw-p 0002 08:13
> 180291 /lib/libpthread-2.5.so
> 0fe3-0ffa r-xp  08:13
> 180288 /lib/libc-2.5.so
> 0ffa-0ffb r--p 0016 08:13
> 180288 /lib/libc-2.5.so
> 0ffb-0ffc rw-p 0017 08:13
> 180288 /lib/libc-2.5.so
> 0ffc-0ffe r-xp  08:13
> 180287 /lib/ld-2.5.so
> 0ffe-0fff r--p 0001 08:13
> 180287 /lib/ld-2.5.so
> 0fff-1000 rw-p 0002 08:13
> 180287 /lib/ld-2.5.so
> 1000-1005 r-xp  08:13
> 7487795/root/o2image.ppc.dbg
> 1005-1006 rw-p 0004 08:13
> 7487795/root/o2image.ppc.dbg
> 10060000-10090000 rwxp 1006 00:00 0
> [heap]
> f768-f7ff rw-p f768 00:00 0
> ff9a-ffaf rw-p ff9a 00:00 0
> [stack]
> Aborted (core dumped)
>
>
> On Thu, 2011-09-08 at 12:10 -0700, Sunil Mushran wrote:
>> http://oss.oracle.com/~smushran/o2image.ppc.dbg
>>
>> Use the above executable. Hoping it won't fault. But if it does
>> email me the backtrace. That trace will be readable as the exec
>> has debugging symbols enabled.
>>
>> On 09/07/2011 11:24 PM, Betzos Giorgos wrote:
>>> # rpm -q ocfs2-tools
>>> ocfs2-tools-1.4.4-1.el5.ppc
>>>
>>> On Wed, 2011-09-07 at 09:13 -0700, Sunil Mushran wrote:
>>>> version of ocfs2-tools?
>>>>
>>>> On 09/07/2011 09:10 AM, Betzos Giorgos wrote:
>>>>> Hello,
>>>>>
>>>>> I tried what you suggested but here is what I got:
>>>>>
>>>>> # o2image /dev/mapper/mpath0 /files_shared/u02.o2image
>>>>> *** glibc detected *** o2image: corrupted double-linked list: 0x10045000 
>>>>> ***
>>>>> === Backtrace: =
>>>>> /lib/libc.so.6[0xfeb1ab4]
>>>>> /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
>>>>> o2image[0x10007bb0]
>>>>> o2image[0x10002748]
>>>>> o2image[0x10001f50]
>>>>> o2image[0x10002334]
>>>>> o2image[0x100026a0]
>>>>> o2image[0x10001f50]
>>>>> o2image[0x10002334]
>>>>> o2image[0x100026a0]
>>>>> o2image[0x1000358c]
>>>>> o2image[0x10003e28]
>>>>> /lib/libc.so.6[0xfe4dc60]
>>>>> /lib/libc.so.6[0xfe4dea0]
>>>>> === Memory map: 
>>>>> 0010-0012 r-xp 0010 00:00 0

Re: [Ocfs2-users] Syslog reports (ocfs2_wq, 15527, 2):ocfs2_orphan_del:1841 ERROR: status = -2

[Sunil Mushran]

-xr-x   2 0 04096
> 22-May-2008 12:01 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
>
> Working on /dev/mapper/ph1d_archp1
> debugfs.ocfs2 1.4.4
> debugfs: ls -l //orphan_dir:
>  12  drwxr-xr-x   2 0 0   12288
> 14-Sep-2011 11:30 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0001
>  13  drwxr-xr-x   2 0 08192
> 23-Mar-2010 18:03 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0002
>  14  drwxr-xr-x   2 0 04096
> 22-May-2008 12:01 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0003
>  15  drwxr-xr-x   2 0 04096
> 22-May-2008 12:01 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
>
> Working on /dev/mapper/mpath21p1
> debugfs.ocfs2 1.4.4
> debugfs: ls -l //orphan_dir:
>  12  drwxr-xr-x   2 0 04096
> 5-Jun-2008 14:50 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0001
>  13  drwxr-xr-x   2 0 04096
> 21-Jun-2008 16:42 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0002
>  14  drwxr-xr-x   2 0 04096
> 22-May-2008 12:01 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
> debugfs: ls -l //orphan_dir:0003
>  15  drwxr-xr-x   2 0 04096
> 22-May-2008 12:01 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 12:01 ..
>
> Working on /dev/mapper/mpath20p1
> debugfs.ocfs2 1.4.4
> debugfs: ls -l //orphan_dir:
>  12  drwxr-xr-x   2 0 04096
> 3-Jun-2008 16:59 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:58 ..
> debugfs: ls -l //orphan_dir:0001
>  13  drwxr-xr-x   2 0 04096
> 21-Jun-2008 17:39 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:58 ..
> debugfs: ls -l //orphan_dir:0002
>  14  drwxr-xr-x   2 0 04096
> 22-May-2008 11:58 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:58 ..
> debugfs: ls -l //orphan_dir:0003
>  15  drwxr-xr-x   2 0 04096
> 22-May-2008 11:58 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:58 ..
>
> Working on /dev/mapper/mpath18p1
> debugfs.ocfs2 1.4.4
> debugfs: ls -l //orphan_dir:
>  12  drwxr-xr-x   2 0 04096
> 9-Jun-2008 13:54 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:56 ..
> debugfs: ls -l //orphan_dir:0001
>  13  drwxr-xr-x   2 0 04096
> 22-May-2008 11:56 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:56 ..
> debugfs: ls -l //orphan_dir:0002
>  14  drwxr-xr-x   2 0 04096
> 22-May-2008 11:56 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:56 ..
> debugfs: ls -l //orphan_dir:0003
>  15  drwxr-xr-x   2 0 04096
> 22-May-2008 11:56 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:56 ..
>
> Working on /dev/mapper/mpath19p1
> debugfs.ocfs2 1.4.4
> debugfs: ls -l //orphan_dir:
>  12  drwxr-xr-x   2 0 04096
> 3-Jun-2008 17:47 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:57 ..
> debugfs: ls -l //orphan_dir:0001
>  13  drwxr-xr-x   2 0 04096
> 30-Aug-2009 14:55 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:57 ..
> debugfs: ls -l //orphan_dir:0002
>  14  drwxr-xr-x   2     0     04096
> 22-May-2008 11:57 .
>  6   drwxr-xr-x   6 0 04096
> 22-May-2008 11:57 ..
> debugfs: ls -l //orphan_dir:00

Re: [Ocfs2-users] Syslog reports (ocfs2_wq, 15527, 2):ocfs2_orphan_del:1841 ERROR: status = -2

[Sunil Mushran]

The issue that caused it has been fixed. The fix is here.
http://oss.oracle.com/git/?p=ocfs2-1.4.git;a=commit;h=b6f3de3fd54026df748bfd1449bbe31b9803f8f7

The actual problem could have happened much earlier.
1.4.4 is showing the messages as it is more aggressive (than 1.4.1)
in cleaning up the orphans. By default, the fs scans for orphans
once every 10 mins on a node in the cluster.

fsck should fix it. I would have to think you must not have fscked
that volume.

You can use debugfs.ocfs2 to look at the orphan dirs. List the
system dir using "ls-l //". Then list orphan dirs using "ls -l 
//orphan_dir:",

etc. Look at the timestamp on the inodes. The one that is oldest could
be the problematic one. That way you know which volume to fsck.
BTW, it is safe to run debugfs while the fs is mounted. At worst it will
provide you stale info.

On 09/15/2011 07:40 AM, Daniel Keisling wrote:

Hello,
I recently upgraded from OCFS2 v1.4.1 running on RHEL 5.1 with 
kernel-2.6.18-92.1.13.el5 to OCFS2 v1.4.4 running on RHEL 5.6 with 
kernel-2.6.18-194.32.1.el5.  I now see this is syslog every couple of 
minutes:
Sep 15 09:31:51 ausracdbd01 kernel: 
(ocfs2_wq,15527,2):ocfs2_orphan_del:1841 ERROR: status = -2
Sep 15 09:31:51 ausracdbd01 kernel: 
(ocfs2_wq,15527,2):ocfs2_remove_inode:628 ERROR: status = -2
Sep 15 09:31:51 ausracdbd01 kernel: 
(ocfs2_wq,15527,2):ocfs2_wipe_inode:754 ERROR: status = -2
Sep 15 09:31:51 ausracdbd01 kernel: 
(ocfs2_wq,15527,2):ocfs2_delete_inode:999 ERROR: status = -2
This is happening on all cluster nodes, on 3 separate OCFS2 clusters.  
I have performed a 'fsck.ocfs2 -f -y ' on _most_ of the 
filesystems, but not all.  fsck has always came back clean.

Does anyone know the source and fix for this error?
Thanks,
Daniel
*Daniel Keisling*
*Sr. Systems Administrator II*
*Information Technology*

PPD
7551 Metro Center Drive, Suite 300
Austin, TX 78744


*Phone* +1 512 747 5256
*Cell*  +1 512 653 1895
*Fax*   +1 512 685 7256
*e-mail* 	daniel.keisl...@ppdi.com 
mailto:daniel.keisl...@ppdi.com>

*Web site*  www.ppdi.com http://www.ppdi.com/>


This email transmission and any documents, files or previous email 
messages attached to it may contain information that is confidential 
or legally privileged.
If you are not the intended recipient or a person responsible for 
delivering this transmission to the intended recipient, you are hereby 
notified
that you must not read this transmission and that any disclosure, 
copying, printing, distribution or use of this transmission is 
strictly prohibited.
If you have received this transmission in error, please immediately 
notify the sender by telephone or return email and delete the original 
transmission and its attachments without reading or saving in any manner.



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] No space left on device

[Sunil Mushran]

On 09/14/2011 03:21 PM, Florin Andrei wrote:
> On 09/14/2011 03:11 PM, Florin Andrei wrote:
>> It's a 2-node cluster. I rebooted one node, waited until it came up, and
>> now I can create files on that volume:
> Nope, it's doing it again. :(
>
> # touch test
> touch: cannot touch `test': No space left on device

old issue.

If on mainline:
Upgrade to mainline kernel > 2.6.36
install ocfs2-tools 1.6+
Run tunefs.ocfs2 --fs-feature=discontig-bg

If on el5:
Upgrade to the uek kernel
Upgrade tools and run tunefs.

Basically the vol is fragmented enough to not alllow the fs
to extend the inode allocator. The discontig-bg fixes by
allowing this allocator to be discontiguous.

A quick fix is to free up large files and so some touch of
empty files. If the large file was contiguous enough, you
should be able to extend that allocator. Another short term
fix is to free up unused slots, if you have any (using tunefs).
This will free up the journal which is almost always large
and contiguous.


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] mount type heartbeat=local

[Sunil Mushran]

To mount without heartbeat, you have to make it a local mount.
Maybe mount.ocfs2 should error out if user specifies heartbeat=...

To make it mount locally:
umount on all nodes
tunefs.ocfs2 --fs-features=local /dev/mapper/
mount /dev/mapper/...  /mnt/path

To revert:
umount
tunefs.ocfs2 --fs-features=nolocal /dev/mapper/...

On 09/14/2011 12:23 PM, Ulf Zimmermann wrote:


Sunil, I was trying to see what the speed difference would be if I mount a 
volume with no heartbeat (for copy purpose):

dbtest01 root /home/ulf # mount -o datavolume,nointr,heartbeat=none 
/dev/mapper/350002ac001380dab /export/coptest/

dbtest01 root /home/ulf # mount

/dev/mapper/350002ac001380dab on /export/coptest type ocfs2 
(rw,_netdev,datavolume,nointr,heartbeat=none,heartbeat=local)

This was with o2cb running, is this right? This is on EL5 with ocfs2 1.4.7 and 
tools 1.4.4

*From:*ocfs2-users-boun...@oss.oracle.com 
[mailto:ocfs2-users-boun...@oss.oracle.com] *On Behalf Of *Sunil Mushran
*Sent:* Friday, September 09, 2011 9:46 PM
*To:* Hai Tao
*Cc:* ocfs2-users@oss.oracle.com
*Subject:* Re: [Ocfs2-users] mount type heartbeat=local

That's mount type. Yes, we should not have overloaded the term "local".

On 09/09/2011 07:53 PM, Hai Tao wrote:

but this is what I saw in the guide


  OCFS2 - A Cluster File System For Linux 
<http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.4/ocfs2-1_4-usersguide.pdf>


**

*Mount Type*

Valid types are

/cluster /and /local/, with the former also being the default. Specify

/
local/if you intend to use the file system on one node only.



Thanks.

Hai Tao


--

Date: Fri, 9 Sep 2011 19:04:28 -0700
From: sunil.mush...@oracle.com <mailto:sunil.mush...@oracle.com>
To: taoh...@hotmail.com <mailto:taoh...@hotmail.com>
CC: ocfs2-users@oss.oracle.com <mailto:ocfs2-users@oss.oracle.com>
Subject: Re: [Ocfs2-users] mount type heartbeat=local

That mount option is appended by mount.ocfs2. It tells users
the heartbeat mode. "none" means non-clustered. "local" means
the heartbeat region is on the mounted volume. This is the default
mode. In 1.8 we have "global" which means the heartbeat region
has been configured on 1+ devices. local and global are only
pertinent in the o2cb stack. If you are using pacemaker, you will
see cluster_stack=pcmk and not heartbeat=.

There is no heartbeat=cluster.

On 09/09/2011 04:43 PM, Hai Tao wrote:

Hi,

when I mount ocfs2 file system, what the heartbeat=local option means? what is 
the difference between heartbeat=local and heartbeat=cluster (I know that is 
default)?

Thanks.

Hai



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] node_count=0

[Sunil Mushran]

It is wrong config.

On 09/09/2011 10:15 PM, Hai Tao wrote:

I have a two node ocfs2 cluster, and in the /etc/ocfs2/cluster.conf file, the 
node_count=0 rather than 2. Does this have to be a wrong config, and how would 
this affect the cluster?

Thanks.
Hai Tao



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] disable heartbeat nic caused ocfs2 errors

[Sunil Mushran]

ocfs2 uses disk heartbeat to detect node liveness. It uses net heartbeat
to detect link liveness. Both need to operate for the cluster to function.
If the network link between two nodes snaps, then one of the two nodes
is fenced.

The stack below indicates that the two nodes are not able to communicate.
The two nodes are waiting on the quorum to fence one of the nodes.
It appears you have upped the disk heartbeat timeout > 2mins. I would imagine
one of the nodes reset after that timeout.

On 09/10/2011 08:54 PM, Hai Tao wrote:

is ocfs2 heartbeat transferred over the network, or just updating a file to the 
shared disk?

If the heartbeat lost, what should happen? what if only one node is writing, 
and the other is still? Will it still cause any file system issue?


Thanks.
Hai Tao


--
From: taoh...@hotmail.com
To: ocfs2-users@oss.oracle.com
Date: Sat, 10 Sep 2011 00:50:23 -0700
Subject: [Ocfs2-users] disable heartbeat nic caused ocfs2 errors

I have a two nodes ocfs2 cluster, and I disabled the heartbeat nic with "ifdown 
eth1". I got following weird logs on both nodes:

Sep  7 10:45:49 dbtest-01 kernel: o2net: connection to node dbtest-02 (num 1) 
at 10.194.59.65: has been idle for 30.0 seconds, shutting it down.
Sep  7 10:45:49 dbtest-01 kernel: (swapper,0,3):o2net_idle_timer:1503 here are 
some times that might help debug the situation: (tmr 1315417519.185025 now 
1315417549.183798 dr 1315417519.185016 adv 1315417519.185032:1315417519.185032 
func (b9bb7168:504) 1315417518.872227:1315417518.872268)
Sep  7 10:45:49 dbtest-01 kernel: o2net: no longer connected to node dbtest-02 
(num 1) at 10.194.59.65:
Sep  7 10:45:49 dbtest-01 kernel: 
(dlm_thread,3781,2):dlm_send_proxy_ast_msg:457 ERROR: status = -112
Sep  7 10:45:49 dbtest-01 kernel: (oracle,26129,1):dlm_do_master_request:1334 
ERROR: link to 1 went down!
Sep  7 10:45:49 dbtest-01 kernel: (oracle,26129,1):dlm_get_lock_resource:917 
ERROR: status = -112
Sep  7 10:45:49 dbtest-01 kernel: 
(dlm_thread,4256,1):dlm_send_proxy_ast_msg:457 ERROR: status = -112
Sep  7 10:45:49 dbtest-01 kernel: (dlm_thread,4256,1):dlm_flush_asts:604 ERROR: 
status = -112
Sep  7 10:45:49 dbtest-01 kernel: (dlm_thread,3781,2):dlm_flush_asts:604 ERROR: 
status = -112
Sep  7 10:46:19 dbtest-01 kernel: (o2net,3736,3):o2net_connect_expired:1664 
ERROR: no connection established with node 1 after 30.0 seconds, giving up and 
returning errors.
Sep  7 10:46:19 dbtest-01 kernel: o2net: accepted connection from node 
dbtest-02 (num 1) at 10.194.59.65:
Sep  7 10:48:37 dbtest-01 kernel: INFO: task events/0:10 blocked for more than 
120 seconds.
Sep  7 10:48:37 dbtest-01 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  7 10:48:37 dbtest-01 kernel: events/0  D 810001004420 010  
111 9 (L-TLB)
Sep  7 10:48:37 dbtest-01 kernel:  81083ffedc80 0046 
80333680 0001
Sep  7 10:48:37 dbtest-01 kernel:  0400 000a 
81083ffe1820 80309b60
Sep  7 10:48:37 dbtest-01 kernel:  0030b62498ce7b3f 416b 
81083ffe1a08 
Sep  7 10:48:37 dbtest-01 kernel: Call Trace:
Sep  7 10:48:37 dbtest-01 kernel: Call Trace:
Sep  7 10:48:37 dbtest-01 kernel:  [] 
wait_for_completion+0x79/0xa2
Sep  7 10:48:37 dbtest-01 kernel:  [] 
default_wake_function+0x0/0xe
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_wait_for_mask+0xd/0x19
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_orphan_scan_work+0x0/0x83
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_orphan_scan_lock+0x55/0x84
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_queue_orphan_scan+0x32/0x147
Sep  7 10:48:37 dbtest-01 kernel:  [] 
:ocfs2:ocfs2_orphan_scan_work+0x1a/0x83
Sep  7 10:48:37 dbtest-01 kernel:  [] run_workqueue+0x94/0xe4
Sep  7 10:48:37 dbtest-01 kernel:  [] worker_thread+0x0/0x122
Sep  7 10:48:37 dbtest-01 ker

Re: [Ocfs2-users] mount type heartbeat=local

[Sunil Mushran]

That's mount type. Yes, we should not have overloaded the term "local".

On 09/09/2011 07:53 PM, Hai Tao wrote:

but this is what I saw in the guide


  OCFS2 - A Cluster File System For Linux
  




*

Mount Type

*

Valid types are

/cluster /and /local/, with the former also being the default. Specify

/
local/if you intend to use the file system on one node only.




Thanks.
Hai Tao



Date: Fri, 9 Sep 2011 19:04:28 -0700
From: sunil.mush...@oracle.com
To: taoh...@hotmail.com
CC: ocfs2-users@oss.oracle.com
Subject: Re: [Ocfs2-users] mount type heartbeat=local

That mount option is appended by mount.ocfs2. It tells users
the heartbeat mode. "none" means non-clustered. "local" means
the heartbeat region is on the mounted volume. This is the default
mode. In 1.8 we have "global" which means the heartbeat region
has been configured on 1+ devices. local and global are only
pertinent in the o2cb stack. If you are using pacemaker, you will
see cluster_stack=pcmk and not heartbeat=.

There is no heartbeat=cluster.

On 09/09/2011 04:43 PM, Hai Tao wrote:


Hi,

when I mount ocfs2 file system, what the heartbeat=local option
means? what is the difference between heartbeat=local and
heartbeat=cluster (I know that is default)?


Thanks.
Hai




___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] mount type heartbeat=local

[Sunil Mushran]

That mount option is appended by mount.ocfs2. It tells users
the heartbeat mode. "none" means non-clustered. "local" means
the heartbeat region is on the mounted volume. This is the default
mode. In 1.8 we have "global" which means the heartbeat region
has been configured on 1+ devices. local and global are only
pertinent in the o2cb stack. If you are using pacemaker, you will
see cluster_stack=pcmk and not heartbeat=.

There is no heartbeat=cluster.

On 09/09/2011 04:43 PM, Hai Tao wrote:


Hi,

when I mount ocfs2 file system, what the heartbeat=local option means? 
what is the difference between heartbeat=local and heartbeat=cluster 
(I know that is default)?



Thanks.
Hai


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Linux kernel crash due to ocfs2

[Sunil Mushran]

http://oss.oracle.com/~smushran/o2image.ppc.dbg

Use the above executable. Hoping it won't fault. But if it does
email me the backtrace. That trace will be readable as the exec
has debugging symbols enabled.

On 09/07/2011 11:24 PM, Betzos Giorgos wrote:
> # rpm -q ocfs2-tools
> ocfs2-tools-1.4.4-1.el5.ppc
>
> On Wed, 2011-09-07 at 09:13 -0700, Sunil Mushran wrote:
>> version of ocfs2-tools?
>>
>> On 09/07/2011 09:10 AM, Betzos Giorgos wrote:
>>> Hello,
>>>
>>> I tried what you suggested but here is what I got:
>>>
>>> # o2image /dev/mapper/mpath0 /files_shared/u02.o2image
>>> *** glibc detected *** o2image: corrupted double-linked list: 0x10045000 ***
>>> === Backtrace: =
>>> /lib/libc.so.6[0xfeb1ab4]
>>> /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
>>> o2image[0x10007bb0]
>>> o2image[0x10002748]
>>> o2image[0x10001f50]
>>> o2image[0x10002334]
>>> o2image[0x100026a0]
>>> o2image[0x10001f50]
>>> o2image[0x10002334]
>>> o2image[0x100026a0]
>>> o2image[0x1000358c]
>>> o2image[0x10003e28]
>>> /lib/libc.so.6[0xfe4dc60]
>>> /lib/libc.so.6[0xfe4dea0]
>>> === Memory map: 
>>> 0010-0012 r-xp 0010 00:00 0  
>>> [vdso]
>>> 0f55-0f56 r-xp  08:13 2881590
>>> /lib/libcom_err.so.2.1
>>> 0f56-0f57 rw-p  08:13 2881590
>>> /lib/libcom_err.so.2.1
>>> 0f90-0f9c r-xp  08:13 2881576
>>> /lib/libglib-2.0.so.0.1200.3
>>> 0f9c-0f9d rw-p 000b 08:13 2881576
>>> /lib/libglib-2.0.so.0.1200.3
>>> 0fa4-0fa5 r-xp  08:13 2881575
>>> /lib/librt-2.5.so
>>> 0fa5-0fa6 r--p  08:13 2881575
>>> /lib/librt-2.5.so
>>> 0fa6-0fa7 rw-p 0001 08:13 2881575
>>> /lib/librt-2.5.so
>>> 0fce-0fd0 r-xp  08:13 2881574
>>> /lib/libpthread-2.5.so
>>> 0fd0-0fd1 r--p 0001 08:13 2881574
>>> /lib/libpthread-2.5.so
>>> 0fd1-0fd2 rw-p 0002 08:13 2881574
>>> /lib/libpthread-2.5.so
>>> 0fe3-0ffa r-xp  08:13 2881571
>>> /lib/libc-2.5.so
>>> 0ffa-0ffb r--p 0016 08:13 2881571
>>> /lib/libc-2.5.so
>>> 0ffb-0ffc rw-p 0017 08:13 2881571
>>> /lib/libc-2.5.so
>>> 0ffc-0ffe r-xp  08:13 2881570
>>> /lib/ld-2.5.so
>>> 0ffe-0fff r--p 0001 08:13 2881570
>>> /lib/ld-2.5.so
>>> 0fff-1000 rw-p 0002 08:13 2881570
>>> /lib/ld-2.5.so
>>> 1000-1002 r-xp  08:13 15058799   
>>> /sbin/o2image
>>> 1002-1003 rw-p 0001 08:13 15058799   
>>> /sbin/o2image
>>> 1003-1006 rwxp 1003 00:00 0  
>>> [heap]
>>> f768-f7ff rw-p f768 00:00 0
>>> ffc6-ffdb rw-p ffc6 00:00 0  
>>> [stack]
>>> Aborted (core dumped)
>>>
>>> I have the core file, if you need it.
>>>
>>> Here is some information about the fs in question.
>>> It is used to store Oracle Archive Logs and also to store the rman backup 
>>> of the DB
>>> In the last crash case the fs became full while rman was running. Maybe we 
>>> can estimate from
>>> this the size of the write in that particular case. Oracle DB rman backup 
>>> files are from 7 to 11Gb.
>>> Maybe Oracle DataGuard was also using on the same fs.
>>> After the crash, when we rebooted the servers, they would crash again. We 
>>> then noticed that
>>> the fs was full and we removed some unneeded files.
>>>
>>> The system has crashed a couple more times when the above conditions may 
>>> not have been the same.
>>>
>>> Thanks,
>>>
>>> George
>>>
>>> 
>>> From: Sunil Mushran
>>> Sent: Friday, September 02, 2011

Re: [Ocfs2-users] Linux kernel crash due to ocfs2

[Sunil Mushran]

version of ocfs2-tools?

On 09/07/2011 09:10 AM, Betzos Giorgos wrote:
> Hello,
>
> I tried what you suggested but here is what I got:
>
> # o2image /dev/mapper/mpath0 /files_shared/u02.o2image
> *** glibc detected *** o2image: corrupted double-linked list: 0x10045000 ***
> === Backtrace: =
> /lib/libc.so.6[0xfeb1ab4]
> /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
> o2image[0x10007bb0]
> o2image[0x10002748]
> o2image[0x10001f50]
> o2image[0x10002334]
> o2image[0x100026a0]
> o2image[0x10001f50]
> o2image[0x10002334]
> o2image[0x100026a0]
> o2image[0x1000358c]
> o2image[0x10003e28]
> /lib/libc.so.6[0xfe4dc60]
> /lib/libc.so.6[0xfe4dea0]
> === Memory map: 
> 0010-0012 r-xp 0010 00:00 0  
> [vdso]
> 0f55-0f56 r-xp  08:13 2881590
> /lib/libcom_err.so.2.1
> 0f56-0f57 rw-p  08:13 2881590
> /lib/libcom_err.so.2.1
> 0f90-0f9c r-xp  08:13 2881576
> /lib/libglib-2.0.so.0.1200.3
> 0f9c-0f9d rw-p 000b 08:13 2881576
> /lib/libglib-2.0.so.0.1200.3
> 0fa4-0fa5 r-xp  08:13 2881575
> /lib/librt-2.5.so
> 0fa5-0fa6 r--p  08:13 2881575
> /lib/librt-2.5.so
> 0fa6-0fa7 rw-p 0001 08:13 2881575
> /lib/librt-2.5.so
> 0fce-0fd0 r-xp  08:13 2881574
> /lib/libpthread-2.5.so
> 0fd0-0fd1 r--p 0001 08:13 2881574
> /lib/libpthread-2.5.so
> 0fd1-0fd2 rw-p 0002 08:13 2881574
> /lib/libpthread-2.5.so
> 0fe3-0ffa r-xp  08:13 2881571
> /lib/libc-2.5.so
> 0ffa-0ffb r--p 0016 08:13 2881571
> /lib/libc-2.5.so
> 0ffb-0ffc rw-p 0017 08:13 2881571
> /lib/libc-2.5.so
> 0ffc-0ffe r-xp  08:13 2881570
> /lib/ld-2.5.so
> 0ffe-0fff r--p 0001 08:13 2881570
> /lib/ld-2.5.so
> 0fff-1000 rw-p 0002 08:13 2881570
> /lib/ld-2.5.so
> 1000-1002 r-xp  08:13 15058799   
> /sbin/o2image
> 1002-1003 rw-p 0001 08:13 15058799   
> /sbin/o2image
> 1003-1006 rwxp 1003 00:00 0  
> [heap]
> f768-f7ff rw-p f768 00:00 0
> ffc6-ffdb rw-p ffc6 00:00 0  
> [stack]
> Aborted (core dumped)
>
> I have the core file, if you need it.
>
> Here is some information about the fs in question.
> It is used to store Oracle Archive Logs and also to store the rman backup of 
> the DB
> In the last crash case the fs became full while rman was running. Maybe we 
> can estimate from
> this the size of the write in that particular case. Oracle DB rman backup 
> files are from 7 to 11Gb.
> Maybe Oracle DataGuard was also using on the same fs.
> After the crash, when we rebooted the servers, they would crash again. We 
> then noticed that
> the fs was full and we removed some unneeded files.
>
> The system has crashed a couple more times when the above conditions may not 
> have been the same.
>
> Thanks,
>
> George
>
> 
> From: Sunil Mushran
> Sent: Friday, September 02, 2011 8:24 PM
> To: Betzos Giorgos
> Cc: ocfs2-users@oss.oracle.com
> Subject: Re: [Ocfs2-users] Linux kernel crash due to ocfs2
>
> Can you provide me with the o2image. It includes the entire fs metadata.
> The size of the image file depends on the number of files/dirs.
>
> # o2image /dev/sdX  /path/to/image/file
>
> So the error is clear. We have underestimated the amount of credits
> (num of blocks that need to be dirtied in that transaction). This is the most
> common write path in the fs and thus hit heavily. So I am surprised by this.
>
> One way to fix it is by reproducing it inhouse. And having the image will 
> allow
> us to mount the fs and reproduce the issue. Do you know the size of the write?
>
> On 09/02/2011 07:23 AM, Betzos Giorgos wrote:
>> Hello,
>>
>> we have a pair of IBM P570 servers running RHEL5.2
>> kernel 2.6.18-92.el5.ppc64
>> We have Oracle RAC on ocfs2 storage
>> ocfs2 is 1.4.7-1 for the above kernel (downloaded from oracle oss site)
>>
>> Recently both servers have been crashing with 

Re: [Ocfs2-users] (mount.ocfs2, 3315, 4):ocfs2_global_read_info:403 ERROR: status = 24

[Sunil Mushran]

kernel/fs.

On 09/06/2011 10:31 PM, Stefan Priebe - Profihost AG wrote:
> could you point me to the code? Is it ocfs2tools code or kernel code. 
> I wasn't able to find it.
>
> Stefan
>
> Am 06.09.2011 22:52, schrieb Sunil Mushran:
>> harmless. the message needs to be silenced.
>>
>> On 09/06/2011 01:31 PM, Stefan Priebe - Profihost AG wrote:
>>> Hi List,
>>>
>>> i've upgraded some machines to linux kernel from 2.8.38 to 3.0.4. Now
>>> i'm always seeing this message when mounting an ocfs2 volume:
>>>
>>> [ 38.745584] (mount.ocfs2,3315,4):ocfs2_global_read_info:403 ERROR:
>>> status = 24
>>> [ 38.776395] (mount.ocfs2,3315,4):ocfs2_global_read_info:403 ERROR:
>>> status = 24
>>>
>>> ocfs2-tools 1.6.3-1
>>>
>>> Stefan
>>>
>>> ___
>>> Ocfs2-users mailing list
>>> Ocfs2-users@oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] (mount.ocfs2, 3315, 4):ocfs2_global_read_info:403 ERROR: status = 24

[Sunil Mushran]

harmless. the message needs to be silenced.

On 09/06/2011 01:31 PM, Stefan Priebe - Profihost AG wrote:
> Hi List,
>
> i've upgraded some machines to linux kernel from 2.8.38 to 3.0.4. Now
> i'm always seeing this message when mounting an ocfs2 volume:
>
> [   38.745584] (mount.ocfs2,3315,4):ocfs2_global_read_info:403 ERROR:
> status = 24
> [   38.776395] (mount.ocfs2,3315,4):ocfs2_global_read_info:403 ERROR:
> status = 24
>
> ocfs2-tools 1.6.3-1
>
> Stefan
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Linux kernel crash due to ocfs2

[Sunil Mushran]

Can you provide me with the o2image. It includes the entire fs metadata.
The size of the image file depends on the number of files/dirs.

# o2image /dev/sdX  /path/to/image/file

So the error is clear. We have underestimated the amount of credits
(num of blocks that need to be dirtied in that transaction). This is the most
common write path in the fs and thus hit heavily. So I am surprised by this.

One way to fix it is by reproducing it inhouse. And having the image will allow
us to mount the fs and reproduce the issue. Do you know the size of the write?

On 09/02/2011 07:23 AM, Betzos Giorgos wrote:
> Hello,
>
> we have a pair of IBM P570 servers running RHEL5.2
> kernel 2.6.18-92.el5.ppc64
> We have Oracle RAC on ocfs2 storage
> ocfs2 is 1.4.7-1 for the above kernel (downloaded from oracle oss site)
>
> Recently both servers have been crashing with the following error:
>
> Assertion failure in journal_dirty_metadata() at
> fs/jbd/transaction.c:1130: "handle->h_buffer_credits>  0"
> kernel BUG in journal_dirty_metadata at fs/jbd/transaction.c:1130!
>
> We get some kind of kernel debug prompt.
>
> the stack is as follows:
>
> .ocfs2_journal_dirty+0x78/0x13c [ocfs2]
> .ocfs2_search_chain+0x131c/0x165c [ocfs2]
> .ocfs2_claim_suballoc_bits+0xadc/0xd94 [ocfs2]
> .__ocfs2_claim_clusters+0x1b0/0x348 [ocfs2]
> .ocf2_do_extend_allocation+0x1f8/0x5b4 [ocfs2]
> .ocfs2_write_cluster_by_desc+0x128/0x850 [ocfs2]
> .ocfs2_write_begin_nolock+0xdc0/0xfbc [ocfs2]
> .ocfs2_write_begin+0x124/0x224 [ocfs2]
> .ocfs2_file_aio_write+0x6a4/0xb40 [ocfs2]
> .aio_pwrite+0x50/0xb4
> .aio_run_iocb+0x140/0x214
> .io_submit_one+0x2fc/0x3a8
> .sys_io_submit+0xd0/0x17c
> syscall_exit+0x0/0x40
>
> In the last crash case, the file system was full.
>
> Any clues?
>
> There seems to be a ocfs2 kernel patch some time ago for the 2.6.20.2
> kernel that fixed some journal credits updates.
>
> Is this another bug?
>
> Any help will be greatly appreciated, because this is a production
> system.
>
> Thanks,
>
> George


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] dlm locking bug?

[Sunil Mushran]

Log what you have in a bz. I can take a look. I doubt you will be able to
attach that file though. You'll need to provide me with a link.

On 09/02/2011 07:28 AM, Sérgio Surkamp wrote:
> Hello,
>
> We have got a problem this morning with our cluster.
>
> Cluster setup:
>
> Servers:
> * Two R800 Dell servers running CentOS 5.5 and ULEK
>2.6.32-100.0.19.el5, with 8G ram each;
> * OCFS2 1.6.4;
> * iSCSI connection using two bonded Gbit nics.
>
> Storage:
> * Dell EqualLogic 4000VX -- iSCSI
>
> Network:
> * Two dell 1Gbit trunked switches;
>
> Problem description:
>
> The node #1 has hanged access to the filesystem and the hung tasks has
> almost the same stack trace as one of the following:
>
> ---
> INFO: task maildirsize:17252 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> maildirsize   D 0004 0 17252  17249 0x0080
>   8800b181d7f8 0086  880098de3c40
>   88001293a1c0 88022bb7c4c0 88001293a598 a036a654
>   8800b181d7e8 81043d10 88001293a1f8 7fff
> Call Trace:
>   [] ? dlmlock+0x12e2/0x13bb [ocfs2_dlm]
>   [] ? update_curr+0xc9/0xd2
>   [] schedule_timeout+0x36/0xe7
>   [] ? need_resched+0x23/0x2d
>   [] wait_for_common+0xb7/0x12c
>   [] ? default_wake_function+0x0/0x19
>   [] ? lockres_clear_flags+0x15/0x17 [ocfs2]
>   [] wait_for_completion+0x1d/0x1f
>   [] ocfs2_wait_for_mask+0x1a/0x29 [ocfs2]
>   [] __ocfs2_cluster_lock+0x83c/0x861 [ocfs2]
>   [] ? ocfs2_inode_cache_io_unlock+0x12/0x14 [ocfs2]
>   [] ? ocfs2_metadata_cache_io_unlock+0x1e/0x20 [ocfs2]
>   [] ? ocfs2_validate_inode_block+0x0/0x1cd [ocfs2]
>   [] ? ocfs2_read_inode_block_full+0x3e/0x5a [ocfs2]
>   [] ocfs2_inode_lock_full_nested+0x194/0xb8d [ocfs2]
>   [] ? ocfs2_rename+0x49e/0x183d [ocfs2]
>   [] ? ocfs2_validate_inode_block+0x0/0x1cd [ocfs2]
>   [] ocfs2_rename+0x49e/0x183d [ocfs2]
>   [] ? brelse+0x13/0x15 [ocfs2]
>   [] ? init_completion+0x1f/0x21 [ocfs2]
>   [] ? ocfs2_init_mask_waiter+0x26/0x3f [ocfs2]
>   [] ? init_completion+0x1f/0x21 [ocfs2]
>   [] ? ocfs2_should_refresh_lock_res+0x8f/0x1ad [ocfs2]
>   [] ? need_resched+0x23/0x2d
>   [] ? kstrdup+0x2b/0xc0
>   [] vfs_rename+0x221/0x3c0
>   [] sys_renameat+0x18b/0x201
>   [] ? autoremove_wake_function+0x0/0x3d
>   [] ? fsnotify_modify+0x6c/0x74
>   [] ? path_put+0x22/0x27
>   [] sys_rename+0x1b/0x1d
>   [] system_call_fastpath+0x16/0x1b
>
> INFO: task imapd:17386 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> imapd D 000b 0 17386   4367 0x0080
>   880208709c08 0086  0286
>   8801501ac800 88012b008180 8801501acbd8 0001bbc4c49f
>      880127bf8d9c
> Call Trace:
>   [] __mutex_lock_common+0x12f/0x1a1
>   [] __mutex_lock_slowpath+0x19/0x1b
>   [] mutex_lock+0x23/0x3a
>   [] do_lookup+0x85/0x162
>   [] __link_path_walk+0x49e/0x5fb
>   [] ? __strncpy_from_user+0x31/0x4a
>   [] path_walk+0x53/0x9c
>   [] do_path_lookup+0x2f/0x7a
>   [] user_path_at+0x57/0x91
>   [] ? handle_mm_fault+0x14b/0x7d9
>   [] vfs_fstatat+0x3a/0x67
>   [] ? ocfs2_inode_unlock+0x140/0x1a5 [ocfs2]
>   [] vfs_stat+0x1b/0x1d
>   [] sys_newstat+0x1f/0x39
>   [] ? do_page_fault+0x25d/0x26c
>   [] ? audit_syscall_entry+0x103/0x12f
>   [] system_call_fastpath+0x16/0x1b
> ---
>
> When we rebooted the node#1, the fallowing recovery messages was
> logged by node#0:
>
> ---
> o2net: connection to node XX (num 1) at ip.ip.ip.2: has
> been idle for 60.0 seconds, shutting it down.
> (swapper,0,0):o2net_idle_timer:1498 here are some times that might help
> debug the situation: (tmr 1314962116.650772 now 1314962176.650058 dr
> 1314962116.650749 adv 1314962116.650781:1314962116.650782 func
> (3f8ab666:504) 1314962114.651682:1314962114.651687)
> o2net: no longer connected to node XX (num 1) at
> ip.ip.ip.2:
> (dlm_thread,4143,2):dlm_send_proxy_ast_msg:456 ERROR: Error -112 when
> sending message 505 (key 0x3f8ab666) to node 1
> (dlm_thread,4143,2):dlm_flush_asts:599 ERROR: status = -112
> (o2net,4052,0):o2net_connect_expired:1659 ERROR: no connection
> established with node 1 after 60.0 seconds, giving up and returning
> errors.
> (o2hb-E4FCE3DA14,4121,1):o2dlm_eviction_cb:267 o2dlm has evicted node 1
> from group E4FCE3DA14E94515B327E8558F4641DA
> (ocfs2rec,13693,10):ocfs2_replay_journal:1605 Recovering node 1 from
> slot 1 on device (8,16)
> (dlm_reco_thread,4144,0):dlm_get_lock_resource:836
> E4FCE3DA14E94515B327E8558F4641DA:$RECOVERY: at least one node (1) to
> recover before lock mastery can begin
> (dlm_r

Re: [Ocfs2-users] Zero allocated blocks for nonempty file

[Sunil Mushran]

Yes. Files and directories under 3800 bytes (or so) are inlined.
The max inline size depends on the features enabled.

On 09/02/2011 05:42 AM, Sérgio Surkamp wrote:
> Hi.
>
> I *suppose* its the inline-data feature, as it's permit the allocation
> of small files and directories inside the inode itself.
>
> > From mkfs.ocfs2 man page:
>
> inline-data
>Enable  inline-data  support.  If  this
> feature  is turned on, OCFS2 will store small files
> and directo- ries  inside  the inode block. Data is
> transparently moved out to an extent when it no
> longer fits inside the inode block. In some cases,
> this can also make a positive impact on  cold-cache
> directory  and  file operations.
>
> Regards,
> Sérgio
>
> Em Fri, 02 Sep 2011 12:36:08 +0200
> Michal Vyoral  escreveu:
>
>> Hello,
>> A question: some non empty files on ocfs2 appears to have zero
>> allocated blocks, see
>>
>> $ stat SXCZ80
>>File: `SXCZ80'
>>Size: 3133Blocks: 0  IO Block: 4096   regular
>> file Device: fe01h/65025dInode: 9484572 Links: 1
>> Access: (0644/-rw-r--r--)  Uid: ( 1001/   webms)   Gid:
>> (   33/www-data) Access: 2011-08-31 11:38:23.272211360 +
>> Modify: 2011-08-29 13:27:17.080403887 +
>> Change: 2011-08-29 13:27:17.080403887 +
>>
>> How is it possible?
>>
>> Environment:
>> OS: Debian 6.0.1
>> Kernel: 2.6.38.5-custom-1
>> Ocfs2-tools: 1.4.4-3
>>
>> Thanks.
>> Michal Vyoral
>>
>>
>>
>> ___
>> Ocfs2-users mailing list
>> Ocfs2-users@oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Max number of files in OCFS2 file system

[Sunil Mushran]

There is no such limit. You are running into a bug that has been fixed
in mainline kernel 2.6.35 and is available with the UEK kernel. Upgrade
to that kernel, install ocfs2-tools 1.6 and enable the discontig-bg feature.

On 08/31/2011 03:15 PM, Omega Xtreme wrote:

Hi All,
Please I would like to know what the maximum number of files (limit) is for an 
OCFS2 file system created on a x86_64 system with default options on a 900GB 
LUN. The system specs are listed below. It was noticed that at some point any 
attempt to create an additional file on the file system hangs the session and 
hangs accessto the file system as well. Attempts to start the application 
(zimbra) installed on the file system has the same effect.
The file system had a zimbra install with the mail files in it. I counted the 
directories and it was over 19,000 (the limit is 31,000 I think) and a count of 
the files came up with over 2.5m.
I suspected it may be something to do with the number of files and after trying everything else, I 
decided to delete some files on the file system - I randomly deleted a small sub-directory 
containing maybe a 200 or so files. Immediately afterwards, the "touch" command worked 
without hanging the file system (usually I have to open another remote session and forcefully 
unmount it) and the response was ok (immediate return to the CLI and "ls" worked without 
any issue)
Just to be sure I copied back the directory I had deleted and the original 
issue started again. Removed the small directory and all was well again.
RedHat 5: 2.6.18-274.el5
OCFS2 tools 1.4.4 and 1.4.7-1
OCFS2 version 1.4.2 and 1.4.4



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/24/2011 09:56 PM, Stefan Priebe - Profihost AG wrote:
>
> ok here is a new complete test with values before and after and all bonnie 
> details.
>
> File creation and seq. delete drops again massively.
>

Version  1.96   --Sequential Output-- --Sequential Input- --Random-
Concurrency   1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server123-han   16G   519  99 113505 22 44504   8  2601  99 97508   6  1241  18 
  1node
server123-han   16G   441  99 113301 23 45351   6  2641  99 102175  6  1158  16 
  2node
server123-han   16G  1715  98 117309  6 47932   4  4656  98 141380  7  796.2 24 
  xfs


Version  1.96   --Sequential Create-- Random Create
server123-han   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
   1024:32768:0/1024  3147  75   437  10  1670  23  3145  7668   1   197   
2 1node
   1024:32768:0/1024   569  39   178   7   866  20   556  3963   3   183   
4 2node
   1024:32768:0/1024  2611  27   873   5  8528  76  2604  2793   0  3875  
51 xfs


Yes, there is a drop in creates. Creates are a bit heavy on cluster locks. There
is scope for improvement at our end.

Read/writes are fairly stable though.

Unfortunately, I've forgotten your original qs.


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

Was on vacation. May take me few days to clean up my mail box.

On 08/29/2011 11:35 PM, Stefan Priebe - Profihost AG wrote:
> Am 22.08.2011 23:27, schrieb Sunil Mushran:
>> On 08/22/2011 11:11 AM, Stefan Priebe - Profihost AG wrote:
>>>>>> BTW, how much memory do the nodes have?
>>>>>> Lock Resources: 1863 (497079)
>>>>>>
>>>>>> This means that that node has created 497K lock resources
>>>>>> but have under 2000 cached. That could be due to deletes
>>>>>> or could be due to lack of memory (lock churning).
>>>>> They all have 8GB of mem but only 1,5-2,5 are in use regarding free
>>>>> command. What does lock churning mean and is there another setting /
>>>>> value i can look at?
>>>>
>>>> Its tied to the inode cache. As long as the inode is cached,
>>>> the lock resource will be cached.
>>> Thanks for your answer but i didn't get it. It is enough "main" memory
>>> free. Or is this to some special kind of memory? perhaps tuneable via
>>> sysctl?
>>>
>>
>> inode cache. I don't think you can pin the size of this cache.
>> It size varies. But we are jumping to conclusions. Right now we
>> don't know whether that is the problem as the numbers you sent
>> me were fairly random.
> Hi Sunil,
>
> had you any chance to look into my newest logfiles?
>
> Stefan


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] bug report: ocfs2 sparc64 panic

[Sunil Mushran]

We appear to be underestimating block credits for quota synching.
OCFS2_QSYNC_CREDITS.

Please file a bugzilla at oss.oracle.com/bugzilla so that we don't forget this.

Possible temporary workarounds include:
1. Incrementing the above #define by a few.
2. Disabling quotas until we have a fix.

Sunil

On 08/30/2011 11:18 AM, Ryan wrote:

Server B is a mirror of Server A, both servers share identical software&  
kernel but different sparc64 cpu..
Using Kernel 3.0.3 on Debian Squeeze w/ocfs2-tools 1.6 from back ports.
Server A shows no problems but Server B crashes regularly with:

[24361.903485] kernel BUG at fs/jbd2/transaction.c:1083!
[24361.969968]   \|/  \|/
[24361.969971]   "@'/ .. \`@"
[24361.969974]   /_| \__/ |_\
[24361.969976]  \__U_/
[24362.163308] kworker/1:3(29218): Kernel bad sw trap 5 [#1]
[24362.234313] TSTATE: 008080001607 TPC: 10292964 TNPC: 
10292968 Y: Not tainted
[24362.363592] TPC:
[24362.442576] g0:  g1: 008a2c00 g2: 0001 
g3: 008d2000
[24362.556987] g4: f8103df5d400 g5: f80001d5e000 g6: f81036e1c000 
g7: 0e80
[24362.671456] o0: 003c o1: 1029c200 o2: 043b 
o3: 0001
[24362.785983] o4: f81036e1f788 o5: 0016 sp: f81036e1efd1 
ret_pc: 1029295c
[24362.905004] RPC:
[24362.984026] l0: f8123af5a000 l1: f8123e3c35e0 l2: f8003f8b1a00 
l3: 0010
[24363.098509] l4: 001704f565f50017 l5: f81229af3018 l6: 004b4262 
l7: 46b4b880
[24363.212977] i0: f8123e3541f8 i1: f81229c18478 i2: 0004 
i3: 0001
[24363.327448] i4:  i5:  i6: f81036e1f081 
i7: 106d499c
[24363.442013] I7:
[24363.510626] Call Trace:
[24363.542676]  [106d499c] ocfs2_journal_dirty+0x48/0x74 [ocfs2]
[24363.627464]  [1071a2bc] ocfs2_modify_bh+0x1a0/0x250 [ocfs2]
[24363.709888]  [1071cc10] ocfs2_local_write_dquot+0xe0/0x194 [ocfs2]
[24363.800344]  [10720ca0] ocfs2_sync_dquot_helper+0x20c/0x2c4 [ocfs2]
[24363.891923]  [0055d95c] dquot_scan_active+0x98/0xf4
[24363.965190]  [1071f6c4] qsync_work_fn+0x18/0x34 [ocfs2]
[24364.043029]  [0047f87c] process_one_work+0x2bc/0x434
[24364.117443]  [0047feb8] worker_thread+0x264/0x454
[24364.188357]  [0048369c] kthread+0x5c/0x70
[24364.250123]  [0042ad78] kernel_thread+0x30/0x48
[24364.318740]  [00483784] kthreadd+0xd4/0x120
[24364.382789] Disabling lock debugging due to kernel taint
[24364.452618] Caller[106d499c]: ocfs2_journal_dirty+0x48/0x74 [ocfs2]
[24364.544243] Caller[1071a2bc]: ocfs2_modify_bh+0x1a0/0x250 [ocfs2]
[24364.633527] Caller[1071cc10]: ocfs2_local_write_dquot+0xe0/0x194 
[ocfs2]
[24364.730878] Caller[10720ca0]: ocfs2_sync_dquot_helper+0x20c/0x2c4 
[ocfs2]
[24364.829320] Caller[0055d95c]: dquot_scan_active+0x98/0xf4
[24364.909499] Caller[1071f6c4]: qsync_work_fn+0x18/0x34 [ocfs2]
[24364.994206] Caller[0047f87c]: process_one_work+0x2bc/0x434
[24365.075491] Caller[0047feb8]: worker_thread+0x264/0x454
[24365.153262] Caller[0048369c]: kthread+0x5c/0x70
[24365.221895] Caller[0042ad78]: kernel_thread+0x30/0x48
[24365.297377] Caller[00483784]: kthreadd+0xd4/0x120
[24365.368286] Instruction DUMP: 11040a70  7c065c67  90122200<91d02005>  
7c0ccdba  92100019  c25c6028  80a04012  226c
[24365.510168] Unable to handle kernel paging request at virtual address 
e000
[24365.614294] tsk->{mm,active_mm}->context = 0c9e
[24365.687485] tsk->{mm,active_mm}->pgd = f81033b5e000
[24365.756111]   \|/  \|/
[24365.756113]   "@'/ .. \`@"
[24365.756116]   /_| \__/ |_\
[24365.756118]  \__U_/
[24365.949404] kworker/1:3(29218): Oops [#2]
[24366.002020] TSTATE: 11e01605 TPC: 00483314 TNPC: 
0047ec78 Y: Tainted: G  D
[24366.142709] TPC:
[24366.193025] g0: f8103b5a2b18 g1:  g2: f80002602280 
g3: 0006
[24366.307407] g4: f8103df5d400 g5: f80001d5e000 g6: f81036e1c000 
g7: f0825c48
[24366.421781] o0: f8103df5d400 o1: f8103df5d400 o2: 0001 
o3: 0059c82c
[24366.536159] o4: 03b1 o5:  sp: f81036e1e951 
ret_pc: 0047ec70
[24366.655108] RPC:
[24366.713433] l0: 008f4800 l1: 0059c82c l2: 0001 
l3: f8123f6c3260
[24366.827815] l4: 007d36a0 l5: f8003f852300 l6:  
l7: 7222
[24366.942188] i0: f8103df5d400 i1: 0001 i2: 0001 
i3: 0001
[24367.056565] i4: f8123f6c3260 i5: 0004 i6: f81036e1ea01 
i7: 00731420
[24367.170945] I7:
[24367.220112] Call Trac

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/22/2011 11:11 AM, Stefan Priebe - Profihost AG wrote:
 BTW, how much memory do the nodes have?
 Lock Resources: 1863 (497079)

 This means that that node has created 497K lock resources
 but have under 2000 cached. That could be due to deletes
 or could be due to lack of memory (lock churning).
>>> They all have 8GB of mem but only 1,5-2,5 are in use regarding free
>>> command. What does lock churning mean and is there another setting /
>>> value i can look at?
>>
>> Its tied to the inode cache. As long as the inode is cached,
>> the lock resource will be cached.
> Thanks for your answer but i didn't get it. It is enough "main" memory 
> free. Or is this to some special kind of memory? perhaps tuneable via 
> sysctl?
>

inode cache. I don't think you can pin the size of this cache.
It size varies. But we are jumping to conclusions. Right now we
don't know whether that is the problem as the numbers you sent
me were fairly random.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/22/2011 09:57 AM, Stefan Priebe - Profihost AG wrote:
>
>> Well the values during and after the test will give better info.
> I will create a dedicated test setup and provide new values and stats.
>
>> BTW, how much memory do the nodes have?
>> Lock Resources: 1863 (497079)
>>
>> This means that that node has created 497K lock resources
>> but have under 2000 cached. That could be due to deletes
>> or could be due to lack of memory (lock churning).
> They all have 8GB of mem but only 1,5-2,5 are in use regarding free 
> command. What does lock churning mean and is there another setting / 
> value i can look at?

Its tied to the inode cache. As long as the inode is cached,
the lock resource will be cached.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Lost ocfs mount point on one node between two shared servers

[Sunil Mushran]

The user's guide explains all that.

On 08/22/2011 09:34 AM, Kalra, Pratima wrote:


It could be possible that it wasn't auto-mounted on reboot. Is there a 
separate setting for that?


*From:*Sunil Mushran [mailto:sunil.mush...@oracle.com]
*Sent:* Monday, August 22, 2011 9:27 AM
*To:* Kalra, Pratima
*Cc:* ocfs2-users@oss.oracle.com
*Subject:* Re: [Ocfs2-users] Lost ocfs mount point on one node between 
two shared servers


On 08/22/2011 09:22 AM, Kalra, Pratima wrote:

Hello All,

Is it normal to lose a shared node in OCFS shared servers? We have 
either lost the whole ocfs mount point or lost mount point on one of 
the shared nodes couple of times. Is this due to some misconfiguration?



Define lose?

Could it be that it was just not auto-mounted at boot.



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Lost ocfs mount point on one node between two shared servers

[Sunil Mushran]

On 08/22/2011 09:22 AM, Kalra, Pratima wrote:


Hello All,

Is it normal to lose a shared node in OCFS shared servers? We have 
either lost the whole ocfs mount point or lost mount point on one of 
the shared nodes couple of times. Is this due to some misconfiguration?




Define lose?

Could it be that it was just not auto-mounted at boot.
___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Please help me to find some answers

[Sunil Mushran]

On 08/22/2011 09:11 AM, Medapuram, Gopala wrote:
> To all OCFS gurus,
>
> We have discussion scheduled on pros and cons of OCFS2 over NFS.
>
> Can you please guide me to some good notes and documentation to prepare for 
> proper discussion?
>
> Appreciate the help.
>
> Thank you,
> Gopal

NFS and OCFS2 are different file systems. Infact one can NFS export
an OCFS2 volume.

Your qs probably relates to a particular use case.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/22/2011 09:08 AM, Stefan Priebe - Profihost AG wrote:
> HI,
>
> here are all values. Just a side note all machines had a fresh reboot. 
> So these values are not right "after" the test.
>
 Network latency:
 cat /sys/kernel/debug/o2net/stats
> 1,3,992696,1284798776,6583557287,129962217930,15822,103963160
> 1,0,991090,1242030681,7180479582,150003421435,74,189353
> 1,2,1002340,1360444027,7088690084,131107733099,28796,112167428
>
 Number of locks, etc:
 cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
>>> /sys/kernel/debug/ is empty - what do i miss?
> Domain: 2E60AF2024744D63848A13417DE66984  Key: 0xcdc07c07  Protocol: 1.1
> Thread Pid: 3669  Node: 1  State: JOINED
> Number of Joins: 1  Joining Node: 255
> Domain Map: 0 1 2 3
> Live Map: 0 1 2 3
> Lock Resources: 1863 (497079)
> MLEs: 0 (510131)
>   Blocking: 0 (13211)
>   Mastery: 0 (496920)
>   Migration: 0 (0)
> Lists: Dirty=Empty  Purge=Empty  PendingASTs=Empty  PendingBASTs=Empty
> Purge Count: 0  Refs: 1
> Dead Node: 255
> Recovery Pid: 3670  Master: 255  State: INACTIVE
> Recovery Map:
> Recovery Node State:
>
>> mount -t debugfs none /sys/kernel/debug

Well the values during and after the test will give better info.

BTW, how much memory do the nodes have?

Lock Resources: 1863 (497079)

This means that that node has created 497K lock resources
but have under 2000 cached. That could be due to deletes
or could be due to lack of memory (lock churning).

The network looks ok. 150usecs ping time between the nodes.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/22/2011 09:01 AM, Stefan Priebe - Profihost AG wrote:
>
>> What features do you have enabled on disk?
>> debugfs.ocfs2 -R "stats" /dev/sdX
>
> The disks were formated using max-features flag. The output looks like 
> this:
>   Revision: 0.90
> Mount Count: 0   Max Mount Count: 20
> State: 0   Errors: 0
> Check Interval: 0   Last Check: Wed Aug  3 12:20:20 2011
> Creator OS: 0
> Feature Compat: 3 backup-super strict-journal-super
> Feature Incompat: 16208 sparse extended-slotmap inline-data 
> metaecc xattr indexed-dirs refcount discontig-bg
> Tunefs Incomplete: 0
> Feature RO compat: 7 unwritten usrquota grpquota
> Root Blknum: 5   System Dir Blknum: 6
> First Cluster Group Blknum: 3
> Block Size Bits: 12   Cluster Size Bits: 12
> Max Node Slots: 8
> Extended Attributes Inline Size: 256
> Label: ocfs2
> UUID: 2E60AF2024744D63848A13417DE66984
> Hash: 2985671674 (0xb1f5bbfa)
> DX Seed[0]: 0xd7cb3bf0
> DX Seed[1]: 0xb630f905
> DX Seed[2]: 0xc126a149
> Cluster stack: classic o2cb
> Inode: 2   Mode: 00   Generation: 2304813421 (0x8960a96d)
> FS Generation: 2304813421 (0x8960a96d)
> CRC32: 8a254ba7   ECC: 0896
> Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
> Dynamic Features: (0x0)
> User: 0 (root)   Group: 0 (root)   Size: 0
> Links: 0   Clusters: 21482248
> ctime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
> atime: 0x0 -- Thu Jan  1 01:00:00 1970
> mtime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
> dtime: 0x0 -- Thu Jan  1 01:00:00 1970
> ctime_nsec: 0x -- 0
> atime_nsec: 0x -- 0
> mtime_nsec: 0x -- 0
> Refcount Block: 0
> Last Extblk: 0   Orphan Slot: 0
> Sub Alloc Slot: Global   Sub Alloc Bit: 65535
>
>> Network latency:
>> cat /sys/kernel/debug/o2net/stats
>> Number of locks, etc:
>> cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
> /sys/kernel/debug/ is empty - what do i miss?
>


mount -t debugfs none /sys/kernel/debug

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow OCFS2 file creation / how to make ocfs2 generally faster?

[Sunil Mushran]

On 08/21/2011 11:36 PM, Stefan Priebe - Profihost AG wrote:
> Hi Guys,
>
> all in all ocfs2 is a nice piece of software and works very well. Last
> week i made some benchmarks and was thinking if there is a way to make
> it faster.
>
> Here are some results perhaps someone can comment them:
>
> XFS iSCSI target:
> 100MB/s block write
>50MB/s block rewrite
> 140MB/s block read
>
> OCFS2 iSCSI target (one node connected / two nodes connected)
>75MB/s block write
>24MB/s block rewrite
>82MB/s block read
>
>
> File Creation / Deletion:
>
> XFS iSCSI target:
> 5800 files/s Seq. create
> 5700 files/s Seq. delete
> 5800 files/s Rand. create
> 1300 files/s Rand. delete
>
> OCFS2 iSCSI target (only ONE node connected):
> 5000 files/s Seq. create
> 5400 files/s Seq. delete
> 5300 files/s Rand. create
>480 files/s Rand. delete
>
> OCFS2 iSCSI target (two or more nodes connected):
>800 files/s Seq. create
> 5800 files/s Seq. delete
>800 files/s Rand. create
>480 files/s Rand. delete
>
> I would be very happy if someone can tell me if this is normal or if
> there are ways to speed this up.
>
> All tests were made using vanilla 2.6.38.8 kernel and these mount
> options noatime,data=writeback,commit=60.

What features do you have enabled on disk?
debugfs.ocfs2 -R "stats" /dev/sdX

Network latency:
cat /sys/kernel/debug/o2net/stats

Number of locks, etc:
cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] IO performance appears slow

[Sunil Mushran]

The cluster stack uses the interconnect to negotiate the locks.
That's how it is able to provide data coherency. Other solutions
do not provide that kind of coherency.

If you are referring to interconnect speeds in ms, it is not good.
That unit is typically used for disk access.

On 08/19/2011 01:30 PM, Nick Geron wrote:


Actually those first numbers were from GigE links going out to physical 
switches and back in.  To optimize the private link, I upgraded the VMs NICs to 
10GE (VMXNet3 which is the VMware para virt driver), moved them onto the same 
host system with a dedicated software switch between them.  The numbers only 
improved slightly, and got worse on 1 of 100 pings (1ms).

10GE between VMs under the same hypervisor: rtt min/avg/max/mdev = 
0.194/0.307/1.003/0.132 ms

What I don't understand is why my OCFS2 cluster suffers so greatly.  There's 
quite a big difference between wall time of 0.17 seconds to traverse the data 
on an iSCSI link and the 4 minutes to do the same on OCFS2 with a sub 1ms 
average latent private interconnect.  For that matter, the whole setup is 
running on another clustered FS (VMFS3) over the same network to the same SAN.  
I guess I'm just a little dumbfounded that OCFS2 is so much more demanding than 
other clustered FSs and alternative network storage options.

Is the network really the most likely candidate?  If so, is anyone else running 
OCFS2 from within a VM environment?  Is this technology only worthwhile in the 
physical world?  Is there a sweet spot for network latency that I should strive 
for?  The user guide only makes mention of 'low latency' but lacks figures save 
for heartbeat and timeouts.

-nick

*From:*Sunil Mushran [mailto:sunil.mush...@oracle.com]
*Sent:* Friday, August 19, 2011 2:30 PM
*To:* Nick Geron
*Cc:* ocfs2-users@oss.oracle.com
*Subject:* Re: [Ocfs2-users] IO performance appears slow

Somewhat equivalent but it misses the effect of the workload at that time.

BTW, those are awful number for 10G NICs. I get better numbers with gige.
rtt min/avg/max/mdev = 0.149/0.168/0.188/0.020 ms

You should check the config, etc. Use ethtool, etc.

On 08/19/2011 10:54 AM, Nick Geron wrote:

Thanks for the feedback Sunil,

You are correct that the sys and user times were very low.  I did check the 
response and latency between the two nodes thinking that could be an issue.   I 
didn't see an issue there, but then again I do not know what they should be.  
Is there a document that outlines the base and/or recommendations for that 
link?  The best I can do in this environment is break my host redundancy and 
move both nodes to the same VMware vSwitch with 10g NICs.

Average latency between the two: rtt min/avg/max/mdev = 0.207/0.268/0.360/0.046 
ms.

Are the ping stats dumped from o2net not equivalent to a simple ping between 
the hosts?  Is my reported latency too great for OCFS2 to function well?

Thanks for your assistance.

-Nick

*From:*Sunil Mushran [mailto:sunil.mush...@oracle.com]
*Sent:* Thursday, August 18, 2011 10:26 PM
*To:* Nick Geron
*Cc:* ocfs2-users@oss.oracle.com <mailto:ocfs2-users@oss.oracle.com>
*Subject:* Re: [Ocfs2-users] IO performance appears slow

The network interconnect between the vms is slow. What
would have helped is the sys and user times. But my guess
is that that is low. Most of it is spent in wall time.

In mainline, o2net dumps stats showing the ping time between
nodes. Unfortunately this kernel is too old.

On 08/18/2011 04:24 PM, Nick Geron wrote:

Greetings,

I'm rather new to OCFS2, so please forgive any glaringly ignorant statements.

I'm evaluating file systems and storage layout for a simple 2 node mail cluster 
using Maildir email directories.  I have created a 2 node cluster with related 
tutorials.  The problem I'm seeing is that general file access using cp, find, 
du, ls, etc. is a significant factor slower on ocfs2 than alternative local and 
remote disk configurations.  I'm hoping someone can clue me into whether this 
behavior is normal, or if I'm missing something in my lab.

*Hosts are identical CentOS 5.5 virtual machines (VMware) with 
2.6.18-238.19.1.el5. (2 ESXi hosts)

*OCFS2 build is ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5 (tools v 1.4.4-1.el5).

*SAN is an EMC Clariion.  LUN is accessed via iSCSI with EMC PowerPath 
5.5.0.00.00-275

*Nodes share a gigabit network for their private interconnect via two 
interconnected switches (ESXi host into each).

*Test data is a 181MB Maildir directory (~12K emails) copied to various types 
of storage.

*Tests involve simple bash scripts running (bash) time with the mentioned 
command line utilities and strace inspection.

The OCFS2 file system was created with the following (mount cannot load xattr 
or extended-slotmap added with max-features):

mkfs.ocfs2 -N 2 -T mail --fs-features=backup-super,sparse,unwritten,inline-data 
-v /dev/emcpowera

Mount options are limited to '_netdev' at t

Re: [Ocfs2-users] IO performance appears slow

[Sunil Mushran]

Somewhat equivalent but it misses the effect of the workload at that time.

BTW, those are awful number for 10G NICs. I get better numbers with gige.
rtt min/avg/max/mdev = 0.149/0.168/0.188/0.020 ms

You should check the config, etc. Use ethtool, etc.

On 08/19/2011 10:54 AM, Nick Geron wrote:


Thanks for the feedback Sunil,

You are correct that the sys and user times were very low.  I did check the 
response and latency between the two nodes thinking that could be an issue.   I 
didn't see an issue there, but then again I do not know what they should be.  
Is there a document that outlines the base and/or recommendations for that 
link?  The best I can do in this environment is break my host redundancy and 
move both nodes to the same VMware vSwitch with 10g NICs.

Average latency between the two: rtt min/avg/max/mdev = 0.207/0.268/0.360/0.046 
ms.

Are the ping stats dumped from o2net not equivalent to a simple ping between 
the hosts?  Is my reported latency too great for OCFS2 to function well?

Thanks for your assistance.

-Nick

*From:*Sunil Mushran [mailto:sunil.mush...@oracle.com]
*Sent:* Thursday, August 18, 2011 10:26 PM
*To:* Nick Geron
*Cc:* ocfs2-users@oss.oracle.com
*Subject:* Re: [Ocfs2-users] IO performance appears slow

The network interconnect between the vms is slow. What
would have helped is the sys and user times. But my guess
is that that is low. Most of it is spent in wall time.

In mainline, o2net dumps stats showing the ping time between
nodes. Unfortunately this kernel is too old.

On 08/18/2011 04:24 PM, Nick Geron wrote:

Greetings,

I'm rather new to OCFS2, so please forgive any glaringly ignorant statements.

I'm evaluating file systems and storage layout for a simple 2 node mail cluster 
using Maildir email directories.  I have created a 2 node cluster with related 
tutorials.  The problem I'm seeing is that general file access using cp, find, 
du, ls, etc. is a significant factor slower on ocfs2 than alternative local and 
remote disk configurations.  I'm hoping someone can clue me into whether this 
behavior is normal, or if I'm missing something in my lab.

*Hosts are identical CentOS 5.5 virtual machines (VMware) with 
2.6.18-238.19.1.el5. (2 ESXi hosts)

*OCFS2 build is ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5 (tools v 1.4.4-1.el5).

*SAN is an EMC Clariion.  LUN is accessed via iSCSI with EMC PowerPath 
5.5.0.00.00-275

*Nodes share a gigabit network for their private interconnect via two 
interconnected switches (ESXi host into each).

*Test data is a 181MB Maildir directory (~12K emails) copied to various types 
of storage.

*Tests involve simple bash scripts running (bash) time with the mentioned 
command line utilities and strace inspection.

The OCFS2 file system was created with the following (mount cannot load xattr 
or extended-slotmap added with max-features):

mkfs.ocfs2 -N 2 -T mail --fs-features=backup-super,sparse,unwritten,inline-data 
-v /dev/emcpowera

Mount options are limited to '_netdev' at the moment.  I've read a bit about 
changing 'data' from ordered to writeback, but that seems to be related to 
waits on flushing cache to disk.  So far, I'm just focusing on reads/lstats.

With a maildir in place, any operation that must inspect all files takes quite 
a while to complete without cached entries.  The alarming thing is the 
discrepancy between my OCFS2 data and identical data on local, NFS and iSCSI 
mounts.

Here's some simple data that should illustrate my problem and my confusion:

Command: 'du --hs /path/to/maildir/on/various/mounts

StorageReal time to complete Min:Sec

--

Local disk 0:0.078

NFS0:2

iSCSI (EXT3)0:1.7

iSCSI (OCFS2) 4:24

Other tests including recursive chowns or chmods, and ls report similar results.

Most telling is perhaps strace output.  There I can see system calls on 
individual Maildir files.  Times between each call/operation take far longer on 
OCFS2 and there is no hint of externally derived waits.  Nor are there any 
indicators of load issues from competing processes; nothing else (significant) 
is going on and du has full reign of the OS resources.

Output from strace with --tt --T using du --hs against the Maildir on my EXT3 
iSCSI LUN (/dev/emcpowerb1)

18:03:17.572879 lstat("1313705228.000737.mbox:2,S", {st_mode=S_IFREG|0644, 
st_size=715, ...}) = 0 <0.18>

18:03:17.572944 lstat("1313705228.008426.mbox:2,S", {st_mode=S_IFREG|0644, 
st_size=2779, ...}) = 0 <0.24>

18:03:17.573016 lstat("1313705228.006345.mbox:2,S", {st_mode=S_IFREG|0644, 
st_size=2703, ...}) = 0 <0.20>

18:03:17.573083 lstat("1313705228.001305.mbox:2,S", {st_mode=S_IFREG|0644, 
st_size=1831, ...}) = 0 <0.17>

Output from the same trace against the OCFS2 store

Re: [Ocfs2-users] IO performance appears slow

[Sunil Mushran]

The network interconnect between the vms is slow. What
would have helped is the sys and user times. But my guess
is that that is low. Most of it is spent in wall time.

In mainline, o2net dumps stats showing the ping time between
nodes. Unfortunately this kernel is too old.

On 08/18/2011 04:24 PM, Nick Geron wrote:


Greetings,

I'm rather new to OCFS2, so please forgive any glaringly ignorant 
statements.


I'm evaluating file systems and storage layout for a simple 2 node 
mail cluster using Maildir email directories.  I have created a 2 node 
cluster with related tutorials.  The problem I'm seeing is that 
general file access using cp, find, du, ls, etc. is a significant 
factor slower on ocfs2 than alternative local and remote disk 
configurations.  I'm hoping someone can clue me into whether this 
behavior is normal, or if I'm missing something in my lab.


*Hosts are identical CentOS 5.5 virtual machines (VMware) with 
2.6.18-238.19.1.el5. (2 ESXi hosts)


*OCFS2 build is ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5 (tools v 
1.4.4-1.el5).


*SAN is an EMC Clariion.  LUN is accessed via iSCSI with EMC PowerPath 
5.5.0.00.00-275


*Nodes share a gigabit network for their private interconnect via two 
interconnected switches (ESXi host into each).


*Test data is a 181MB Maildir directory (~12K emails) copied to 
various types of storage.


*Tests involve simple bash scripts running (bash) time with the 
mentioned command line utilities and strace inspection.


The OCFS2 file system was created with the following (mount cannot 
load xattr or extended-slotmap added with max-features):


mkfs.ocfs2 -N 2 -T mail 
--fs-features=backup-super,sparse,unwritten,inline-data -v /dev/emcpowera


Mount options are limited to '_netdev' at the moment.  I've read a bit 
about changing 'data' from ordered to writeback, but that seems to be 
related to waits on flushing cache to disk.  So far, I'm just focusing 
on reads/lstats.


With a maildir in place, any operation that must inspect all files 
takes quite a while to complete without cached entries.  The alarming 
thing is the discrepancy between my OCFS2 data and identical data on 
local, NFS and iSCSI mounts.


Here's some simple data that should illustrate my problem and my 
confusion:


Command: 'du --hs /path/to/maildir/on/various/mounts

StorageReal time to complete Min:Sec

--

Local disk 0:0.078

NFS0:2

iSCSI (EXT3)0:1.7

iSCSI (OCFS2) 4:24

Other tests including recursive chowns or chmods, and ls report 
similar results.


Most telling is perhaps strace output.  There I can see system calls 
on individual Maildir files.  Times between each call/operation take 
far longer on OCFS2 and there is no hint of externally derived waits.  
Nor are there any indicators of load issues from competing processes; 
nothing else (significant) is going on and du has full reign of the OS 
resources.


Output from strace with --tt --T using du --hs against the Maildir on 
my EXT3 iSCSI LUN (/dev/emcpowerb1)


18:03:17.572879 lstat("1313705228.000737.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=715, ...}) = 0 <0.18>


18:03:17.572944 lstat("1313705228.008426.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2779, ...}) = 0 <0.24>


18:03:17.573016 lstat("1313705228.006345.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2703, ...}) = 0 <0.20>


18:03:17.573083 lstat("1313705228.001305.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=1831, ...}) = 0 <0.17>


Output from the same trace against the OCFS2 store

18:06:52.876713 lstat("1313707554.003441.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2322, ...}) = 0 <0.040896>


18:06:52.917723 lstat("1313707554.003442.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2316, ...}) = 0 <0.040663>


18:06:52.958473 lstat("1313707554.003443.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2899, ...}) = 0 <0.000938>


18:06:52.959471 lstat("1313707554.003444.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=2522, ...}) = 0 <0.001106>


18:06:52.960641 lstat("1313707554.003445.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=3451, ...}) = 0 <0.039904>


18:06:53.000644 lstat("1313707554.003446.mbox:2,S", 
{st_mode=S_IFREG|0644, st_size=3150, ...}) = 0 <0.041060>


Is this normal behavior for a current kernel and the most recent 1.4.7 
code?  Does someone suspect I've blundered somewhere along the way? 
 I've seen many posts to this list related to a mail cluster setup 
like mine.  Is anyone on the list running a production mail cluster 
with OCFS2?  I apologize for the length of this email.  Thanks.


-Nick Geron


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS2 unmount problems after online resize

[Sunil Mushran]

The umount and the hb stop threads are deadlocking on the s_umount lock.

This problem is due to the local heartbeat scheme employed in which the
hb device is the same as the mounted one. umount trigger hb stop which
calls open() => ... => rescan_partitions() => ... => get_super() => down_read().
The same lock should be held by the umount thread.

Unfortunately there is no fix for this other than using a different hb scheme.

Later this year we will be releasing global heartbeat as part of the o2cb stack
that will allow users to specify different hb devices.

Another option is to move to sles11 and make use of the pacemaker cluster
stack.

On 07/25/2011 06:58 AM, Simon Hargrave wrote:

A further update, which simplifies the situation.  It appears to be more 
fundamental, and not actually anything to do with the online resize.
Basically it appears that simply the act of resizing the LUN and performing the 
scsi rescan is enough to make the next unmount fail, i.e.:-
  * create filesystem
  * mount filesystem
  * unmounts and mounts fine
  * extend LUN on storage
  * echo 1 to /sys/block/sdb/device/rescan
  * unmount filesystem, which hangs
The above happens even if only one node is in the cluster, so it doesn't appear 
to be a locking issue between the hosts.
I have tried exactly the same with ext3 (one node obviously!) and the same 
resize doesn't cause a hang.
I have also configure ocfs on a single physical machine (to rule out VMware), 
and the symptoms are identical.
So for whatever reason, the system call to umount() for an ocfs2 filesystem 
hangs if the underlying block device has changed size?
Simon
-
Simon Hargrave szhargr...@ybs.co.uk 
mailto:szhargr...@ybs.co.uk>
Enterprise Systems Team Leader x2831
Yorkshire Building Society 01274 472831
http://wwwtech/sysint/tsgcore.asp http://wwwtech/sysint/tsgcore.asp>

--
*From:* ocfs2-users-boun...@oss.oracle.com 
[mailto:ocfs2-users-boun...@oss.oracle.com] *On Behalf Of *Simon Hargrave
*Sent:* 25 July 2011 13:50
*To:* ocfs2-users@oss.oracle.com
*Subject:* Re: [Ocfs2-users] OCFS2 unmount problems after online resize

Further to this, I get the following in dmesg every 120 seconds after the 
attempted unmount: -
INFO: task ocfs2_hb_ctl:3794 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ocfs2_hb_ctl  D 810003db6420 0  3794   3793 (NOTLB)
 8100b9d05cf8 0086 f20a 8002d0ee
  0007 8100d801e820 80310b60
 00887c712d88 791a 8100d801ea08 80009852
Call Trace:
 [] wake_up_bit+0x11/0x22
 [] __down_read+0x7a/0x92
 [] get_super+0x48/0x95
 [] fsync_bdev+0xe/0x3b
 [] invalidate_partition+0x28/0x40
 [] rescan_partitions+0x37/0x279
 [] do_open+0x231/0x30f
 [] blkdev_open+0x0/0x4f
 [] blkdev_open+0x23/0x4f
 [] __dentry_open+0xd9/0x1dc
 [] do_filp_open+0x2a/0x38
 [] iput+0x4b/0x84
 [] alternate_node_alloc+0x70/0x8c
 [] do_sys_open+0x44/0xbe
 [] tracesys+0xd5/0xe0
-
Simon Hargrave szhargr...@ybs.co.uk 
mailto:szhargr...@ybs.co.uk>
Enterprise Systems Team Leader x2831
Yorkshire Building Society 01274 472831
http://wwwtech/sysint/tsgcore.asp http://wwwtech/sysint/tsgcore.asp>

---

Re: [Ocfs2-users] sudden crash, possibly OCFS was the cause?

[Sunil Mushran]

The log is not complete. It is best to configure netconsole/kdump/etc
to capture the full oops trace.

Having said that, the following patch fits the issue best. Available in
releases after 1.4.7.
http://oss.oracle.com/git/?p=ocfs2-1.4.git;a=commitdiff;h=adbd097b5bdc15c999bc04b16c6fba379cd5d3f2

Yes, ping support to get the update.

On 07/22/2011 02:03 PM, Florin Andrei wrote:

kernel-2.6.18-128.el5
Oracle Linux 5.3
Cluster of two machines, several volumes shared.
Pillar SAN, over fiber channel.

# rpm -qa | grep ocfs
ocfs2console-1.4.4-1.el5
ocfs2-2.6.18-128.el5-1.4.7-1.el5
ocfs2-tools-1.4.4-1.el5

Volumes were not full, system load was low. Everything was quiet.

Looks like an OCFS problem, right? Should I file a bug report with Oracle?

Screenshot of crash screen attached. Last lines in syslog were:


Jul 22 13:09:26  kernel: lockres: N220ab9f03d644bc0, owner=1, 
state=0
Jul 22 13:09:26  kernel:   last used: 18869807601, refcnt: 5, on purge 
list: yes
Jul 22 13:09:26  kernel:   on dirty list: no, on reco list: no, migrating 
pending: no
Jul 22 13:09:26  kernel:   inflight locks: 1, asts reserved: 0
Jul 22 13:09:26  kernel:   refmap nodes: [ 1 ], inflight=1
Jul 22 13:09:26  kernel:   granted queue:
Jul 22 13:09:26  kernel:   converting queue:
Jul 22 13:09:26  kernel:   blocked queue:



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow umounts on SLES10 patchlevel 3 ocfs2

[Sunil Mushran]

Well, half a million on its own does not account for the time. But if one were
to add heavily loaded servers, slower interconnect, high% of shared resources,
the numbers could add up.

I mean, this is a fairly old release. We have made improvements since then.
Having said that, the biggest improvement, parallel migration, is still in our
todo list.

I guess, for now, this is it.

Sunil

On 07/14/2011 05:33 AM, Marc Grimme wrote:
> So I now have two figures from two different clusters. Both are quite slow 
> during restarts. Having two filesystems mounted.
>
> Cluster1 (that one that last time took very long):
> Clusterlocks held by filesystem..
> 1788AD39151A4E76997420D62A778E65: 274258 locks
> 1EFA64C36FD54AB48B734A99E7F45A73: 576842 locks
> Clusterresources held by filesystem..
> 1788AD39151A4E76997420D62A778E65: 214545 resources
> 1EFA64C36FD54AB48B734A99E7F45A73: 469319 resources
>
> Second cluster (also takes quite long):
> Clusterlocks held by filesystem..
> 1EDBCFF0CAB24D0CAE91CB2DA241E8CA: 717186 locks
> 585462C2FA5A428D913A3CBDBC77E116: 68 locks
> Clusterresources held by filesystem..
> 1EDBCFF0CAB24D0CAE91CB2DA241E8CA: 587471 resources
> 585462C2FA5A428D913A3CBDBC77E116: 20 resources
>
>
> Let me know if you need more information.
>
> Thanks
> Marc.
> - "Sunil Mushran"  wrote:
>
>> It was designed to run in prod envs.
>>
>> On 07/07/2011 12:21 AM, Marc Grimme wrote:
>>> Sunil,
>>> can I query those figures during runtime of a productive cluster?
>>> Or might it influence the availability performance what ever?
>>>
>>> Thanks for your help.
>>> Marc.
>>> - "Sunil Mushran"   wrote:
>>>
>>>> umount is a two step process. First the fs frees the inodes. Then
>> the
>>>> o2dlm takes stock of all active resources and migrates ones that
>> are
>>>> still in use. This typically takes some time. But I have never
>> heard
>>>> of it taking 45 mins.
>>>>
>>>> But I guess it could be if one has a lot of resources. Lets start
>> by
>>>> getting a count.
>>>>
>>>> This will dump the number of cluster locks held by the fs.
>>>> # for vol in /sys/kernel/debug/ocfs2/*
>>>>do
>>>>count=$(wc -l ${vol}/locking_state | cut -f1 -d' ');
>>>>echo "$(basename ${vol}): ${count} locks" ;
>>>>done;
>>>>
>>>> This will dump the number of lock resources known to the dlm.
>>>> # for vol in /sys/kernel/debug/o2dlm/*
>>>>do
>>>>count=$(grep -c "^NAME:" ${vol}/locking_state);
>>>>echo "$(basename ${vol}): ${count} resources" ;
>>>>done;
>>>>
>>>> The debugfs needs to be mounted for this to work.
>>>> mount -t debugfs none /sys/kernel/debug
>>>>
>>>> Sunil
>>>>
>>>> On 07/06/2011 08:20 AM, Marc Grimme wrote:
>>>>> Hi,
>>>>> we are using a SLES10 Patchlevel 3 with 12 Nodes hosting tomcat
>>>> application servers.
>>>>> The cluster was running some time (about 200 days) without
>>>> problems.
>>>>> Recently we needed to shutdown the cluster for maintenance and
>>>> experianced very long times for the umount of the filesystem. It
>> took
>>>> something like 45 minutes each node and filesystem (12 x 45
>> minutes
>>>> shutdown time).
>>>>> As a result the planned downtime had to be extended ;-) .
>>>>>
>>>>> Is there any tuning option or the like to make those umounts
>> faster
>>>> or is this something we have to live with?
>>>>> Thanks for your help.
>>>>> If you need more information let me know.
>>>>>
>>>>> Marc.
>>>>>
>>>>> Some info on the configuration:
>>>>> ---X8---
>>>>> # /sbin/modinfo ocfs2
>>>>> filename:
>>>> /lib/modules/2.6.16.60-0.54.5-smp/kernel/fs/ocfs2/ocfs2.ko
>>>>> license:GPL
>>>>> author: Oracle
>>>>> version:1.4.1-1-SLES
>>>>> description:OCFS2 1.4.1-1-SLES Wed Jul 23 18:33:42 UTC 2008
>>>> (build f922955d99ef972235bd0c1fc236c5ddbb368611)
>>>>> srcversion: 986DD1EE4F5ABD8A44FF925
>>>>> depends:ocfs2_dlm,jbd

Re: [Ocfs2-users] reset all ocfs2 data

[Sunil Mushran]

If you've rebooted, then there is not much more to do.

# /sbin/lsmod | grep ocfs2
# egrep "ocfs2|dlm" /proc/slabinfo
After shutting down o2cb, run the above commands. The first one lists
the modules. The second lists the slabs. Both should show no entries.

Did you file a bugzilla for this? If not, please do so on 
oss.oracle.com/bugzilla.
This is the most unique issue I have heard off. If you put all the info in a 
bug,
it will allow us to diagnose it better.

On 07/11/2011 12:28 PM, Richard Pickett wrote:

You guys may remember I'm having difficulty mounting more than 1 device at a 
time.

Is there a way to reset all o2cb/ocfs2 cache/data and "start over" just in case 
there's something weird going on in that regard?

Of course I've rebooted the machines and reformatted the partitions (several 
times by now), but nothing seems to be doing the trick.

Thanks!


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow umounts on SLES10 patchlevel 3 ocfs2

[Sunil Mushran]

It was designed to run in prod envs.

On 07/07/2011 12:21 AM, Marc Grimme wrote:
> Sunil,
> can I query those figures during runtime of a productive cluster?
> Or might it influence the availability performance what ever?
>
> Thanks for your help.
> Marc.
> - "Sunil Mushran"  wrote:
>
>> umount is a two step process. First the fs frees the inodes. Then the
>> o2dlm takes stock of all active resources and migrates ones that are
>> still in use. This typically takes some time. But I have never heard
>> of it taking 45 mins.
>>
>> But I guess it could be if one has a lot of resources. Lets start by
>> getting a count.
>>
>> This will dump the number of cluster locks held by the fs.
>> # for vol in /sys/kernel/debug/ocfs2/*
>>   do
>>   count=$(wc -l ${vol}/locking_state | cut -f1 -d' ');
>>   echo "$(basename ${vol}): ${count} locks" ;
>>   done;
>>
>> This will dump the number of lock resources known to the dlm.
>> # for vol in /sys/kernel/debug/o2dlm/*
>>   do
>>   count=$(grep -c "^NAME:" ${vol}/locking_state);
>>   echo "$(basename ${vol}): ${count} resources" ;
>>   done;
>>
>> The debugfs needs to be mounted for this to work.
>> mount -t debugfs none /sys/kernel/debug
>>
>> Sunil
>>
>> On 07/06/2011 08:20 AM, Marc Grimme wrote:
>>> Hi,
>>> we are using a SLES10 Patchlevel 3 with 12 Nodes hosting tomcat
>> application servers.
>>> The cluster was running some time (about 200 days) without
>> problems.
>>> Recently we needed to shutdown the cluster for maintenance and
>> experianced very long times for the umount of the filesystem. It took
>> something like 45 minutes each node and filesystem (12 x 45 minutes
>> shutdown time).
>>> As a result the planned downtime had to be extended ;-) .
>>>
>>> Is there any tuning option or the like to make those umounts faster
>> or is this something we have to live with?
>>> Thanks for your help.
>>> If you need more information let me know.
>>>
>>> Marc.
>>>
>>> Some info on the configuration:
>>> ---X8---
>>> # /sbin/modinfo ocfs2
>>> filename:
>> /lib/modules/2.6.16.60-0.54.5-smp/kernel/fs/ocfs2/ocfs2.ko
>>> license:GPL
>>> author: Oracle
>>> version:1.4.1-1-SLES
>>> description:OCFS2 1.4.1-1-SLES Wed Jul 23 18:33:42 UTC 2008
>> (build f922955d99ef972235bd0c1fc236c5ddbb368611)
>>> srcversion: 986DD1EE4F5ABD8A44FF925
>>> depends:ocfs2_dlm,jbd,ocfs2_nodemanager
>>> supported:  yes
>>> vermagic:   2.6.16.60-0.54.5-smp SMP gcc-4.1
>>> atix@CAS12:~>   /sbin/modinfo ocfs2_dlm
>>> filename:
>> /lib/modules/2.6.16.60-0.54.5-smp/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko
>>> license:GPL
>>> author: Oracle
>>> version:1.4.1-1-SLES
>>> description:OCFS2 DLM 1.4.1-1-SLES Wed Jul 23 18:33:42 UTC 2008
>> (build f922955d99ef972235bd0c1fc236c5ddbb368611)
>>> srcversion: 16FE87920EA41CA613E6609
>>> depends:ocfs2_nodemanager
>>> supported:  yes
>>> vermagic:   2.6.16.60-0.54.5-smp SMP gcc-4.1
>>> parm:   dlm_purge_interval_ms:int
>>> parm:   dlm_purge_locks_max:int
>>> # rpm -qa ocfs2*
>>> ocfs2-tools-1.4.0-0.9.9
>>> ocfs2console-1.4.0-0.9.9
>>> ---X8---
>>> The kernel version is 2.6.16.60-0.54.5-smp
>>>
>>>
>> __
>>> Marc Grimme
>>>
>>> E-Mail: gri...@atix.de


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Slow umounts on SLES10 patchlevel 3 ocfs2

[Sunil Mushran]

umount is a two step process. First the fs frees the inodes. Then the
o2dlm takes stock of all active resources and migrates ones that are
still in use. This typically takes some time. But I have never heard
of it taking 45 mins.

But I guess it could be if one has a lot of resources. Lets start by
getting a count.

This will dump the number of cluster locks held by the fs.
# for vol in /sys/kernel/debug/ocfs2/*
 do
 count=$(wc -l ${vol}/locking_state | cut -f1 -d' ');
 echo "$(basename ${vol}): ${count} locks" ;
 done;

This will dump the number of lock resources known to the dlm.
# for vol in /sys/kernel/debug/o2dlm/*
 do
 count=$(grep -c "^NAME:" ${vol}/locking_state);
 echo "$(basename ${vol}): ${count} resources" ;
 done;

The debugfs needs to be mounted for this to work.
mount -t debugfs none /sys/kernel/debug

Sunil

On 07/06/2011 08:20 AM, Marc Grimme wrote:
> Hi,
> we are using a SLES10 Patchlevel 3 with 12 Nodes hosting tomcat application 
> servers.
> The cluster was running some time (about 200 days) without problems.
>
> Recently we needed to shutdown the cluster for maintenance and experianced 
> very long times for the umount of the filesystem. It took something like 45 
> minutes each node and filesystem (12 x 45 minutes shutdown time).
> As a result the planned downtime had to be extended ;-) .
>
> Is there any tuning option or the like to make those umounts faster or is 
> this something we have to live with?
>
> Thanks for your help.
> If you need more information let me know.
>
> Marc.
>
> Some info on the configuration:
> ---X8---
> # /sbin/modinfo ocfs2
> filename:   /lib/modules/2.6.16.60-0.54.5-smp/kernel/fs/ocfs2/ocfs2.ko
> license:GPL
> author: Oracle
> version:1.4.1-1-SLES
> description:OCFS2 1.4.1-1-SLES Wed Jul 23 18:33:42 UTC 2008 (build 
> f922955d99ef972235bd0c1fc236c5ddbb368611)
> srcversion: 986DD1EE4F5ABD8A44FF925
> depends:ocfs2_dlm,jbd,ocfs2_nodemanager
> supported:  yes
> vermagic:   2.6.16.60-0.54.5-smp SMP gcc-4.1
> atix@CAS12:~>  /sbin/modinfo ocfs2_dlm
> filename:   
> /lib/modules/2.6.16.60-0.54.5-smp/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko
> license:GPL
> author: Oracle
> version:1.4.1-1-SLES
> description:OCFS2 DLM 1.4.1-1-SLES Wed Jul 23 18:33:42 UTC 2008 (build 
> f922955d99ef972235bd0c1fc236c5ddbb368611)
> srcversion: 16FE87920EA41CA613E6609
> depends:ocfs2_nodemanager
> supported:  yes
> vermagic:   2.6.16.60-0.54.5-smp SMP gcc-4.1
> parm:   dlm_purge_interval_ms:int
> parm:   dlm_purge_locks_max:int
> # rpm -qa ocfs2*
> ocfs2-tools-1.4.0-0.9.9
> ocfs2console-1.4.0-0.9.9
> ---X8---
> The kernel version is 2.6.16.60-0.54.5-smp
>
> __
>
> Marc Grimme
>
> E-Mail: gri...@atix.de

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] inotify

[Sunil Mushran]

This is because we have not hooked up inotify to the cluster stack.

On 06/30/2011 07:26 AM, Jeroen Koekkoek wrote:
> Hi,
>
> I'm running a 2 node OCFS2 + DRBD cluster to host maildirs. The IMAP server 
> (Dovecot) uses inotify to track changes to the maildir, and informs the 
> client when changes occur.
>
> In the following scenario the client is informed immediately:
> The client is connected to host A and a message is delivered on host A.
>
> In the following scenario the client is not informed immediately:
> The client is connected to host A and a message is delivered on host B.
>
> It looks like the inotify updates are more or less batched and transmitted to 
> the other host every n seconds.
>
> My question:
> Is there a way for me to speed up the process of sending inotify updates?
>
> Best regards,
> Jeroen Koekkoek


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] GPF when mounting second device in same cluster

[Sunil Mushran]

Strange. Both udevd and mount thread encountered issue in memory
allocation routine. I would suggest you ping the kernel vendor. This looks
more than just the fs.

On 06/28/2011 04:36 PM, Richard Pickett wrote:

Gents,

OK, back to the single cluster, 2-node, w/ 3 devices. Here's my cluster.conf
cluster:
node_count = 2
name = xport1

node:
ip_port = 
ip_address = 10.8.4.81
number = 1
name = wings
cluster = xport1
node:
ip_port = 
ip_address = 10.8.4.82
number = 2
name = xtc
cluster = xport1

I formatted each device from one node.

I mount the first device on both nodes, all seems well, `mount` reflects the 
newly loaded device.

I mount the second device on the first node (just testing), it returns w/out an 
error, `mount` doesn't show it mounted, and I get the below in the syslog, a 
couple minutes later the box is unresponsive and hast to be powered off/on to 
come back up.

How do I start tracking down the cause of this problem?

Jun 28 23:17:06 wings kernel: [ 1016.037931] o2dlm: Nodes in domain 
4F1AF6E3CA514B8AB64022249255DD26: 1
Jun 28 23:17:06 wings kernel: [ 1016.040713] general protection fault:  
[#1] SMP
Jun 28 23:17:06 wings kernel: [ 1016.040758] last sysfs file: 
/sys/devices/pci:00/:00:11.0/:02:06.0/host4/target4:0:0/4:0:0:0/block/sdc/uevent
Jun 28 23:17:06 wings kernel: [ 1016.040827] CPU 0
Jun 28 23:17:06 wings kernel: [ 1016.040839] Modules linked in: ocfs2 
quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager 
ocfs2_stackglue configfs acpiphp xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 
xt_conntrack nf_conntrack iptable_filter ip_tables x_tables ppdev 
vmware_balloon psmouse serio_raw parport_pc lp intel_agp shpchp i2c_piix4 
parport floppy mptspi mptscsih e1000 mptbase scsi_transport_spi [last unloaded: 
configfs]
Jun 28 23:17:06 wings kernel: [ 1016.041099]
Jun 28 23:17:06 wings kernel: [ 1016.041110] Pid: 542, comm: udevd Not tainted 
2.6.35-28-generic #49-Ubuntu 440BX Desktop Reference Platform/VMware Virtual 
Platform
Jun 28 23:17:06 wings kernel: [ 1016.041184] RIP: 0010:[]  
[] __kmalloc_track_caller+0x84/0x1b0
Jun 28 23:17:06 wings kernel: [ 1016.041244] RSP: 0018:88007c701cc8  
EFLAGS: 00010086
Jun 28 23:17:06 wings kernel: [ 1016.041278] RAX: 880001e128c0 RBX: 
00d0 RCX: 88007f72eb40
Jun 28 23:17:06 wings kernel: [ 1016.041322] RDX:  RSI: 
00d0 RDI: 0004
Jun 28 23:17:06 wings kernel: [ 1016.041367] RBP: 88007c701d08 R08: 
 R09: 0246
Jun 28 23:17:06 wings kernel: [ 1016.041540] R10:  R11: 
0202 R12: 81a24d20
Jun 28 23:17:06 wings kernel: [ 1016.041634] R13: 000f R14: 
e2c3f000ff53f000 R15: 00d0
Jun 28 23:17:06 wings kernel: [ 1016.041743] FS:  7ff946e737a0() 
GS:880001e0() knlGS:
Jun 28 23:17:06 wings kernel: [ 1016.041869] CS:  0010 DS:  ES:  CR0: 
80050033
Jun 28 23:17:06 wings kernel: [ 1016.041943] CR2: 7ff946e7a000 CR3: 
7c6fc000 CR4: 06f0
Jun 28 23:17:06 wings kernel: [ 1016.042046] DR0:  DR1: 
 DR2: 
Jun 28 23:17:06 wings kernel: [ 1016.042149] DR3:  DR6: 
0ff0 DR7: 0400
Jun 28 23:17:06 wings kernel: [ 1016.042232] Process udevd (pid: 542, 
threadinfo 88007c70, task 88007c6dadc0)
Jun 28 23:17:06 wings kernel: [ 1016.042358] Stack:
Jun 28 23:17:06 wings kernel: [ 1016.042409]  88007c701d18 8115f492 
88007c701ce8 
Jun 28 23:17:06 wings kernel: [ 1016.042495] <0> 880066e28160 
000f 00d0 88007a617c30
Jun 28 23:17:06 wings kernel: [ 1016.042623] <0> 88007c701d38 
81118155  880066e280c0
Jun 28 23:17:06 wings kernel: [ 1016.042789] Call Trace:
Jun 28 23:17:06 wings kernel: [ 1016.042851]  [] ? 
vfs_rename+0xc2/0x280
Jun 28 23:17:06 wings kernel: [ 1016.042925]  [] 
kstrdup+0x45/0x80
Jun 28 23:17:06 wings kernel: [ 1016.043006]  [] 
vfs_rename+0xc2/0x280
Jun 28 23:17:06 wings kernel: [ 1016.043091]  [] ? 
__lookup_hash+0x5d/0x160
Jun 28 23:17:06 wings kernel: [ 1016.043179]  [] 
sys_renameat+0x252/0x280
Jun 28 23:17:06 wings kernel: [ 1016.043267]  [] ? 
free_pgtables+0xce/0x120
Jun 28 23:17:06 wings kernel: [ 1016.043357]  [] ? 
cpumask_any_but+0x2b/0x40
Jun 28 23:17:06 wings kernel: [ 1016.043448]  [] ? 
flush_tlb_mm+0x42/0xa0
Jun 28 23:17:06 wings kernel: [ 1016.043536]  [] ? 
remove_vma+0x6e/0x90
Jun 28 23:17:06 wings kernel: [ 1016.043621]  [] ? 
do_munmap+0x309/0x3a0
Jun 28 23:17:06 wings kernel: [ 1016.043709]  [] 
sys_rename+0x1b/0x20
Jun 28 23:17:06 wings kernel: [ 1016.043800]  [] 
system_call_fastpath+0x16/0x1b
Jun 28 23:17:06 wings kernel: [ 1016.043889] Code: 44 00 00 49 89 c1 fa 66 0f 1f 44 
00 00 65 48 8b 14 25 58 eb 00 00 49 8b 04 24 48 8d 04 02 4c 8b 30 4d 85 f6 74 78 49 
63 54 24 18 <49> 8b 14 16 48 89 

Re: [Ocfs2-users] OCFS2 Crash

[Sunil Mushran]

You should ping your kernel vendor. While this does not look ocfs2
related, even if it did, you will be first asked to upgrade to a more
recent kernel, etc. And all those bits will come from the vendor.

On 06/29/2011 02:20 PM, B Leggett wrote:
> Sunril,
> After that first attempt I tried severla more times and got actual oops. I 
> think try #3 has the most details.
>
> Try #2:
>
> Oops:  [#1]
> SMP
> last sysfs file: /firmware/edd/int13_dev80/mbr_signature
> Modules linked in: ocfs2 jbd sg ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager 
> configfs ipv6 iscsi_tcp libiscsi scsi_transport_iscsi xofs button battery ac 
> apparmor aamatch_pcre loop dm_mod netconsole usbhid cpqphp i2c_piix4 ohci_hcd 
> sworks_agp ide_cd cdrom pci_hotplug i2c_core agpgart usbcore tg3 reiserfs edd 
> fan thermal processor cciss serverworks sd_mod scsi_mod ide_disk ide_core
> CPU:0
> EIP:0060:[]Tainted: P X VLI
> EFLAGS: 00210086   (2.6.16.21-0.8-bigsmp #1)
> EIP is at do_page_fault+0x8e/0x5f6
> eax: f3f64000   ebx: c02fbc00   ecx:    edx: 
> esi: f3f6605c   edi: c02971b0   ebp: 0098   esp: f3f64088
> ds: 007b   es: 007b   ss: 0068
>
>
> Try#3
>
> Oops:  [#1]
> SMP
> last sysfs file: /firmware/edd/int13_dev80/mbr_signature
> Modules linked in: ocfs2 jbd sg ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager 
> configfs ipv6 iscsi_tcp libiscsi scsi_transport_iscsi xofs button battery ac 
> apparmor aamatch_pcre loop dm_mod netconsole usbhid i2c_piix4 ide_cd cpqphp 
> cdrom ohci_hcd i2c_core usbcore sworks_agp pci_hotplug agpgart tg3 reiserfs 
> edd fan thermal processor cciss serverworks sd_mod scsi_mod ide_disk ide_core
> CPU:2
> EIP:0060:[]Tainted: P X VLI
> EFLAGS: 00210006   (2.6.16.21-0.8-bigsmp #1)
> EIP is at do_page_fault+0x8e/0x5f6
> eax: f3f2c000   ebx: 880f0133   ecx: 64656e77   edx: 64656e77
> esi: f3f30058   edi: c02971b0   ebp: 64656f0f   esp: f3f2c084
> ds: 007b   es: 007b   ss: 0068
> Unable to handle kernel paging request at virtual address 01110954
>   printing eip:
> c029723e
> *pde = 33dda001
> Unable to handle kernel NULL pointer dereference at virtual address 0030
>   printing eip:
> c015c752
> *pde = 3629c001
> o2net: connection to node node-02 (num 2) at 192.168.1.173: has been idle 
> for 10 seconds, shutting it down.
> (10,0):o2net_idle_timer:1309 here are some times that might help debug the 
> situation: (tmr 1309364991.767445 now 1309365001.767502 dr 1309364996.769068 
> adv 1309364991.767450:1309364991.767451 func (9987e679:2) 
> 1309364870.220076:1309364870.220078)
> o2net: connection to node node-05 (num 4) at 192.168.1.62: has been idle 
> for 10 seconds, shutting it down.
> (10,0):o2net_idle_timer:1309 here are some times that might help debug the 
> situation: (tmr 1309364991.769291 now 1309365001.767537 dr 1309364996.770248 
> adv 1309364991.769302:1309364991.769303 func (3768d12f:505) 
> 1309364991.769291:1309364991.769296)
> Unable to handle kernel paging request at virtual address 4e0b5293
>   printing eip:
> c024c829
> *pde = 36b61001
>
> Try #4
>
> Unable to handle kernel paging request at virtual address fffc
>   printing eip:
> c016e54e
> *pde = 
> Oops:  [#1]
> SMP
> last sysfs file: /firmware/edd/int13_dev80/mbr_signature
> Modules linked in: ocfs2 jbd sg ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager ipv6 
> configfs iscsi_tcp libiscsi scsi_transport_iscsi xofs button battery ac 
> apparmor aamatch_pcre loop dm_mod netconsole usbhid ide_cd cpqphp cdrom 
> i2c_piix4 ohci_hcd sworks_agp i2c_core usbcore agpgart pci_hotplug tg3 
> reiserfs edd fan thermal processor cciss serverworks sd_mod scsi_mod ide_disk 
> ide_core
> CPU:3
> EIP:0060:[]Tainted: P X VLI
> EFLAGS: 00010297   (2.6.16.21-0.8-bigsmp #1)
> EIP is at poll_freewait+0xd/0x3a
> eax: f5ab5f90   ebx: ffe4   ecx: d040   edx: c100
> esi: f31c4000   edi: bffa3bf4   ebp: f34b8310   esp: f5ab5f60
> ds: 007b   es: 007b   ss: 0068
> Process iscsid (pid: 3206, threadinfo=f5ab4000 task=f54521b0)
> Stack:<0>  c016e85a f5ab5fb0 bffa3bf4 bffa3bf4  
> f34b8310
> 0002 0002  f34b8300 c016f12a f31c4000  
> bffa3be4
>  b7f08ff4 f5ab4000 c016e8a8   c0103cab 
> bffa3be4
> Call Trace:
>   [] do_sys_poll+0x2df/0x2e9
>   [] __pollwait+0x0/0x95
>   [] sys_poll+0x44/0x47
>   [] sysenter_past_esp+0x54/0x79
> Code: c4 10 89 d8 5b 5e 5f 5d c3 c7 00 2a f1 16 c0 c7 40 08 00 00 00 00 c7 40 
> 04 00 00 00 00 c3 56 53 8b 70 04 eb 2c 8b 5e 04 83 eb 1c<8b>  43 18 8d 53 04 
> e8 6d 3d fc ff 8b 03 e8 a8 12 ff ff 8d 46 08
>
> - Original Message -----
> From: &q

Re: [Ocfs2-users] OCFS2 Crash

[Sunil Mushran]

1.2.1? That's 5 years old. We've had a few fixes since then. ;)

You have to catch the oops trace to figure out the reason. And one
way to get it by using netconsole. Check the sles10 docs to see how to
configure netconsole. Or, whatever is recommended for capturing the
oops log in that release.

On 06/29/2011 11:28 AM, B Leggett wrote:
> Hi,
> I am running the OCFS2 1.2.1 on SLES 10, just the stuff right out of the box. 
> This is a 3 node cluster that's been running for 2 years with just about zero 
> modification. The storage is a high end SAN and the transport is iscsi. We 
> went two years without an issue and all a sudden node 1 in the cluster keeps 
> crashing. I have never had to troubleshoot OCFS2, so I started with what I 
> could control.
>
> I checked /var/log/messages and nothing there suggests a problem. I replaced 
> hardware that went as far as me popping the scsi drives out and putting them 
> in another server and trying it with all new hardware. The problem still 
> persists.
>
> I had the network team check the iscsi port on the private iscsi network and 
> they are not seeing errors.
>
> I've check the few OCFS2 settings in play and they all look good.
>
> My question to the group is how go I continue troubleshooting this issue? I'm 
> not aware of any native logs etc to reference. I would appreciate any help 
> that gets this diagnosis moving to a solution.
>
> Thanks,
> Bruce


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Heartbeat stays active & stops o2cb shutdown

[Sunil Mushran]

Manually umounting /dlm is not a good idea. Let the o2cb script handle that.
It'll be easier to diagnose if you followed the steps I had listed.

On 06/28/2011 06:41 PM, Shave, Chris wrote:
> Thanks for the info on how to shut the heartbeat down..
> I haven't had a chance to test it as of yet.
> I had previously found info on /dlm being mounted, I unmounted it on 
> both nodes & heartbeat still stayed active.
> Also post a reboot, the process of offlining the cluster did not 
> return an error anymore but a status check still indicated heartbeat 
> was active & an attempt to unload then threw back the heartbeat active 
> error.
>
> *Christopher Shave*, Global UNIX/Linux Projects Team
> *Marsh & McLennan Companies*
> Global Technology Infrastructure (MGTI) | Centralised Operations
> 555 Lonsdale Street, Level 5, Melbourne, VIC 3000, Australia
> +61 3 9623 5488 | Mobile +61 0402 885 057 | _chris.shave@mercer.com_ 
> <mailto:chris.sh...@mercer.com>
> _www.mmc.com_ <http://www.mmc.com/>
> Working Hours:
> Mon-Fri: 8:00am-4:00pm AEST
>
>
> 
> *From:* Sunil Mushran [mailto:sunil.mush...@oracle.com]
> *Sent:* Tuesday, 28 June 2011 3:16 AM
> *To:* Shave, Chris
> *Cc:* ocfs2-users@oss.oracle.com
> *Subject:* Re: [Ocfs2-users] Heartbeat stays active & stops o2cb shutdown
>
> So by default, the hb is supposed to stop on umount.
>
> Do:
> # find /sys/kernel/config/cluster//heartbeat/* -type d | 
> xargs basename
> 77D95EF51C0149D2823674FCC162CF8B
>
> This will list the active heartbeats.
>
> For each hb, do:
> # ocfs2_hb_ctl -I -u 77D95EF51C0149D2823674FCC162CF8B
> 77D95EF51C0149D2823674FCC162CF8B: 1 refs
>
> Notice the references. > 0 is active heartbeat.
>
> If you are sure there are no mounts and "ls /dlm" also has not entries,
> then hb failed to stop for some reason.
>
> To stop, do:
> # ocfs2_hb_ctl -K -u 77D95EF51C0149D2823674FCC162CF8B
>
> It could be that this is failing. What do you see?
>
> I remember we had a problem in this in tools 1.4.1. But that was
> fixed in 1.4.2.
>
> Sunil
>
> On 06/25/2011 06:03 PM, Shave, Chris wrote:
>> Hi,
>> I have an issue with shutting down o2cb & offlining the cluster, the 
>> heartbeat is staying active & is blocking any attempts to shut it 
>> down, despite there being zero ocfs2 filesystems mounted.
>> This is what I see, even happens if using force-offline option:
>> [root]# /etc/init.d/o2cb force-offline clustername
>> Stopping O2CB cluster clustername Failed
>> Unable to stop cluster as heartbeat region still active
>> I have no ocfs2 filesystems curently mounted on either node (2 node 
>> cluster)
>> [root]# mount | grep ocfs
>> [root]#
>> Versions of ocfs2 as below:
>> [root]# rpm -qa | grep ocfs
>> ocfs2-tools-1.4.4-1.el5.x86_64
>> ocfs2-tools-devel-1.4.4-1.el5.x86_64
>> ocfs2console-1.4.4-1.el5.x86_64
>> ocfs2-2.6.18-128.el5-1.4.4-1.el5.x86_64
>> ocfs2-tools-debuginfo-1.4.4-1.el5.x86_64
>> Redhat Linux kernel version: 2.6.18-128.el5
>> A collegue of mine stated that he usually disables all ocfs2 from the 
>> startup scripts, comments out the filesystems in /etc/fstab & 
>> reboots, is there another option to get the heartbeat offline or is 
>> this an ocfs2 or Linux bug I am encountering here??
>> Cheers,
>>
>> *Christopher Shave*, Global UNIX/Linux Projects Team
>> *Marsh & McLennan Companies*
>> Global Technology Infrastructure (MGTI) | Centralised Operations
>> 555 Lonsdale Street, Level 5, Melbourne, VIC 3000, Australia
>> +61 3 9623 5488 | Mobile +61 0402 885 057 | _chris.shave@mercer.com_ 
>> <mailto:chris.sh...@mercer.com>
>> _www.mmc.com_ <http://www.mmc.com/>
>>
>
>
> 
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended 
> recipient, you
> should destroy the e-mail message and any attachments or copies, and 
> you are
> prohibited from retaining, distributing, disclosing or using any 
> information
> contained herein. Please inform us of the erroneous delivery by return
> e-mail.
>
> Thank you for your cooperation.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] how to do rolling upgrade the ocfs2 2 node cluster along with new kernel without application outage.

[Sunil Mushran]

You have to be more specific than that. Maybe best if you ping support.

On 06/28/2011 09:26 AM, veeraa bose wrote:

Hi ALL,

I have to do rolling upgrade on two node ocfs2 cluster, patch the server one by 
one, with out application outage.

I tested in pre-prod 2 node cluster, once the DB is stopped by DBA team, I  ran 
"umount -at ocfs2" and then rebooted the node1, but this also rebooted the 
node2.

could any one please suggest me the best practice for upgrading/rebooting the 
cluster nodes.

I have to upgrade the 3 node cluster in prod ENV with out outage.

Thanks
Veeera.



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] multiple cluster doesn't work

[Sunil Mushran]

While you can do that, it is not recommended.

ocfs2 does not use the label. It is only used by the user to identify a volume.

On 06/28/2011 08:18 AM, Richard Pickett wrote:

That's good news. When I go to format them, do I give them all the same name:

mkfs.ocfs2 -b 4k -C 32K -L "*OCFS2Cluster*" -N 2 /dev/*sdb1*
mkfs.ocfs2 -b 4k -C 32K -L "*OCFS2Cluster*" -N 2 /dev/*sdc1*
mkfs.ocfs2 -b 4k -C 32K -L "*OCFS2Cluster*" -N 2 /dev/*sdd1*

If "no", then how do I do it?

If "yes", then how does o2cb know the difference between each device when they 
are mounted? (how do the nodes distinguish each device when they are communicating to the 
other nodes in the cluster that they are making a change?)


Thanks and God Bless,

Richard W. Pickett, Jr.
www.MyHaitianAdoption.org <http://www.MyHaitianAdoption.org>

P.S. Have you downloaded the journal from my trip to Haiti:

http://www.myhaitianadoption.org/trips/journal-earthquake-rescue-jan-23-feb-2/



2011/6/28 Sérgio Surkamp mailto:ser...@gruposinternet.com.br>>

Hi.

You don't need 3 clusters configured to mount 3 different devices. You
can configure only one cluster and mount as many devices as you want
using it.

bz is the short name for BugZilla, the bug tracker.

Regards,
Sérgio

Em Mon, 27 Jun 2011 23:26:23 -0500
Richard Pickett mailto:richard.pick...@csrtechnologies.com>> escreveu:

> We need 3 clusters concurrently because each one is only 1T each, the
> underlying infrastructure won't allow us to combine the 3 into one
> device without shelling out more money than the company is willing to
> pay.
>
> Where/what is the 'bz' so I can go find the link?
>
>
> Thanks and God Bless,
>
> Richard W. Pickett, Jr.
> www.MyHaitianAdoption.org <http://www.MyHaitianAdoption.org>
>
> P.S. Have you downloaded the journal from my trip to Haiti:
>
> 
http://www.myhaitianadoption.org/trips/journal-earthquake-rescue-jan-23-feb-2/
>
>
>
> On Mon, Jun 27, 2011 at 9:05 PM, Sunil Mushran
> mailto:sunil.mush...@oracle.com>>wrote:
>
> > **
> > Whereas the cluster.conf allows users to define multiple
> > clusters, only one cluster can be active at any time. The bug
> > you ran into has probably been fixed. The link has been posted
> > in the bz.
> >
> > Why do you need multiple clusters active concurrently?
> >
> >
> > On 06/27/2011 04:44 PM, Richard Pickett wrote:
> >
> > We're trying to setup 3 PRDM partitions (VMware) across 2 nodes. As
> > long as only one is configured in cluster.conf, there's not
> > problem. As soon as we try to use 2 or more we get issues.
> >
> > It looks the same as bug 636:
> >
> > http://oss.oracle.com/bugzilla/show_bug.cgi?id=636
> >
> > I posted my cluster.conf and command line results there. I'm
> > including them here in the hopes that someone on this list can
> > offer a shorter solution than recompiling the o2cb_ctl (which we
> > have in process).
> >
> > Thanks in advance for any and all help. Here's my post on the bug:
> >
> > seems like we're back to having this bug again. I'm posting both
> > cluster.conf and the error that happens at the command line.
> >
> > 
> > cluster:
> > node_count = 2
> > name = xport1
> >
> > node:
> > ip_port = 
> > ip_address = 10.8.4.81
> > number = 1
> > name = wings
> > cluster = xport1
> > node:
> > ip_port = 
> > ip_address = 10.8.4.82
> > number = 2
> > name = xtc
> > cluster = xport1
> >
> > cluster:
> >node_count = 2
> >name = xport2
> >
> > node:
> >ip_port = 7778
> >ip_address = 10.8.4.81
> >number = 1
> >name = wings
> >cluster = xport2
> > node:
> >ip_port = 7778
> >ip_address = 10.8.4.82
> >number = 2
> >name = xtc
> >cluster = xport2
> >
> > cluster:
> >node_count = 2
> >name = xport3
> >
> > node:
> >ip_port = 7779
> >ip_address = 10.8.4.81
> >  

Re: [Ocfs2-users] multiple cluster doesn't work

[Sunil Mushran]

On 06/28/2011 08:07 AM, Richard Pickett wrote:
> 1 Terabyte. We have 3 1Terabyte drives. They are already being replicated by 
> the lower-layer architecture, so we don't need to raid them. We'd like to be 
> able to use all three devices at the same time for archive purposes.
>
> I'm surprised to find out ocfs2 only allows access to one of these devices at 
> a time.
>
> What release was your free() fix in?

Not been released yet.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] multiple cluster doesn't work

[Sunil Mushran]

Sorry not following you. What's 1T?

The link has been added to the bug you posted.

On 6/27/2011 9:26 PM, Richard Pickett wrote:
We need 3 clusters concurrently because each one is only 1T each, the 
underlying infrastructure won't allow us to combine the 3 into one 
device without shelling out more money than the company is willing to 
pay.


Where/what is the 'bz' so I can go find the link?


Thanks and God Bless,

Richard W. Pickett, Jr.
www.MyHaitianAdoption.org <http://www.MyHaitianAdoption.org>

P.S. Have you downloaded the journal from my trip to Haiti:

http://www.myhaitianadoption.org/trips/journal-earthquake-rescue-jan-23-feb-2/



On Mon, Jun 27, 2011 at 9:05 PM, Sunil Mushran 
mailto:sunil.mush...@oracle.com>> wrote:


Whereas the cluster.conf allows users to define multiple
clusters, only one cluster can be active at any time. The bug
you ran into has probably been fixed. The link has been posted
in the bz.

Why do you need multiple clusters active concurrently?


On 06/27/2011 04:44 PM, Richard Pickett wrote:

We're trying to setup 3 PRDM partitions (VMware) across 2 nodes.
As long as only one is configured in cluster.conf, there's not
problem. As soon as we try to use 2 or more we get issues.

It looks the same as bug 636:

http://oss.oracle.com/bugzilla/show_bug.cgi?id=636

I posted my cluster.conf and command line results there. I'm
including them here in the hopes that someone on this list can
offer a shorter solution than recompiling the o2cb_ctl (which we
have in process).

Thanks in advance for any and all help. Here's my post on the bug:

seems like we're back to having this bug again. I'm posting both
cluster.conf and the error that happens at the command line.


cluster:
node_count = 2
name = xport1

node:
ip_port = 
ip_address = 10.8.4.81
number = 1
name = wings
cluster = xport1
node:
ip_port = 
ip_address = 10.8.4.82
number = 2
name = xtc
cluster = xport1

cluster:
   node_count = 2
   name = xport2

node:
   ip_port = 7778
   ip_address = 10.8.4.81
   number = 1
   name = wings
   cluster = xport2
node:
   ip_port = 7778
   ip_address = 10.8.4.82
   number = 2
   name = xtc
   cluster = xport2

cluster:
   node_count = 2
   name = xport3

node:
   ip_port = 7779
   ip_address = 10.8.4.81
   number = 1
   name = wings
   cluster = xport3
node:
   ip_port = 7779
   ip_address = 10.8.4.82
   number = 2
   name = xtc
   cluster = xport3


If I comment out the second and third "cluster" I don't hit this
error, but then I can't load multiple clusters on one machine.

Command line:

root@wings:/var/log# /etc/init.d/o2cb start
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Setting cluster stack "o2cb": OK
Starting O2CB cluster xport1: *** glibc detected *** o2cb_ctl:
free(): invalid pointer: 0x01179080 ***
=== Backtrace: =
/lib/libc.so.6(+0x774b6)[0x7fab33f234b6]
/lib/libc.so.6(cfree+0x73)[0x7fab33f29c83]
o2cb_ctl[0x40511a]
o2cb_ctl[0x403a22]
/lib/libc.so.6(__libc_start_main+0xfe)[0x7fab33ecad8e]
o2cb_ctl[0x402b59]
=== Memory map: 
0040-00411000 r-xp  08:01
131365 /sbin/o2cb_ctl
0061-00611000 r--p 0001 08:01
131365 /sbin/o2cb_ctl
00611000-00612000 rw-p 00011000 08:01
131365 /sbin/o2cb_ctl
01178000-01199000 rw-p  00:00
0  [heap]
7fab2c00-7fab2c021000 rw-p  00:00 0
7fab2c021000-7fab3000 ---p  00:00 0
7fab33848000-7fab3385d000 r-xp  08:01
917566 /lib/libgcc_s.so.1
7fab3385d000-7fab33a5c000 ---p 00015000 08:01
917566 /lib/libgcc_s.so.1
7fab33a5c000-7fab33a5d000 r--p 00014000 08:01
917566 /lib/libgcc_s.so.1
7fab33a5d000-7fab33a5e000 rw-p 00015000 08:01
917566 /lib/libgcc_s.so.1
7fab33a5e000-7fab33a76000 r-xp  08:01
921986 /lib/libpthread-2.12.1.so
<http://libpthread-2.12.1.so>
7fab33a76000-7fab33c75000 ---p 00018000 08:01
921986 /lib/libpthread-2.12.1.so
  

Re: [Ocfs2-users] multiple cluster doesn't work

[Sunil Mushran]

Whereas the cluster.conf allows users to define multiple
clusters, only one cluster can be active at any time. The bug
you ran into has probably been fixed. The link has been posted
in the bz.

Why do you need multiple clusters active concurrently?

On 06/27/2011 04:44 PM, Richard Pickett wrote:
We're trying to setup 3 PRDM partitions (VMware) across 2 nodes. As 
long as only one is configured in cluster.conf, there's not problem. 
As soon as we try to use 2 or more we get issues.


It looks the same as bug 636:

http://oss.oracle.com/bugzilla/show_bug.cgi?id=636

I posted my cluster.conf and command line results there. I'm including 
them here in the hopes that someone on this list can offer a shorter 
solution than recompiling the o2cb_ctl (which we have in process).


Thanks in advance for any and all help. Here's my post on the bug:

seems like we're back to having this bug again. I'm posting both 
cluster.conf and the error that happens at the command line.



cluster:
node_count = 2
name = xport1

node:
ip_port = 
ip_address = 10.8.4.81
number = 1
name = wings
cluster = xport1
node:
ip_port = 
ip_address = 10.8.4.82
number = 2
name = xtc
cluster = xport1

cluster:
   node_count = 2
   name = xport2

node:
   ip_port = 7778
   ip_address = 10.8.4.81
   number = 1
   name = wings
   cluster = xport2
node:
   ip_port = 7778
   ip_address = 10.8.4.82
   number = 2
   name = xtc
   cluster = xport2

cluster:
   node_count = 2
   name = xport3

node:
   ip_port = 7779
   ip_address = 10.8.4.81
   number = 1
   name = wings
   cluster = xport3
node:
   ip_port = 7779
   ip_address = 10.8.4.82
   number = 2
   name = xtc
   cluster = xport3


If I comment out the second and third "cluster" I don't hit this 
error, but then I can't load multiple clusters on one machine.


Command line:

root@wings:/var/log# /etc/init.d/o2cb start
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Setting cluster stack "o2cb": OK
Starting O2CB cluster xport1: *** glibc detected *** o2cb_ctl: free(): 
invalid pointer: 0x01179080 ***

=== Backtrace: =
/lib/libc.so.6(+0x774b6)[0x7fab33f234b6]
/lib/libc.so.6(cfree+0x73)[0x7fab33f29c83]
o2cb_ctl[0x40511a]
o2cb_ctl[0x403a22]
/lib/libc.so.6(__libc_start_main+0xfe)[0x7fab33ecad8e]
o2cb_ctl[0x402b59]
=== Memory map: 
0040-00411000 r-xp  08:01 
131365 /sbin/o2cb_ctl
0061-00611000 r--p 0001 08:01 
131365 /sbin/o2cb_ctl
00611000-00612000 rw-p 00011000 08:01 
131365 /sbin/o2cb_ctl
01178000-01199000 rw-p  00:00 
0  [heap]

7fab2c00-7fab2c021000 rw-p  00:00 0
7fab2c021000-7fab3000 ---p  00:00 0
7fab33848000-7fab3385d000 r-xp  08:01 
917566 /lib/libgcc_s.so.1
7fab3385d000-7fab33a5c000 ---p 00015000 08:01 
917566 /lib/libgcc_s.so.1
7fab33a5c000-7fab33a5d000 r--p 00014000 08:01 
917566 /lib/libgcc_s.so.1
7fab33a5d000-7fab33a5e000 rw-p 00015000 08:01 
917566 /lib/libgcc_s.so.1
7fab33a5e000-7fab33a76000 r-xp  08:01 
921986 /lib/libpthread-2.12.1.so 

7fab33a76000-7fab33c75000 ---p 00018000 08:01 
921986 /lib/libpthread-2.12.1.so 

7fab33c75000-7fab33c76000 r--p 00017000 08:01 
921986 /lib/libpthread-2.12.1.so 

7fab33c76000-7fab33c77000 rw-p 00018000 08:01 
921986 /lib/libpthread-2.12.1.so 


7fab33c77000-7fab33c7b000 rw-p  00:00 0
7fab33c7b000-7fab33cab000 r-xp  08:01 
917607 /lib/libpcre.so.3.12.1
7fab33cab000-7fab33eaa000 ---p 0003 08:01 
917607 /lib/libpcre.so.3.12.1
7fab33eaa000-7fab33eab000 r--p 0002f000 08:01 
917607 /lib/libpcre.so.3.12.1
7fab33eab000-7fab33eac000 rw-p 0003 08:01 
917607 /lib/libpcre.so.3.12.1
7fab33eac000-7fab34026000 r-xp  08:01 
921984 /lib/libc-2.12.1.so 
7fab34026000-7fab34225000 ---p 0017a000 08:01 
921984 /lib/libc-2.12.1.so 
7fab34225000-7fab34229000 r--p 00179000 08:01 
921984 /lib/libc-2.12.1.so 
7fab34229000-7fab3422a000 rw-p 0017d000 08:01 
921984 /lib/libc-2.12.1.so 

7fab3422a000-7fab3422f000 rw-p  00:00 0
7fab3422f000-7fab34232000 r-xp 000

Re: [Ocfs2-users] Heartbeat stays active & stops o2cb shutdown

[Sunil Mushran]

So by default, the hb is supposed to stop on umount.

Do:
# find /sys/kernel/config/cluster//heartbeat/* -type d | xargs 
basename
77D95EF51C0149D2823674FCC162CF8B

This will list the active heartbeats.

For each hb, do:
# ocfs2_hb_ctl -I -u 77D95EF51C0149D2823674FCC162CF8B
77D95EF51C0149D2823674FCC162CF8B: 1 refs

Notice the references. > 0 is active heartbeat.

If you are sure there are no mounts and "ls /dlm" also has not entries,
then hb failed to stop for some reason.

To stop, do:
# ocfs2_hb_ctl -K -u 77D95EF51C0149D2823674FCC162CF8B

It could be that this is failing. What do you see?

I remember we had a problem in this in tools 1.4.1. But that was
fixed in 1.4.2.

Sunil

On 06/25/2011 06:03 PM, Shave, Chris wrote:

Hi,
I have an issue with shutting down o2cb & offlining the cluster, the heartbeat is 
staying active & is blocking any attempts to shut it down, despite there being zero 
ocfs2 filesystems mounted.
This is what I see, even happens if using force-offline option:
[root]# /etc/init.d/o2cb force-offline clustername
Stopping O2CB cluster clustername Failed
Unable to stop cluster as heartbeat region still active
I have no ocfs2 filesystems curently mounted on either node (2 node cluster)
[root]# mount | grep ocfs
[root]#
Versions of ocfs2 as below:
[root]# rpm -qa | grep ocfs
ocfs2-tools-1.4.4-1.el5.x86_64
ocfs2-tools-devel-1.4.4-1.el5.x86_64
ocfs2console-1.4.4-1.el5.x86_64
ocfs2-2.6.18-128.el5-1.4.4-1.el5.x86_64
ocfs2-tools-debuginfo-1.4.4-1.el5.x86_64
Redhat Linux kernel version: 2.6.18-128.el5
A collegue of mine stated that he usually disables all ocfs2 from the startup 
scripts, comments out the filesystems in /etc/fstab & reboots, is there another 
option to get the heartbeat offline or is this an ocfs2 or Linux bug I am 
encountering here??
Cheers,

*Christopher Shave*, Global UNIX/Linux Projects Team
*Marsh & McLennan Companies*
Global Technology Infrastructure (MGTI) | Centralised Operations
555 Lonsdale Street, Level 5, Melbourne, VIC 3000, Australia
+61 3 9623 5488 | Mobile +61 0402 885 057 | _chris.shave@mercer.com_ 

_www.mmc.com_ 



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Kernel oops: ocfs2_read_blocks

[Sunil Mushran]

How many nodes?
Does it happen on all the nodes or one in particular?
Are you running the same kernel version on all nodes?
Did this issue start reproducing after some update?
How often does it happen?

Maybe best if you file a bugzilla on oss.oracle.com/bugzilla and
answer the qs there. This could be squeeze specific.

Also, attach the objdump generated as follows;
# objdump -DSl /lib/modules/`uname -r`/kernel/fs/ocfs2/ocfs2.ko >/tmp/ocfs2.out

Ensure it is the same binary that generated the stack below.

Also, cut-paste the following instead of the one you posted. (I have
removed the unnecessary bits to make it more readable.)

===
BUG: unable to handle kernel NULL pointer dereference at 0002
IP: [] ocfs2_read_blocks+0x2e2/0x5ba [ocfs2]
*pdpt = 01 446001 *pde = 
Oops:  [#1] SMP
last sysfs file: 
/sys/devices/pci:00/:00:07.0/:0d:00.0/host3/rport-3:0-7/target3:0:0/3:0:0:10/state
Modules linked in: ipt_REJECT xt_tcpudp iptable_filter ip_tables ocfs2 
quota_tree
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue 
configfs
arpt_mangle arptable_filter arp_tables x_tables bonding dm_round_robin 
dm_multipath
scsi_dh loop radeon ttm snd_pcm drm_kms_helper snd_timer snd drm soundcore
i2c_algo_bit ipmi_si ses i2c_core ipmi_msghandler snd_page_alloc sd_mod psmouse 
hpwdt
pcspkr enclosure hpilo crc_t10dif serio_raw processor container power_meter 
button
evdev ext4 mbcache jbd2 crc16 dm_mod sg usbhid sr_mod hid cdrom ata_generic 
cciss
uhci _hcd thermal qla2xxx scsi_transport_fc ata_piix scsi_tgt ehci_hcd libata 
usbcore
nls_base thermal_sys bnx2 scsi_mod [last unloaded: scsi_wait_scan]

Pid: 32337, comm: ocfs2rec Not tainted (2.6.32-5-686-bigmem #1) ProLiant DL380 
G6
EIP: 0060:[] EFLAGS: 00010202 CPU: 10
EIP is at ocfs2_read_blocks+0x2e2/0x5ba [ocfs2]
EAX: f5647d48 EBX: f3ec31dc ECX:  EDX: 0001
ESI: 0002 EDI:  EBP: 0001 ESP: f40f1e6c
  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process ocfs2rec (pid: 32337, ti=f40f task=f658aec0 task.ti=f40f)
Stack:
0001    f5dc8dd8 f5673000 0001 f3ec31dc
f6b34d20  f3ec3000 fb53937f 0001 f5647d48 0001 
f3ec31dc 0001 f3ec3000  fb50eeab f383be64 f3ec3000 
Call Trace:
[] ? ocfs2_refresh_slot_info+0x80/0xad [ocfs2]
[] ? ocfs2_super_lock+0x1b4/0x27c [ocfs2]
[] ? __ocfs2_recovery_thread+0xc4/0x146d [ocfs2]
[] ? __switch_to+0xcf/0x141
[] ? finish_task_switch+0x34/0x95
[] ? schedule+0x7a4/0x7f1
[] ? __wake_up_common+0x34/0x59
[] ? __ocfs2_recovery_thread+0x0/0x146d [ocfs2]
[] ? kthread+0x61/0x66
[] ? kthread+0x0/0x66
[] ? kernel_thread_helper+0x7/0x10
Code: 01
00 00 68 bd 0e 56 fb e8 d9 00 d8 c5 c7 44 24 3c 01 00 00 00 83 c4 24 eb
11 8b 14 24 89 54 24 18 eb 08 c7 44 24 18 01 00 00 00 <8b> 06 a9 00 00
01 00 74 50 83 7c 24 18 00 0f 84 5b 01 00 00 f6
EIP: [] ocfs2_read_blocks+0x2e2/0x5ba [ocfs2] SS:ESP 0068:f40f1e6c
CR2: 0002
---[ end trace c5a96dd4578cc061 ]---
===


On 06/24/2011 08:48 AM, Stefan Upietz wrote:
> Hello there,
>
> we're experiencing strange behaviour with our ocfs2-enabled systems
> after one node goes down. Right now it is not possible to recreate
> this situation, for it is (alas!) on a critical system...
> This happened on a HP ProLiant DL 360 with an ocfs2 volume on a SAN.
> We're running Debian Squeeze with o2cb_ctl version 1.4.4.
> This is my first post on an oops, so if I missed any information I'd be
> glad if you gave me some hints. Here's the trace:
>
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225294] BUG:
> unable to handle kernel NULL pointer dereference at 0002
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225346] IP:
> [] ocfs2_read_blocks+0x2e2/0x5ba [ocfs2]
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225393] *pdpt =
> 01 446001 *pde = 
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225426] Oops: 
> [#1] SMP
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225453] last sysfs
> file:
> /sys/devices/pci:00/:00:07.0/:0d:00.0/host3/rport-3:0-7/target3:0:0/3:0:0:10/state
> Jun 23 02:27:45 s_local@dante/dante kernel: : [638743.225506] Modules
> linked in:
>ipt_REJECT xt_tcpudp iptable_filter ip_tables ocfs2 quota_tree
> ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue
> configfs arpt_mangle arptable_filter arp_tables x_tables bonding
> dm_round_robin dm_multipath scsi_dh loop radeon ttm snd_pcm
> drm_kms_helper snd_timer snd drm soundcore i2c_algo_bit ipmi_si ses
> i2c_core ipmi_msghandler snd_page_alloc sd_mod psmouse hpwdt pcspkr
> enclosure hpilo crc_t10dif serio_raw processor container power_meter
> button evdev ext4 mbcache jbd2 crc16 dm_mod sg usbhid sr_mod hid cdrom
> ata_generic cciss uhci _hcd thermal qla2xxx scsi_transpor

Re: [Ocfs2-users] ocfs2 with cman luster stack

[Sunil Mushran]

So this is ubuntu 11.04. The qs is is anyone using that distro/version
gotten this to work. If not, then one possibility is a build issue. Maybe
file a bug with ubuntu to see if they have tested it with their binaries.

On 06/23/2011 01:55 PM, charles wrote:

hello,

i opened a bug on the ocfs2 bugzilla reporting difficulty i have utilize a  
different cluster stack than o2cb.
link: http://oss.oracle.com/bugzilla/show_bug.cgi?id=1325

to summarize i'd like to use ocfs2 on an ubuntu server with drbd and ctdb. ctdb 
requires fcntl locks for its lock files. however o2cb doesn't provide that 
functionality whereas cman (redhat's cluster manager does).

but i'm unable to mount ocfs2 file systems even when cman is configured and 
running with appropriate cluster name.

in the bug ticket it was suggested that i use the update stack option for 
tune2fs.ocfs2 but that didn't work either.

does anyone have any experience using the cman stack with ocfs2?

thanks

--
Charles
Belmopan, Belize

*clubmotorsports.bz* 
*
*
***club motorsports belize facebook* 
**
*
*
**http://www.cardomain.com/ride/2400106**
"... we just love cars and we love driving them!"
"... Do things like double clutch automatics, traction control, and lane departure 
warnings really make cars better? I suppose by some metrics they do. But for these 
amenities we trade character; for luxury we surrender sensory experience; for comfort, we 
give up romance."

*
*


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] number of nodes is LUN dependent or cluster group dependent ?

[Sunil Mushran]

On 6/21/2011 9:41 PM, Thomas Lau wrote:
> As title, because LUN A on cluster group 1 is using N=8, if I have new
> LUN B, can I set number of node to something bigger and join same
> cluster group 1?

Yes.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Unable to umount a filesystem - OCFS still thinks it has it mounted?‏

[Sunil Mushran]

Check /proc/mounts. That's the kernels view of the mounts.

mount looks at /etc/mtab. And ocfs2 1.2 adds and removes
entries /proc/fs/ocfs2 during mount/umount.

Also, see if there are relevant errors in dmesg.

On 06/16/2011 07:16 PM, Neil Campbell wrote:


Hi all,

Not sure what has happened but I am trying to remove a directory that 
I thought was copied from an OCFS filesystem.


When I try and remove it I get the typical Device or resource busy error.

This is on RHEL 4 update 8 kernel 2.6.9-89.0.26.ELsmp   running OCFS2 
version OCFS2 1.2.9 Mon Jan 17 07:38:31 EST 2011 (build 
5e8325ec7f66b5189c65c7a8710fe8cb)


Running lsof (as root) on the directory returns nothing.

If I do a mount | grep ocfs I get the following (which is what I expect)

 # mount | grep ocfs
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/emcpowerq1 on /d00/cfs type ocfs2 (rw,_netdev,heartbeat=local)
/dev/emcpowere1 on /d00/html type ocfs2 (rw,_netdev,heartbeat=local)


However, under cat /proc/fs/ocfs2 there are 3 device entries

ls -l /proc/fs/ocfs2/

dr-xr-xr-x  2 root root 0 Jun  9 15:51 120_257
dr-xr-xr-x  2 root root 0 Jun  9 15:51 120_65
dr-xr-xr-x  2 root root 0 Jun  9 15:51 120_66
-r--r--r--  1 root root 0 Jun  9 15:51 nodename
-r--r--r--  1 root root 0 Jun  9 15:51 version


But on the other 3 servers in this cluster there are only 2

If I look at /dev for those devices they are as follows

[root@dc-c02b01-app01 log]# ls -l /dev | grep 120 | egrep '257|65|66'
brw---  1 root root 120,  65 Feb  2 08:16 emcpowere1
brw---  1 root root 120,  66 Feb  2 08:16 emcpowere2
brw---  1 root root 120, 257 Feb  2 08:16 emcpowerq1


emcpowere2 should be umounted and doesn't show up when running mount, 
but it seems OCFS still thinks it is mounted.


So what can I do to release this ?

Many thanks
Neil



___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Any suggestions how to copy between two OCFS2 volumes faster?

[Sunil Mushran]

Try "dd bs=1M iflag=direct" on few files.

See if that helps.

On 06/15/2011 01:00 PM, Ulf Zimmermann wrote:
> I need to copy a number of volumes from one SAN to another SAN. Most of our 
> volumes are snapclone based, so moving those has been easy. But we got 
> several 700GB volumes, which I can't move that way. Doing an umount/mount 
> local (no network lock?) doesn't seem help, speed is only about 20MB/sec 
> using rsync. Any suggestions how to maybe copy faster? These are all Oracle 
> RAC 10g database files.
>
> Ulf.
>
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] ocfs2 slow write performance on Linux 2.6.38

[Sunil Mushran]

What type of writes are these... sequential or random?

On 6/12/2011 5:37 PM, fibrer...@gmail.com wrote:
> Hello all,
>
> I am benchmarking OCFS2 in a single node environment to see how its
> performance stacks up against other Linux file systems. My hardware is
> dual CPU, 6-cores per CPU, 2.4GHz Westmere with 24GB RAM. My system
> has 24 HDD's of which 23 are active drives in a md RAID 5 array with
> 64K chunk size, and one hot-spare. I have a LVM2 volume created out of
> this md array. It is running Linux 2.6.38 kernel.
>
> I am creating the OCFS2 file system as follows:
>
> /sbin/mkfs.ocfs2 -N 2 -M local
> --fs-features=sparse,refcount,xattr,indexed-dirs,discontig-bg,inline-data
> -F /dev/md0/ocfs2max
>
> I am then mounting it as:
>
> /bin/mount /volumes/md0/ocfs2max
>
> I am using fio 1.55 to measure performance. My benchmark script uses
> uncached IO in 4K and 4M block sizes. I am running 8 fio jobs and 512
> IO depth. I am using a 20GB file as the test file for this benchmark.
>
> OCFS2 reads (both at 4M and 4K) are very good...within +-3% of the
> other major Linux file systems. However, 4K writes are substantially
> lower, about half the speed of other Linux FS's.
>
> Can you provide some recommendations on how I can improve the write
> speed with OCFS2? At the current write rate, the trade-off is too
> much, but I really like the other features of OCFS2 so I am hopeful
> that some tuning/tweaking will get the performance up where it needs
> to be.
>
> Thank you very much in advance!
>
> - TG
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Errors about a hole in an inode, not fixed by fsck.ocfs2

[Sunil Mushran]

On 06/08/2011 03:36 PM, Herman wrote:
> Hi all,
>
> Using:  RHEL 6 / DRBD 8.3.10-2 kmod from ElRepo / OCFS2 compiled from
> Redhat's kernel source 2.6.32-71.18.2.el6.x86_64
>
> I have a system running DRBD with OCFS2.  The OCFS2 filesystem is not
> being used for databases.  I had a split-brain due to
> RHEL6's /etc/init.d/iptables script.  Immediately after the split-brain
> the OCFS2 filesystem started giving errors and they came in pretty fast.
> I recovered from the split-brain manually, but that didn't stop the
> messages.  Rebooting, switching primary nodes, fsck.ocfs2, all didn't
> work... the messages still came in fast.  I finally had to find out what
> that inode was pointing to and remove it before the messages stopped.
>
> > From looking through the archives, I guess it's due to there being a
> hole in the inode from the sparse filesystem?  But I don't know why it's
> complaining about it, and I don't know why fsck.ocfs2 couldn't fix it.
>
> Here's the errors I got:
> Apr 18 07:53:07 server-2 kernel: (8163,0):ocfs2_read_virt_blocks:853
> ERROR: Inode #5377026 contains a hole at offset 466944
> Apr 18 07:53:07 server-2 kernel: (8163,0):ocfs2_read_dir_block:533
> ERROR: status = -5
> Apr 18 07:53:08 server-2 kernel: (8163,12):ocfs2_read_virt_blocks:853
> ERROR: Inode #5377026 contains a hole at offset 466944
> Apr 18 07:53:08 server-2 kernel: (8163,12):ocfs2_read_dir_block:533
> ERROR: status = -5
> Apr 18 07:53:08 server-2 kernel: (8508,0):ocfs2_read_virt_blocks:853
> ERROR: Inode #5377026 contains a hole at offset 466944
> Apr 18 07:53:08 server-2 kernel: (8508,0):

The directory cannot have holes. That's the error. I am assuming
that triggered due to the network/drbd. If so, atleast one should
be ok.

fsck should detect the problem. Don't know offhand whether we
have a fix code for it or not. First time seeing this issue.

File a bug on oss.oracle.com/bugzilla.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] ocfs2 writing files bigger than 4MB

[Sunil Mushran]

Upgrade to a more recent kernel (2.6.35+).
Upgrade ocfs2-tools to 1.6.x.
Run tunefs.ocfs2 and enable feature discontig-bg.

This will address this issue.

On 06/08/2011 02:16 PM, Osvaldo Alvarez Pozo wrote:
> Hi
> I can not copy files bigger than 4Mbytes!
> I have an ocfs2 cluster with 4 nodes using debian lenny the version of
> ocfs2 is 1.4.1 the partition is use up to 70%
>
>
> this is the output of df -h
>
> FilesystemSize  Used Avail Use% Mounted on
>
> /dev/mapper/vmail
>   120G   84G   37G  70% /var/vmail
>
> and this is the output of df -hi
> df -hi
> FilesystemInodes   IUsed   IFree IUse% Mounted on
>
> /dev/mapper/vmail
> 30M 21M9.2M   70% /var/vmail
>
> Any idea please??
>
> Thanks
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] cannot write to filesystem, permission denied?

[Sunil Mushran]

On 06/07/2011 05:01 AM, Sven Karlsson wrote:
> Hello,
>
> We have installed Fedora 15 to get the latest ocfs2 release nicely
> packaged in a 2.6.38 kernel and ocfs2-tools 1.6.3.
> Setup went fine, mkfs.ocfs2 went fine, a cluster was created and the
> local node added:
>
> # o2cb_ctl -C -i -n mycluster -t cluster
> Cluster mycluster created
> # o2cb_ctl -C -i -n lab -t node -a number=1 -a
> ip_address=192.168.111.5 -a ip_port=7778 -a cluster=mycluster
>
>
> O2CB is configured and starts up as it should and reports that
> mycluster is active, the O2CB heartbeat is active and the filesystem
> mounts OK.
>
> But: writes to the filesystem all fail with 'permission denied'. Example:
>
> # date>  /o2/file.txt
> -bash: /o2/file.txt: Permission denied
> # mkdir /o2/dir
> mkdir: cannot create directory `/o2/dir': Permission denied
> # id
> uid=0(root) gid=0(root)
> groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> # mount | grep o2
> /dev/mapper/vg01-lv_mycluster on /o2 type ocfs2
> (rw,relatime,_netdev,heartbeat=local,nointr,data=ordered,errors=remount-ro,coherency=full,user_xattr,acl)
>
> According to strace, the permission denied is from
>
> mkdir("/o2/dir", 0777)  = -1 EACCES (Permission denied)
>
>
>
> Do you know what is the cause of this apparent "read-only" mode?
> (although it is reported as a mounted rw)
>
> Any pointers for further troubleshooting is greatly appreciated!

You have acl enabled. Maybe that is what is triggering EACCES.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Problems with descriptions.

[Sunil Mushran]

  
  
That's the number of files open on the system. So this looks like
an app problem. Some app has many files open.

On 06/01/2011 10:37 PM, Vasyl S. Kostroma wrote:

  

  Hi guys!
  I can’t find an answer in google, so my last hope is this
mailing list.
   
  Story.
  I have two servers with same arrays. Servers connected by
DRBD.
  I used ocfs2 as file system, also I used NFS4 to access
to the 
  ocfs2 drive. I do not have any idea, but the allocated
descriptors in
  /proc/sys/fs/file-nr increasing every time while drive
accessed.
  So after some time allocated descriptions over max value
and all
  processes make error “To much opened files” (something
like
  this). I do not see any error messages in log files...
Any idea?
  I haven’t sleep two days 
   
  Thank you all in advance.
   
  Configs:
  cat /etc/drbd.conf
  # You can find an example in 
  /usr/share/doc/drbd.../drbd.conf.example
   
  include "drbd.d/global_common.conf";
  include "drbd.d/*.res";
   
   
  resource nfs {
   
      protocol C;
   
      handlers {
      split-brain
  "/usr/lib/drbd/notify-split-brain.sh root";
      pri-on-incon-degr "echo o >
  /proc/sysrq-trigger ; halt -f";
      pri-lost-after-sb "echo o >
  /proc/sysrq-trigger ; halt -f";
      local-io-error "echo o >
  /proc/sysrq-trigger ; halt -f";
      }
   
      startup {
      become-primary-on both;
      degr-wfc-timeout 120;
      }
   
      disk {
      on-io-error detach;
      }
   
      net {
      cram-hmac-alg sha1;
      shared-secret "password";
  

    allow-two-primaries;
      after-sb-0pri discard-zero-changes;
      after-sb-1pri discard-secondary;
      after-sb-2pri disconnect;
      rr-conflict disconnect;
      }
   
      syncer {
      rate 500M;
      verify-alg sha1;
      al-extents 257;
      }
   
      on st01 {
      device  /dev/drbd0;
      disk    /dev/sdb;
      address 192.168.3.151:7788;
      meta-disk internal;
      }
   
      on st02 {
      device  /dev/drbd0;
      disk    /dev/sdb;
      address 192.168.3.152:7788;
      meta-disk internal;
      }
  }
   
  ---
  cat /etc/ocfs2/cluster.conf
  #/etc/ocfs2/cluster.conf
  node:
      ip_port = 
      ip_address = 192.168.1.151
      number = 1
      name = st01
      cluster = ocfs2
   
  node:
      ip_port = 
      ip_address = 192.168.1.152
      number = 2
      name = st02
      cluster = ocfs2
   
  cluster:
      node_count = 2
      name = ocfs2
  ---
  cat /etc/exports
  # /etc/exports: the access control list for
  filesystems which may be exported
  #   to NFS clients.  See exports(5).
  /snfs  
  192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=0)
  /snfs/projects 
  192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)
  /snfs/configs  
  192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)
  /snfs/variables
  192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)
  /snfs/backups  
  192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)
  ---

  
  

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


  

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] mkfs.ocfs2 optimal options for web server

[Sunil Mushran]

On 06/01/2011 02:03 AM, Alex Sobrino wrote:
> We're planning a three web server cluster based on OCFS2. Basically, it
> will handle a huge CMS, with lots of PHP code, and some file uploads
> (but mainly file reads).
>
> Initially, I was thinking in:
>
> - Block size 4K
> - Cluster size 4K
> - Node slots 3
> - FS type mail
> - Journal options - let mkfs.ocfs2 guess them
>
> Do you think that I should modify anything? Do any of you have any
> similar configuration?

It makes sense to add additional slots upfront to allow you to grow later.
It will be easier if you have slots available. So go with 4 atleast if not 6.
The only downside is that it will reserve more journal space. But it's space
you can get it back later.

Also enable most features. The list depends on the version of the tools/fs you
have. If you are on a kernel that is based off 2.6.32+, then you should
be able to enable most features. For more, refer to man mkfs.ocfs2.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Large Files Hang Server

[Sunil Mushran]

If your apps do not care about atime, then noatime is helpful.
data=writeback should performs better than data=ordered. But
there is a small chance that files having trailing nulls if a node were
to reboot after a journal commit but before a data flush. This is
documented in the manpages and the user's guide. I don't believe
nodiratime is a standard mount op. May be specific to a fs. noatime
should be enough.

So I would use noatime. Journal mode will depend on the app/use case.

On 05/24/2011 11:42 PM, Stefan Priebe - Profihost AG wrote:
> @Sunil
> So could you recommand generally mount options to set? At the moment 
> i've set none.
>
> So is it always a good idea (don't need atime) to set 
> data=writeback,noatime,nodiratime any other options?
>
> Stefan
>
>> Did you set the mount option on both nodes or only on the node
>> on which you were doing the ls?
>>
>> Setting it on both nodes, or on the node that  is doing the cp should
>> solve the perf issue. What's happening is that the ls on node2 is 
>> forcing
>> node1 to journal commit. With the ordered data journal mode, the data
>> is flushed on commit. Switching to writeback will allow it to commit
>> without flushing the data.


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Large Files Hang Server

[Sunil Mushran]

Did you set the mount option on both nodes or only on the node
on which you were doing the ls?

Setting it on both nodes, or on the node that  is doing the cp should
solve the perf issue. What's happening is that the ls on node2 is forcing
node1 to journal commit. With the ordered data journal mode, the data
is flushed on commit. Switching to writeback will allow it to commit
without flushing the data.

On 05/24/2011 01:58 PM, Keith W wrote:
> Yes, I am finding that if I do the large file copy on node1 and
> do an ls -l on node1 it is very fast as expected.
>
> If I do the large file copy on node1 and do an ls -l on node2
> ls -l is showing multi second times. 5+ seconds at least.
>
> If I do a file listing on any other file it is fast regardless
> of which node I am on so long as I don't specify the file in transfer.
>
> Only the file in transfer will hand ls -l when not on the node doing
> the transfer.
>
> I am starting to think this is expected behaviour. Am I correct?
>
> +---+
> +   Keith +
> +-------+
>
> On Tue, 24 May 2011, Sunil Mushran wrote:
>
>> Writeback will help if the writes are on one node and the ls on another.
>> It is not clear if that is the case or not.
>>
>> If both ops are on the same node, then it just could be the disk is slow.
>> The times shows almost all wall time. Very little sys and no user. top
>> will show io wait times.
>>
>> On 05/24/2011 11:45 AM, Keith W wrote:
>>> No change in behavior.
>>> My mount options
>>> /dev/sdj1   /u03ocfs2   _netdev,noatime,data=writeback,nointr   0 0
>>>
>>> +---+
>>> + Keith +
>>> +---+
>>>
>>> On Tue, 24 May 2011, Sunil Mushran wrote:
>>>
>>>> Repeat the same test but with volumes mounted with data=writeback
>>>> mount option.
>>>>
>>>> mount -o data=writeback /dev/sdX /path
>>>>
>>>> On 05/24/2011 07:11 AM, Keith W wrote:
>>>>> Hello list.
>>>>> Apologies in advance, this may be a bit long. Just trying to give
>>>>> as much info as I can at the outset.
>>>>>
>>>>> I have a two node setup that share a 500Gig SAS drive via ocfs2.
>>>>> When I move either large files 300Megs+ or a large number of smaller files
>>>>> onto or off of the volume, my terminal session will hang and if I do a
>>>>> directory listing in another terminal while doing a file transfer that
>>>>> terminal will hang as well.
>>>>>
>>>>> The only thing I can see that is not "typical" is that I had to change
>>>>> the port to  due to another application running on .
>>>>>
>>>>>
>>>>> Here is my configuration:
>>>>> 
>>>>> Oracle Enterprise Linux 5.5 (Oracle Updated Kernel 2.6.18-194.0.0.0.3.el5)
>>>>> OCFS2 Version 1.4.4
>>>>> GigE Interconnect
>>>>> SaS Connection to the drive.
>>>>>
>>>>>
>>>>> cluster.conf:
>>>>> -
>>>>> cluster:
>>>>>node_count = 2
>>>>>name = HobCluster
>>>>> node:
>>>>>ip_port = 
>>>>>ip_address = 192.168.0.1
>>>>>number = 0
>>>>>name = hoban1
>>>>>cluster = HobCluster
>>>>> node:
>>>>>ip_port = 
>>>>>ip_address = 192.168.0.2
>>>>>number = 1
>>>>>name = hoban2
>>>>>cluster = HobCluster
>>>>>
>>>>>
>>>>>
>>>>> /etc/sysconfig/o2cb:
>>>>> ---
>>>>> O2CB_ENABLED=true
>>>>> O2CB_STACK=o2cb
>>>>> O2CB_BOOTCLUSTER=HobCluster
>>>>> O2CB_HEARTBEAT_THRESHOLD=
>>>>> O2CB_IDLE_TIMEOUT_MS=
>>>>> O2CB_KEEPALIVE_DELAY_MS=
>>>>> O2CB_RECONNECT_DELAY_MS=
>>>>>
>>>>>
>>>>> Status:
>>>>> 
>>>>> [root@hoban1 u03]# /etc/init.d/o2cb status
>>>>> Driver for "configfs": Loaded
>>>>> Filesystem "configfs": Mounted
>>>>> Driver for

Re: [Ocfs2-users] Ocfs and ASM

[Sunil Mushran]

There should be no conflict.

On 05/24/2011 11:32 AM, Keith W wrote:
> I have a lab system that is currently running Oracle RAC 11g
> with ASM volumes and grid infrastructure
>
> Is it possible to have an ocfs2 cluster running and accessing
> a different disk as well as the oracle clustering for RAC with
> the ASM's? Or will there be a conflict?
>
> +---+
> +   Keith +
> +---+
>
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Large Files Hang Server

[Sunil Mushran]

Writeback will help if the writes are on one node and the ls on another.
It is not clear if that is the case or not.

If both ops are on the same node, then it just could be the disk is slow.
The times shows almost all wall time. Very little sys and no user. top
will show io wait times.

On 05/24/2011 11:45 AM, Keith W wrote:
> No change in behavior.
> My mount options
> /dev/sdj1   /u03ocfs2   _netdev,noatime,data=writeback,nointr 0 0
>
> +---+
> +   Keith +
> +---+
>
> On Tue, 24 May 2011, Sunil Mushran wrote:
>
>> Repeat the same test but with volumes mounted with data=writeback
>> mount option.
>>
>> mount -o data=writeback /dev/sdX /path
>>
>> On 05/24/2011 07:11 AM, Keith W wrote:
>>> Hello list.
>>> Apologies in advance, this may be a bit long. Just trying to give
>>> as much info as I can at the outset.
>>>
>>> I have a two node setup that share a 500Gig SAS drive via ocfs2.
>>> When I move either large files 300Megs+ or a large number of smaller files
>>> onto or off of the volume, my terminal session will hang and if I do a
>>> directory listing in another terminal while doing a file transfer that
>>> terminal will hang as well.
>>>
>>> The only thing I can see that is not "typical" is that I had to change
>>> the port to  due to another application running on .
>>>
>>>
>>> Here is my configuration:
>>> 
>>> Oracle Enterprise Linux 5.5 (Oracle Updated Kernel 2.6.18-194.0.0.0.3.el5)
>>> OCFS2 Version 1.4.4
>>> GigE Interconnect
>>> SaS Connection to the drive.
>>>
>>>
>>> cluster.conf:
>>> -
>>> cluster:
>>>   node_count = 2
>>>   name = HobCluster
>>> node:
>>>   ip_port = 
>>>   ip_address = 192.168.0.1
>>>   number = 0
>>>   name = hoban1
>>>   cluster = HobCluster
>>> node:
>>>   ip_port = 
>>>   ip_address = 192.168.0.2
>>>   number = 1
>>>   name = hoban2
>>>   cluster = HobCluster
>>>
>>>
>>>
>>> /etc/sysconfig/o2cb:
>>> ---
>>> O2CB_ENABLED=true
>>> O2CB_STACK=o2cb
>>> O2CB_BOOTCLUSTER=HobCluster
>>> O2CB_HEARTBEAT_THRESHOLD=
>>> O2CB_IDLE_TIMEOUT_MS=
>>> O2CB_KEEPALIVE_DELAY_MS=
>>> O2CB_RECONNECT_DELAY_MS=
>>>
>>>
>>> Status:
>>> 
>>> [root@hoban1 u03]# /etc/init.d/o2cb status
>>> Driver for "configfs": Loaded
>>> Filesystem "configfs": Mounted
>>> Driver for "ocfs2_dlmfs": Loaded
>>> Filesystem "ocfs2_dlmfs": Mounted
>>> Checking O2CB cluster HobCluster: Online
>>> Heartbeat dead threshold = 31
>>> Network idle timeout: 3
>>> Network keepalive delay: 2000
>>> Network reconnect delay: 2000
>>> Checking O2CB heartbeat: Active
>>>
>>>
>>>
>>> Additional Info:
>>> ---
>>> While transfering a large file I do an ls -l on any file within
>>> the /u03(ocfs2) directory it goes quickly as expected.
>>>
>>> [root@hoban2 u03]# time ls -l asdf
>>> -rw-r--r-- 1 root root 0 May 23 08:23 asdf
>>>
>>> real0m0.003s
>>> user0m0.000s
>>> sys 0m0.003s
>>>
>>>
>>> During a large file transfer, a ls -l on the file being transfered
>>> hangs for a very long time.
>>> [root@hoban2 u03]# time ls -l
>>> total 547340
>>> -rw-r--r-- 1 root   root 0 May 23 08:23 asdf
>>> -rw-r--r-- 1 root   root 560476160 May 24  2011 
>>> Enterprise-R5-U5-x86_64.iso
>>> drwxr-xr-x 2 root   root  3896 May 22 09:29 lost+found
>>> drwxr-xr-x 3 oracle oinstall  3896 May 23 14:32 oracle
>>>
>>> real0m5.552s
>>> user0m0.000s
>>> sys 0m0.004s
>>>
>>> Once the file has completed it's transfer, the ls works just fine and
>>> nothing hangs. On occasion both terminal sessions will lock and need to
>>> be killed, the file never completing it's transfer.
>>>
>>> Any suggestions are greatly appreciated.
>>>
>>> +---+
>>> + Keith +
>>> +---+
>>>
>>>
>>>
>>> ___
>>> Ocfs2-users mailing list
>>> Ocfs2-users@oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Large Files Hang Server

[Sunil Mushran]

Repeat the same test but with volumes mounted with data=writeback
mount option.

mount -o data=writeback /dev/sdX /path

On 05/24/2011 07:11 AM, Keith W wrote:
> Hello list.
> Apologies in advance, this may be a bit long. Just trying to give
> as much info as I can at the outset.
>
> I have a two node setup that share a 500Gig SAS drive via ocfs2.
> When I move either large files 300Megs+ or a large number of smaller files
> onto or off of the volume, my terminal session will hang and if I do a
> directory listing in another terminal while doing a file transfer that
> terminal will hang as well.
>
> The only thing I can see that is not "typical" is that I had to change
> the port to  due to another application running on .
>
>
> Here is my configuration:
> 
> Oracle Enterprise Linux 5.5 (Oracle Updated Kernel 2.6.18-194.0.0.0.3.el5)
> OCFS2 Version 1.4.4
> GigE Interconnect
> SaS Connection to the drive.
>
>
> cluster.conf:
> -
> cluster:
>  node_count = 2
>  name = HobCluster
> node:
>  ip_port = 
>  ip_address = 192.168.0.1
>  number = 0
>  name = hoban1
>  cluster = HobCluster
> node:
>  ip_port = 
>  ip_address = 192.168.0.2
>  number = 1
>  name = hoban2
>  cluster = HobCluster
>
>
>
> /etc/sysconfig/o2cb:
> ---
> O2CB_ENABLED=true
> O2CB_STACK=o2cb
> O2CB_BOOTCLUSTER=HobCluster
> O2CB_HEARTBEAT_THRESHOLD=
> O2CB_IDLE_TIMEOUT_MS=
> O2CB_KEEPALIVE_DELAY_MS=
> O2CB_RECONNECT_DELAY_MS=
>
>
> Status:
> 
> [root@hoban1 u03]# /etc/init.d/o2cb status
> Driver for "configfs": Loaded
> Filesystem "configfs": Mounted
> Driver for "ocfs2_dlmfs": Loaded
> Filesystem "ocfs2_dlmfs": Mounted
> Checking O2CB cluster HobCluster: Online
> Heartbeat dead threshold = 31
>Network idle timeout: 3
>Network keepalive delay: 2000
>Network reconnect delay: 2000
> Checking O2CB heartbeat: Active
>
>
>
> Additional Info:
> ---
> While transfering a large file I do an ls -l on any file within
> the /u03(ocfs2) directory it goes quickly as expected.
>
> [root@hoban2 u03]# time ls -l asdf
> -rw-r--r-- 1 root root 0 May 23 08:23 asdf
>
> real  0m0.003s
> user  0m0.000s
> sys   0m0.003s
>
>
> During a large file transfer, a ls -l on the file being transfered
> hangs for a very long time.
> [root@hoban2 u03]# time ls -l
> total 547340
> -rw-r--r-- 1 root   root 0 May 23 08:23 asdf
> -rw-r--r-- 1 root   root 560476160 May 24  2011 
> Enterprise-R5-U5-x86_64.iso
> drwxr-xr-x 2 root   root  3896 May 22 09:29 lost+found
> drwxr-xr-x 3 oracle oinstall  3896 May 23 14:32 oracle
>
> real  0m5.552s
> user  0m0.000s
> sys   0m0.004s
>
> Once the file has completed it's transfer, the ls works just fine and
> nothing hangs. On occasion both terminal sessions will lock and need to
> be killed, the file never completing it's transfer.
>
> Any suggestions are greatly appreciated.
>
> +---+
> +   Keith +
> +---+
>
>
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS2 1.6 for RHEL?

[Sunil Mushran]

On 05/17/2011 06:00 AM, Sven Karlsson wrote:
> * Joel Becker wrote:
>> The latest ocfs2 is always in mainline.  Any distro building a
>> recent kernel an including ocfs2 will have it.  As Sunil says, this
>> includes Fedora, Debian, Ubuntu, and OpenSuSE.  Whether ocfs2 is enabled
>> in the CentOS kernel is up to CentOS.
>
> Linux 2.6.39-rc7  has version 1.5.0 in Linus'es tree:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/ocfs2/ver.c;h=e2488f4128a20f563459ba936cf263160e7350b2;hb=693d92a1bbc9e42681c42ed190bd42b636ca876f
>
> > From what I can tell from the changelog, you use the upstream-linus
> branch of git://git.kernel.org/pub/scm/linux/kernel/git/jlbec/ocfs2
> to sync with mainline, which also carries version 1.5.0.
>
> However, the latest version seems to be 1.6.3, available in the
> "unbreakable linux kernel" tree:
>
> http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=blob;f=fs/ocfs2/ver.c;h=8da71cb480f9cdd4cb0ff67f70a9b863f62b9a8b;hb=HEAD
>
> It was bumped in september last year...
>
> [~/linux-2.6-unbreakable]$ git log fs/ocfs2/ver.c
> commit c15e040971a341ca7e62448eac77385f91f6702f
> Author: Sunil Mushran
> Date:   Tue Sep 14 14:02:49 2010 -0700
>
>  ocfs2: Ups the version from 1.6.2 to 1.6.3
> ...
>
>
> It's a bit confusing that mainline ver.c was last updated in 2007, at
> the same time there are ocfs2 patches continuously accepted in
> mainline, and the unbreakable kernel is at 1.6.3... is ver.c simply
> forgotten or are there other differences between the trees?

The version number place holder in the mainline is not useful.
Because a mainline kernel ships with only one ocfs2 module,
we don't need the version# in the module. Hence we've stopped
updating it. Maybe we should just remove it. But that does not
mean it does not have the features. If you are looking for the list
of features in the mainline, then just look at mkfs.ocfs2's manpage.
It lists the mainline kernel version the feature was added in.

The ocfs2 module version was only required because we were
shipping ocfs2 outside of the kernel (for RHEL). OCFS2 1.4 is the
last release under that model.

Going forward, ocfs2 will only be available with the kernel. (I mean
there is nothing stopping a person from distributing it however
they want. But that's highly unlikely.)

OL, SLES, Fedora, Debian, OpenSUSE, ubuntu all ship ocfs2 along
with the kernel. If it is not in your distro, ping the vendor.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] fsck.ocfs2

[Sunil Mushran]

Set up a netconsole server to catch oops log.

On 5/16/2011 3:22 AM, Xavier Diumé wrote:
I don't know if is it possible, but kernel panic error is not in 
/var/log/kern.log.


2011/5/13 Sunil Mushran <mailto:sunil.mush...@oracle.com>>


Please do not remove the cc-s.

Hard for me to comment without knowing anything about the panic.

However, assuming that the panic message indicated that the volume
needs to be fsck-ed. In that case, the best course is to umount the
volume on all nodes and running fsck on one node.


On 05/13/2011 12:33 PM, Xavier Diumé wrote:

But initially the system had devices in /etf/fstab with _netdev
option. When system starts mounting a kernel panic appears,
sometimes after few minuts.
The only way that I could start the system was mounting all
devices one by one, with a previups fsck.
I don't know if it is the better way, but is the only that I've
used succesfully.

    2011/5/13 Sunil Mushran mailto:sunil.mush...@oracle.com>>

On 05/13/2011 11:44 AM, Xavier Diumé wrote:

Hello,
Is it possible to fsck a mounted filesystem. When one of
the cluster nodes reboots because a kernel panic, the
device requires fsck.ocfs2 because in mounted.ocfs2 -f
rebooted node is shown.


If mounted.ocfs2 -f shows the rebooted node, that means the
slotmap
has not been cleaned up as yet. That cleanup happens during node
recovery. If the volume is still mounted on another node, it
will get
cleaned up momentarily.

If however it does not get cleaned up, that means that the
volume is
not mounted on any node. In that case, the next mount will clean
up slotmap.

Either way one does not need to fsck just to cleanup the slotmap.




-- 
Xavier Diumé

http://socaqui.cat





--
Xavier Diumé
http://socaqui.cat


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS2 1.6 for RHEL?

[Sunil Mushran]

If big user base is the best way... then we have no worries because
ocfs2 1.6 is available on fedora, opensuse, ubuntu and debian.

On 05/13/2011 12:53 PM, Kristian Jörg wrote:
> We are using CentOS (based on RHEL5). So this basically means we are stuck 
> with ocfs2 1.4 forever when it comes to official releases?
> But I guess when you release the source code for 1.6 I could try and compile 
> it myself for CentOS? 1.6 is still GPL right?
>
> We have been testing ocfs2 since we have been having huge problems with GFS 
> with the kind of filesystems we have (lots and lots of small files and mostly 
> read access). Ocfs2 seems to solve all these performance problems and we had 
> high hopes for it. However the news that 1.6 and forward will not be 
> available for RHEL/CentOS is bad news. I do hope you will reconsider. A big 
> user base is the best way to make successful products IMHO.
>
> /Kristian
>
>
> Sunil Mushran skrev 2011-05-13 18:46:
>> Support is a whole different ballgame. I am only talking
>> about availability. And I interpreted that qs to be asking
>> whether ocfs2 1.6 be available for the standard rhel kernels.
>> And the answer is no. It will be available only for uek.
>>
>> On 05/13/2011 08:34 AM, Fahrenfort, Keith (HP Shared Solutions Architect for 
>> Oracle) wrote:
>>> Hi Sunil, can you clarify - I understood that OCFS2 1.6 on RHEL would be 
>>> supported, as long as you use RHEL with UEK and an Oracle Support agreement 
>>> - per MOS note 1253272.1?
>>>
>>> Thanks! Regards!
>>>
>>> Keith Fahrenfort
>>> Solutions Architect - HP-Oracle Competency Center
>>> Americas Shared Solutions Architecture (SSA)
>>> Hewlett-Packard Company
>>> 281 475 8632 / Tel
>>> kei...@hp.com / Email
>>> Reach the team at s...@hp.com
>>>
>>> -Original Message-
>>> From: ocfs2-users-boun...@oss.oracle.com 
>>> [mailto:ocfs2-users-boun...@oss.oracle.com] On Behalf Of Sunil Mushran
>>> Sent: Friday, May 13, 2011 10:06 AM
>>> To: Kristian Jörg
>>> Cc: ocfs2-users@oss.oracle.com
>>> Subject: Re: [Ocfs2-users] OCFS2 1.6 for RHEL?
>>>
>>> On 05/13/2011 03:13 AM, Kristian Jörg wrote:
>>>> Hello!
>>>>
>>>> When is it planned ocfs2 1.6 will be available for RHEL?
>>>>
>>>> /Kristian
>>> No plans. Only OL/UEK.
>>>
>>> ___
>>> Ocfs2-users mailing list
>>> Ocfs2-users@oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>
>
>


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] fsck.ocfs2

[Sunil Mushran]

Please do not remove the cc-s.

Hard for me to comment without knowing anything about the panic.

However, assuming that the panic message indicated that the volume
needs to be fsck-ed. In that case, the best course is to umount the
volume on all nodes and running fsck on one node.

On 05/13/2011 12:33 PM, Xavier Diumé wrote:

But initially the system had devices in /etf/fstab with _netdev option. When 
system starts mounting a kernel panic appears, sometimes after few minuts.
The only way that I could start the system was mounting all devices one by one, 
with a previups fsck.
I don't know if it is the better way, but is the only that I've used 
succesfully.

2011/5/13 Sunil Mushran mailto:sunil.mush...@oracle.com>>

On 05/13/2011 11:44 AM, Xavier Diumé wrote:

Hello,
Is it possible to fsck a mounted filesystem. When one of the cluster 
nodes reboots because a kernel panic, the device requires fsck.ocfs2 because in 
mounted.ocfs2 -f rebooted node is shown.


If mounted.ocfs2 -f shows the rebooted node, that means the slotmap
has not been cleaned up as yet. That cleanup happens during node
recovery. If the volume is still mounted on another node, it will get
cleaned up momentarily.

If however it does not get cleaned up, that means that the volume is
not mounted on any node. In that case, the next mount will clean
up slotmap.

Either way one does not need to fsck just to cleanup the slotmap.




--
Xavier Diumé
http://socaqui.cat


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] fsck.ocfs2

[Sunil Mushran]

On 05/13/2011 11:44 AM, Xavier Diumé wrote:
> Hello,
> Is it possible to fsck a mounted filesystem. When one of the cluster nodes 
> reboots because a kernel panic, the device requires fsck.ocfs2 because in 
> mounted.ocfs2 -f rebooted node is shown.

If mounted.ocfs2 -f shows the rebooted node, that means the slotmap
has not been cleaned up as yet. That cleanup happens during node
recovery. If the volume is still mounted on another node, it will get
cleaned up momentarily.

If however it does not get cleaned up, that means that the volume is
not mounted on any node. In that case, the next mount will clean
up slotmap.

Either way one does not need to fsck just to cleanup the slotmap.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS2 1.6 for RHEL?

[Sunil Mushran]

Support is a whole different ballgame. I am only talking
about availability. And I interpreted that qs to be asking
whether ocfs2 1.6 be available for the standard rhel kernels.
And the answer is no. It will be available only for uek.

On 05/13/2011 08:34 AM, Fahrenfort, Keith (HP Shared Solutions Architect for 
Oracle) wrote:
> Hi Sunil, can you clarify - I understood that OCFS2 1.6 on RHEL would be 
> supported, as long as you use RHEL with UEK and an Oracle Support agreement - 
> per MOS note 1253272.1?
>
> Thanks! Regards!
>
> Keith Fahrenfort
> Solutions Architect - HP-Oracle Competency Center
> Americas Shared Solutions Architecture (SSA)
> Hewlett-Packard Company
> 281 475 8632 / Tel
> kei...@hp.com / Email
> Reach the team at s...@hp.com
>
> -Original Message-
> From: ocfs2-users-boun...@oss.oracle.com 
> [mailto:ocfs2-users-boun...@oss.oracle.com] On Behalf Of Sunil Mushran
> Sent: Friday, May 13, 2011 10:06 AM
> To: Kristian Jörg
> Cc: ocfs2-users@oss.oracle.com
> Subject: Re: [Ocfs2-users] OCFS2 1.6 for RHEL?
>
> On 05/13/2011 03:13 AM, Kristian Jörg wrote:
>> Hello!
>>
>> When is it planned ocfs2 1.6 will be available for RHEL?
>>
>> /Kristian
> No plans. Only OL/UEK.
>
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] OCFS2 1.6 for RHEL?

[Sunil Mushran]

On 05/13/2011 03:13 AM, Kristian Jörg wrote:
> Hello!
>
> When is it planned ocfs2 1.6 will be available for RHEL?
>
> /Kristian

No plans. Only OL/UEK.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] 答复: 答复: hi,if i can post ocfs2-dmesg to you?i have trouble on ocfs2

[Sunil Mushran]

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=40f165f416bde747d85cdf71bc9dde700912f71f

On 05/13/2011 03:04 AM, Longguang Yue wrote:
>
> Sorry to trouble. I do not know [# git name-rev --tags
> 40f165f416bde747d85cdf71bc9dde700912f71f
> 40f165f416bde747d85cdf71bc9dde700912f71f tags/v2.6.35-rc6~34^2~13]
>
> Would you give me a patch or URL.
>
> I only know kernel.org packag
>
> thanks
>
> *发件人:*Sunil Mushran [mailto:sunil.mush...@oracle.com]
> *发送时间:*2011年5月13日10:09
> *收件人:*Longguang Yue
> *抄送:*ocfs2-users@oss.oracle.com
> *主题:*Re: 答复: hi,if i can post ocfs2-dmesg to you?i have trouble on
> ocfs2
>
> On 05/12/2011 06:50 PM, Longguang Yue wrote:
>
> Thank you first of all
>
> totally, there are 5 kinds of error occur.
>
> Spinlock leads to cpu lockup, o2net modules panic, kernel BUG at
> mm/slub.c:2969, BUG unable to handle kernel paging request at addr
>
> My environment: kernel-2.6.32.23 + xen + ocfs2(1.5)
>
> Thanks again.
>
>
> This looks like the deadlock as explained in
> this patch.
>
> commit 40f165f416bde747d85cdf71bc9dde700912f71f
> Author: Tao Ma  <mailto:tao...@oracle.com>
> Date: Fri May 28 14:22:59 2010 +0800
>
> ocfs2: Move orphan scan work to ocfs2_wq.
>
> # git name-rev --tags 40f165f416bde747d85cdf71bc9dde700912f71f
> 40f165f416bde747d85cdf71bc9dde700912f71f tags/v2.6.35-rc6~34^2~13
>
> Fixed in 2.6.35. This should apply as-is to 2.6.32.
>

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] 答复: hi,if i can post ocfs2-dmesg to you?i have trouble on ocfs2

[Sunil Mushran]

On 05/12/2011 06:50 PM, Longguang Yue wrote:
>
> Thank you first of all
>
> totally, there are 5 kinds of error occur.
>
> Spinlock leads to cpu lockup, o2net modules panic, kernel BUG at 
> mm/slub.c:2969, BUG unable to handle kernel paging request at addr
>
> My environment: kernel-2.6.32.23 + xen + ocfs2(1.5)
>
> Thanks again.
>

This looks like the deadlock as explained in
this patch.

commit 40f165f416bde747d85cdf71bc9dde700912f71f
Author: Tao Ma 
Date: Fri May 28 14:22:59 2010 +0800

ocfs2: Move orphan scan work to ocfs2_wq.

# git name-rev --tags 40f165f416bde747d85cdf71bc9dde700912f71f
40f165f416bde747d85cdf71bc9dde700912f71f tags/v2.6.35-rc6~34^2~13

Fixed in 2.6.35. This should apply as-is to 2.6.32.
___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] How to change node name ?

[Sunil Mushran]

It is a manual process until 1.6. The upcoming release of tools (1.8)
will allow online modification and removal.

On 05/12/2011 05:02 AM, Thomas Lau wrote:
Guys, how could I change node name and delete nodes after add into 
cluster?


/Connected by MOTOBLUR™/


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Server hang after error

[Sunil Mushran]

Your config is sufficient. Hard to say why it did not reboot. Ping the debian
mailing list to see if there are reports of the same on whatever kernel you
are on.

As far as the reason for it goes, there should have been a message just
prior to the Kernel Panic message. Most likely reason is that it hit a on
disk corruption. I would suggest fscking the device.

On 05/09/2011 05:51 AM, Xavier Diumé wrote:

We have two servers with Debian 6.0.  Both have multipath and iscsi lun's from 
Equallogic.
All volumes are in a cluster named kvmcluster.
It was working well but after an iscsiadm login on first server we got:
Kernel Panic - not syncing : ocfs2 (device dm-18 panic forced after error)
Server will reboot after 30 seconds...

But the server didn't reboot. We have to shutdown manually.

The system is virtualizing some systems and obviously all of them got down.

/proc/sys/kernel/panic
30
/proc/sys/kernel/panic_on_oops
1

Why the system didn't reboot? We have to configure something else?
Which problem may cause an ocfs2 error?

Thanks,
--
Xavier Diumé


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] read/write performance across cluster

[Sunil Mushran]

On 05/04/2011 09:56 AM, Florin Andrei wrote:
> On 05/04/2011 09:44 AM, Srinivas Eeda wrote:
>> Yes, there is locking involved. Extending a file needs an exclusive
>> lock. Grepping a file needs read lock. If the same node(lets call it
>> writer node) does extending and grepping, then grep already has a
>> compatible lock and the data is also cached on this node. When another
>> node does the grep(lets call it reader node), then it needs to acquire
>> read lock. This requires the writer node to down grade the lock which
>> requires the node to flush data to the disk. Then the reader node has to
>> read the data to the cache. So when grep is done on the reader node,
>> there are additional messages and additional I/O's to the disk are done.
> Any workarounds?
>
> Obviously doing read and append on the same node is one way.
>
> Could the logger extend the file in a different way? The software is
> homegrown, we could modify it to make it more OCFS2-friendly, if that's
> possible.

Don't append as often. IOW, make large appends and not few bytes
at a time like all loggers.

> OCFS2 settings to tinker with?

Cache coherency is not a configurable option.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] Kernel Feature List?

[Sunil Mushran]

man mkfs.ocfs2 is better. 

On May 3, 2011, at 6:24 PM, Tiger Yang  wrote:

> On 05/02/2011 03:57 PM, Stefan Priebe - Profihost AG wrote:
>> Hi,
>> 
>> is there a list available which ocfs2 feature is available at which
>> vanilla kernel version?
>> 
>> Stefan
> Hi,
> 
> There is one list for mainline kernel. 
> http://oss.oracle.com/osswiki/OCFS2/Roadmap
> 
> Thanks,
> Tiger
> 
> ___
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] How long for an fsck?

[Sunil Mushran]

On 04/23/2011 12:24 AM, Josep Guerrero wrote:
> Hello,
>
 How long did the debugfs output take?
>>> I think about 30 minutes. No more than 50 for sure (just by looking at
>>> the times of the mails).
>>>
 Did fsck eventually finish?
>>> No. I had to cancel it after it stayed 24 hours in the same state,
>>> showing the same message. It never moved beyond "Pass 0a", and always
>>> was using 100% CPU in one core. I don't know if it would have finished
>>> on its own.
>>>
 BTW, you said one of the cores was at 100%. What does top show?
 Is fsck the main contributor or is some other process spinning?
>>> It was fsck (I kept a top opened the whole time, and fsck always was
>>> around 99% CPU usage).
>>>
 I have a theory as to why it is slow. But I would like some
 confirmation. My theory had fsck have high wait%. I seem to be missing
 something.
>>> I didn't look at the wait%, but I checked the physical disk load with
>>> iotop and it was very low, so it didn't look like fsck was being slow
>>> because of the disk. In the filesystem I successfully "fscked" before
>>> (the 3 TB one that took less than 60 minutes), it started doing
>>> something similar (very high CPU usage, low disk load) but after several
>>> minutes (when the rest of the messages after "Pass 0a" appeared), it did
>>> just the opposite: low CPU use, high disk load. Both filesystems are
>>> physically on the same set of disks (the 16TB logical volume is an
>>> striped LVM volume that fills about 75% of the 21 physical disks and the
>>> 3TB is another striped LVM volume filling the remaining space of the
>>> same disks) so I don't think it's a problem with the physical devices
>>> (of course, I could be wrong).
>> File a bz. This will need some investigation.
> Sorry, I suppose it must be an stupid question but, what's a "bz"? I 've found
> several references to it in the list, but no explanation of what it is or how
> it is done.

http://oss.oracle.com/bugzilla

We want to keep track lest we forget.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] can't mount device

[Sunil Mushran]

Is this during boot or is the mount manual?
Does it succeed on second attempt?

On 04/22/2011 06:33 AM, Christophe BOUDER wrote:
> Hello,
> i'm running ocfs2 on 27 nodes
> with 2 devices ( 2 fiber channel disk array  storage)
> on debian system
> vanilla kernel 2.6.38.2
> ocfs2-tools1.6.3-1
>
> sometimes when i want to mount the device1
> after a reboot i can't :
>
>   (mount.ocfs2,9543,2):dlm_join_domain:1857 Timed out joining dlm domain
> EA9679D689F64044BFBCDF0D2F7BCDF0 after 94000 msecs
>
>
> the other nodes have already mounted device1
> and have heavy I/O access on it.
> The node which want to mount device1 have already mounted device2.
>
> any help welcome.
> thank you.
>
> see the file o2cb.
>
> cat /etc/default/o2cb
> #
> # This is a configuration file for automatic startup of the O2CB
> # driver.  It is generated by running 'dpkg-reconfigure ocfs2-tools'.
> # Please use that method to modify this file.
> #
>
> # O2CB_ENABLED: 'true' means to load the driver on boot.
> O2CB_ENABLED=true
>
> # O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
> O2CB_BOOTCLUSTER=bigstock
>
> # O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
> O2CB_HEARTBEAT_THRESHOLD=61
>
> # O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is
> considered dead.
> O2CB_IDLE_TIMEOUT_MS=6
>
> # O2CB_KEEPALIVE_DELAY_MS: Max. time in ms before a keepalive packet is sent.
> O2CB_KEEPALIVE_DELAY_MS=4000
>
> # O2CB_RECONNECT_DELAY_MS: Min. time in ms between connection attempts.
> O2CB_RECONNECT_DELAY_MS=4000
>
>


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] How long for an fsck?

[Sunil Mushran]

On 04/22/2011 02:33 PM, Sunil Mushran wrote:
> On 04/21/2011 10:46 AM, Josep Guerrero wrote:
>> Hello again,
>>
>> It just finished. The output file is almost 9 MB long, but compressed is less
>> than 1 MB. I attach it to the message.
>>
>>> Do:
>>> # debugfs.ocfs2 -R "stat //global_bitmap" /dev/hidrahome/lvol0
>>>
>>> Does this hang too? Redirect the output to a file. That will give us some
>>> clues.
> How long did the debugfs output take?
> Did fsck eventually finish?
> If so, how long did that take?
> Approximately.
>
> I have a theory as to why it is slow. But I would like some confirmation.

BTW, you said one of the cores was at 100%. What does top show?
Is fsck the main contributor or is some other process spinning?

My theory had fsck have high wait%. I seem to be missing something.

___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] How long for an fsck?

[Sunil Mushran]

On 04/21/2011 10:46 AM, Josep Guerrero wrote:
> Hello again,
>
> It just finished. The output file is almost 9 MB long, but compressed is less
> than 1 MB. I attach it to the message.
>
>> Do:
>> # debugfs.ocfs2 -R "stat //global_bitmap" /dev/hidrahome/lvol0
>>
>> Does this hang too? Redirect the output to a file. That will give us some
>> clues.
>

How long did the debugfs output take?
Did fsck eventually finish?
If so, how long did that take?
Approximately.

I have a theory as to why it is slow. But I would like some confirmation.

Thanks
Sunil


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

Re: [Ocfs2-users] How long for an fsck?

[Sunil Mushran]

On 04/21/2011 06:43 AM, Josep Guerrero wrote:
> I have a cluster with 8 nodes, all of them running Debian Lenny (plus some
> additions so multipath and Infiniband works), which share an array of 48 1TB
> disks. Those disks form 22 pairs of hardware RAID1, plus 4 spares). The first
> 21 pairs are organized in two striped LVM logical volumes, of 16 and 3 TB,
> both formatted with ocfs2. The kernel is the version supplied with the
> distribution (2.6.26-2-amd64).
>
> I wanted to run an fsck on both volumes because of some errors I was getting
> (probably unrelated to the filesystems, but I wanted to check). On the 3TB
> volume (around 10% full) the check worked perfectly, and finished in less than
> an hour (this was run with the fsck.ocfs2 provided by Lenny ocfs2-tools,
> version 1.4.1):
>


> but the check for the second filesystem (around 40% full) did this:
>
> 
> hidra0:/usr/local/src# fsck.ocfs2 -f /dev/hidrahome/lvol0
> Checking OCFS2 filesystem in /dev/hidrahome/lvol0:
>label:
>uuid:   6a a9 0e aa cf 33 45 4c b4 72 3a b6 7c 3b 8d 57
>number of blocks:   4168098816
>bytes per block:4096
>number of clusters: 4168098816
>bytes per cluster:  4096
>max slots:  8
>
> /dev/hidrahome/lvol0 was run with -f, check forced.
> Pass 0a: Checking cluster allocation chains
> =
>
> and stayed there for 8 hours (all the time keeping one core around 100% CPU
> usage and with a light load on the disks; this was consistent with the same
> step in the previous run, but of course it didn't take so long). I thought
> that maybe I had run into some bug, so I interrupted the process, downloaded
> ocfs2-tools 1.4.4 sources, compiled them, and tried with that fsck, obtaining
> similar results, since it's been running for almost 7 hours like this:
>
> =
> hidra0:/usr/local/src/ocfs2-tools-1.4.4/fsck.ocfs2# ./fsck.ocfs2 -f
> /dev/hidrahome/lvol0
> fsck.ocfs2 1.4.4
> Checking OCFS2 filesystem in /dev/hidrahome/lvol0:
>Label:
>UUID:   6AA90EAACF33454CB4723AB67C3B8D57
>Number of blocks:   4168098816
>Block size: 4096
>Number of clusters: 4168098816
>Cluster size:   4096
>Number of slots:8
>
> /dev/hidrahome/lvol0 was run with -f, check forced.
> Pass 0a: Checking cluster allocation chains
>
> =
>
> and with one core CPU at 100%.
>
> Could someone tell me if this is normal? I've been searching the web and
> checking manuals for information on how long this checks should take, and
> apart from one message in this list mentioning that 3 days in a 8 TB 
> filesystem
> with 300 GB was too long, I haven't been able to find anything.
>
> If this is normal, is there any way to estimate, taking into account that the
> first filesystem uses exactly the same disks and took less than an hour to
> check, how long it should take for this other filesystem?

Do:
# debugfs.ocfs2 -R "stat //global_bitmap" /dev/hidrahome/lvol0

Does this hang too? Redirect the output to a file. That will give us some clues.


___
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users