Re: [ovirt-users] vm pauses with "vm has paused due to unknown storage error

2016-06-26 Thread Krutika Dhananjay
Hi Bill, After glusterfs 3.7.11, around 4-5 bugs were found in sharding and replicate modules and fixed, some of them causing the VM(s) to pause. Could you share the glusterfs client logs from around the time the issue was seen? This will help me confirm it's the same issue, or even debug further

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

2016-07-26 Thread Krutika Dhananjay
any vms at any point before or after the upgrade? -Krutika On Mon, Jul 25, 2016 at 11:30 PM, David Gossage <dgoss...@carouselchecks.com > wrote: > > On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <kdhan...@redhat.com> > wrote: > >> OK, could you try the followin

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

2016-07-25 Thread Krutika Dhananjay
any improvement? -Krutika On Mon, Jul 25, 2016 at 4:57 PM, Samuli Heinonen <samp...@neutraali.net> wrote: > Hi, > > > On 25 Jul 2016, at 12:34, David Gossage <dgoss...@carouselchecks.com> > wrote: > > > > On Mon, Jul 25, 2016 at 1:01 AM, Krutika Dhananja

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

2016-07-22 Thread Krutika Dhananjay
Hi David, Could you also share the brick logs from the affected volume? They're located at /var/log/glusterfs/bricks/.log. Also, could you share the volume configuration (output of `gluster volume info `) for the affected volume(s) AND at the time you actually saw this issue? -Krutika On

Re: [ovirt-users] VDSM hang

2017-03-12 Thread Krutika Dhananjay
Hi, Could you please share your volume info output? -Krutika On Fri, Mar 10, 2017 at 6:41 PM, p...@email.cz wrote: > freez / freezing > IO operations are paused from any reasons > available posibilities are > 1) net - any tcp framework collapse > 2) gluster interconnect due

Re: [ovirt-users] [Gluster-users] Very poor GlusterFS performance

2017-06-20 Thread Krutika Dhananjay
Couple of things: 1. Like Darrell suggested, you should enable stat-prefetch and increase client and server event threads to 4. # gluster volume set performance.stat-prefetch on # gluster volume set client.event-threads 4 # gluster volume set server.event-threads 4 2. Also glusterfs-3.10.1

Re: [ovirt-users] [Gluster-users] Very poor GlusterFS performance

2017-06-21 Thread Krutika Dhananjay
No, you don't need to do any of that. Just executing volume-set commands is sufficient for the changes to take effect. -Krutika On Wed, Jun 21, 2017 at 3:48 PM, Chris Boot <bo...@bootc.net> wrote: > [replying to lists this time] > > On 20/06/17 11:23, Krutika Dhananjay wr

Re: [ovirt-users] [Gluster-users] Very poor GlusterFS performance

2017-06-21 Thread Krutika Dhananjay
No. It's just that in the internal testing that was done here, increasing the thread count beyond 4 did not improve the performance any further. -Krutika On Tue, Jun 20, 2017 at 11:30 PM, mabi wrote: > Dear Krutika, > > Sorry for asking so naively but can you tell me on

Re: [ovirt-users] vm has been paused due to unknown storage

2017-05-29 Thread Krutika Dhananjay
GID: 133010] > [shard.c:1725:shard_common_lookup_shards_cbk] > 0-gv2-shard: Lookup on shard 173 failed. Base file gfid = > 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available] > > > thanks > > -- > *De: *"Sahina Bose" <sab...@redhat.com> > *

Re: [ovirt-users] oVirt gluster sanlock issue

2017-06-05 Thread Krutika Dhananjay
This seems like a case of O_DIRECT reads and writes gone wrong, judging by the 'Invalid argument' errors. The two operations that have failed on gluster bricks are: [2017-06-05 09:40:39.428979] E [MSGID: 113072] [posix.c:3453:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid

Re: [ovirt-users] oVirt gluster sanlock issue

2017-06-06 Thread Krutika Dhananjay
t (127488) at the log does not seem aligned at 4K. >> >> Alex >> >> On Mon, Jun 5, 2017 at 2:47 PM, Abi Askushi <rightkickt...@gmail.com> >> wrote: >> >>> Hi Krutika, >>> >>> I am saying that I am facing this issue with 4k dri

Re: [ovirt-users] oVirt gluster sanlock issue

2017-06-06 Thread Krutika Dhananjay
I stand corrected. Just realised the strace command I gave was wrong. Here's what you would actually need to execute: strace -y -ff -o -Krutika On Tue, Jun 6, 2017 at 3:20 PM, Krutika Dhananjay <kdhan...@redhat.com> wrote: > OK. > > So for the 'Transport endpoint is not c

[ovirt-users] Re: Gluster problems, cluster performance issues

2018-05-29 Thread Krutika Dhananjay
Adding Ravi to look into the heal issue. As for the fsync hang and subsequent IO errors, it seems a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1497156 and Paolo Bonzini from qemu had pointed out that this would be fixed by the following commit: commit

[ovirt-users] Re: HE + Gluster : Engine corrupted?

2018-07-02 Thread Krutika Dhananjay
error) ... ... Adding Ravi who works on replicate component to hep resolve the mismatches. -Krutika On Mon, Jul 2, 2018 at 12:27 PM, Krutika Dhananjay wrote: > Hi, > > Sorry, I was out sick on Friday. I am looking into the logs. Will get back > to you in some time. > > -

[ovirt-users] Re: HE + Gluster : Engine corrupted?

2018-06-25 Thread Krutika Dhananjay
Could you share the gluster mount and brick logs? You'll find them under /var/log/glusterfs. Also, what's the version of gluster you're using? Also, output of `gluster volume info `? -Krutika On Thu, Jun 21, 2018 at 9:50 AM, Sahina Bose wrote: > > > On Wed, Jun 20, 2018 at 11:33 PM, Hanson

[ovirt-users] Re: VM disk corruption with LSM on Gluster

2019-03-27 Thread Krutika Dhananjay
s really went down - performing inside vm fio tests. > > On Wed, Mar 27, 2019, 07:03 Krutika Dhananjay wrote: > >> Could you enable strict-o-direct and disable remote-dio on the src volume >> as well, restart the vms on "old" and retry migration? >> >> # gluster v

[ovirt-users] Re: Gluster VM image Resync Time

2019-03-28 Thread Krutika Dhananjay
On Thu, Mar 28, 2019 at 2:28 PM Krutika Dhananjay wrote: > Gluster 5.x does have two important performance-related fixes that are not > part of 3.12.x - > i. in shard-replicate interaction - > https://bugzilla.redhat.com/show_bug.cgi?id=1635972 > Sorry, wrong bug-id. This

[ovirt-users] Re: Gluster VM image Resync Time

2019-03-28 Thread Krutika Dhananjay
s the situation now with 5.5 ? > Best Regards, > Strahil Nikolov > On Mar 28, 2019 08:56, Krutika Dhananjay wrote: > > Right. So Gluster stores what are called "indices" for each modified file > (or shard) > under a special hidden directory of the "good" br

[ovirt-users] Re: [Gluster-users] Announcing Gluster release 5.5

2019-03-29 Thread Krutika Dhananjay
Questions/comments inline ... On Thu, Mar 28, 2019 at 10:18 PM wrote: > Dear All, > > I wanted to share my experience upgrading from 4.2.8 to 4.3.1. While > previous upgrades from 4.1 to 4.2 etc. went rather smooth, this one was a > different experience. After first trying a test upgrade on a 3

[ovirt-users] Re: Gluster VM image Resync Time

2019-03-28 Thread Krutika Dhananjay
r node know which shards were modified after it went > down? > Do the other Gluster nodes keep track of it? > > Regards, > > > Indivar Nair > > > On Thu, Mar 28, 2019 at 9:45 AM Krutika Dhananjay > wrote: > >> Each shard is a separate file of size equal to va

[ovirt-users] Re: Gluster VM image Resync Time

2019-03-27 Thread Krutika Dhananjay
will still have to compare each shard to determine whether > there are any changes that need to be replicated. > > Am I right? > > +Krutika Dhananjay > > > > Regards, > > > > Indivar Nair > > > > > > > > On Wed, Mar 27, 2019 at 4:34

[ovirt-users] Re: [Gluster-users] Announcing Gluster release 5.5

2019-03-31 Thread Krutika Dhananjay
Adding back gluster-users Comments inline ... On Fri, Mar 29, 2019 at 8:11 PM Olaf Buitelaar wrote: > Dear Krutika, > > > > 1. I’ve made 2 profile runs of around 10 minutes (see files > profile_data.txt and profile_data2.txt). Looking at it, most time seems be > spent at the fop’s fsync and

[ovirt-users] Re: VM disk corruption with LSM on Gluster

2019-03-26 Thread Krutika Dhananjay
jen wrote: > On 26-03-19 14:23, Sahina Bose wrote: > > +Krutika Dhananjay and gluster ml > > > > On Tue, Mar 26, 2019 at 6:16 PM Sander Hoentjen > wrote: > >> Hello, > >> > >> tl;dr We have disk corruption when doing live storage migration on oVirt &

[ovirt-users] Re: Tracking down high writes in GlusterFS volume

2019-02-25 Thread Krutika Dhananjay
On Fri, Feb 15, 2019 at 12:30 AM Jayme wrote: > Running an oVirt 4.3 HCI 3-way replica cluster with SSD backed storage. > I've noticed that my SSD writes (smart Total_LBAs_Written) are quite high > on one particular drive. Specifically I've noticed one volume is much much > higher total bytes

[ovirt-users] Re: Gluster - performance.strict-o-direct and other performance tuning in different storage backends

2019-02-25 Thread Krutika Dhananjay
Gluster's write-behind translator by default buffers writes for flushing to disk later, *even* when the file is opened with O_DIRECT flag. Not honoring O_DIRECT could mean a reader from another client could be READing stale data from bricks because some WRITEs may not yet be flushed to disk.

[ovirt-users] Re: oVirt Performance (Horrific)

2019-03-13 Thread Krutika Dhananjay
; cluster.granular-entry-heal: enable > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > > On Thu, Mar 7, 2019 at 1:00 AM Krutika Dhananjay > wrote: > >> So from the profile, it appears th

[ovirt-users] Re: oVirt Performance (Horrific)

2019-03-06 Thread Krutika Dhananjay
Hi, Could you share the following pieces of information to begin with - 1. output of `gluster volume info $AFFECTED_VOLUME_NAME` 2. glusterfs version you're running -Krutika On Sat, Mar 2, 2019 at 3:38 AM Drew R wrote: > Saw some people asking for profile info. So I had started a migration

[ovirt-users] Re: oVirt Performance (Horrific)

2019-03-06 Thread Krutika Dhananjay
implementation. This was fixed at https://review.gluster.org/c/glusterfs/+/19503. I need the two things I asked for in the prev mail to confirm if you're hitting the same issue. -Krutika On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay wrote: > Hi, > > Could you share the followi

[ovirt-users] Re: [ovirt-announce] Re: [ANN] oVirt 4.3.4 First Release Candidate is now available

2019-05-21 Thread Krutika Dhananjay
dea was explored sometime back here - https://review.gluster.org/c/glusterfs/+/19133 But there were some issues that were identified with the approach, so it had to be dropped. -Krutika Thanks for the detailed explanation. > > Best Regards, > Strahil Nikolov > On May 21, 2019 08:36, Kr

[ovirt-users] Re: [ovirt-announce] Re: [ANN] oVirt 4.3.4 First Release Candidate is now available

2019-05-20 Thread Krutika Dhananjay
So in our internal tests (with nvme ssd drives, 10g n/w), we found read performance to be better with choose-local disabled in hyperconverged setup. See https://bugzilla.redhat.com/show_bug.cgi?id=1566386 for more information. With choose-local off, the read replica is chosen randomly (based on

[ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing

2019-11-25 Thread Krutika Dhananjay
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer wrote: > On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov > wrote: > >> On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose wrote: >> >> >> >> On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov >> wrote: >> >> Hi All, >> >> another clue in the logs : >>

[ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing

2019-12-01 Thread Krutika Dhananjay
Sorry about the late response. I looked at the logs. These errors are originating from posix-acl translator - *[2019-11-17 07:55:47.090065] E [MSGID: 115050] [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496: LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6

[ovirt-users] Re: Sometimes paused due to unknown storage error on gluster

2020-04-08 Thread Krutika Dhananjay
On Tue, Apr 7, 2020 at 7:36 PM Gianluca Cecchi wrote: > > OK. So I set log at least at INFO level on all subsystems and tried a > redeploy of Openshift with 3 mater nodes and 7 worker nodes. > One worker got the error and VM in paused mode > > Apr 7, 2020, 3:27:28 PM VM worker-6 has been paused

[ovirt-users] Re: HCI cluster single node error making template

2020-03-30 Thread Krutika Dhananjay
Agreed. Please share the bug report when you're done filing it. In addition to the logs Nir requested, include gluster version and the `gluster volume info` output in your report. We'll take the discussion forward on the bz. -Krutika On Wed, Mar 25, 2020 at 11:39 PM Nir Soffer wrote: > On

[ovirt-users] Re: Weird problem starting VMs in oVirt-4.4

2020-06-17 Thread Krutika Dhananjay
Yes, so the bug has been fixed upstream and the backports to release-7 and release-8 of gluster pending merge. The fix should be available in the next .x release of gluster-7 and 8. Until then like Nir suggested, please turn off performance.stat-prefetch on your volumes. -Krutika On Wed, Jun 17,