Hi
I did a make install in my ceph build and also did make install on the fio and
ensured the latest binaries were installed. Now, fio is failing with below
errors for the rbd device with EC pool as data pool. I have shared the "rbd ls"
and my rbd.fio conf file details below. Let me know if you think there is any
configuration issues here.
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ ./bin/rbd ls
2016-12-08 17:14:39.663643 7f487017dec0 -1 WARNING: the following dangerous and
experimental features are enabled: *
rbd_12 <<== normal rbd device
rbdimg_1 <<== rbd device with EC pool as data pool
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ cat rbd.fio
######################################################################
# Example test for the RBD engine.
#
# Runs a 4k random write test against a RBD via librbd
#
# NOTE: Make sure you have either a RBD named 'fio_test' or change
# the rbdname parameter.
######################################################################
[global]
#logging
#write_iops_log=write_iops_log
#write_bw_log=write_bw_log
#write_lat_log=write_lat_log
ioengine=rbd
clientname=admin
pool=rbd
rbdname=rbdimg_1
#rbdname=fio_test
rw=randwrite
bs=4k
[rbd_iodepth32]
iodepth=32
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ ./fio ./rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd,
iodepth=32
fio-2.15-21-g4871
Starting 1 process
rbd engine: RBD version: 0.1.11
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=64757760, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=408551424, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=905359360, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=980332544, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=388386816, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=431546368, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=863858688, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=421429248, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=600182784, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=794877952, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=902078464, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=338862080, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=739086336, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=455880704, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=504672256, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=71749632, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=863690752, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=848175104, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=905744384, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=78327808, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=997011456, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=379588608, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=472981504, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=699166720, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=665198592, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=265089024, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=1060147200, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=4313088, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=1067864064, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=918290432, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=279457792, buflen=4096
fio: io_u error on file rbd_iodepth32.0.0: Unknown error -95: write
offset=271794176, buflen=4096
fio: pid=23094, err=-95/file:io_u.c:1712, func=io_u error, error=Unknown error
-95
Jobs: 1 (f=0)
rbd_iodepth32: (groupid=0, jobs=1): err=-95 (file:io_u.c:1712, func=io_u error,
error=Unknown error -95): pid=23094: Thu Dec 8 17:12:34 2016
cpu : usr=7.32%, sys=29.88%, ctx=96, majf=0, minf=72
IO depths : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=2.9%, 4=94.3%, 8=0.0%, 16=0.0%, 32=2.9%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=32/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
Disk stats (read/write):
sda: ios=0/37, merge=0/43, ticks=0/800, in_queue=856, util=94.87%
ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
-----Original Message-----
From: Venky Shankar [mailto:vshan...@redhat.com]
Sent: Thursday, December 08, 2016 4:05 PM
To: Aravind Ramesh <aravind.ram...@sandisk.com>
Cc: n...@fisk.me.uk; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RBD: Failed to map rbd device with data pool enabled.
On 16-12-08 09:32:02, Aravind Ramesh wrote:
> You can specify the -data-pool option while creating the rbd image.
> Example:
> rbd create rbdimg_EC1 --size 1024 --pool replicated_pool1 --data-pool
> ecpool Once the image is created, you can add the image name(rdbimg_EC1) and
> the replicated pool name(replicated_pool1) in the fio config file and set
> ioengine=rbd. Fio is supposed to do I/O on this new image, but I am seeing
> that it is failing for such devices. For normal replicated pool rbd-images it
> is working as expected.
What errors do you see with an EC data pool? try with "debug rbd = 20" to get
verbose logs.
>
> Aravind
>
> From: Nick Fisk [mailto:n...@fisk.me.uk]
> Sent: Thursday, December 08, 2016 1:46 PM
> To: Aravind Ramesh <aravind.ram...@sandisk.com>; n...@fisk.me.uk;
> ceph-users@lists.ceph.com
> Subject: RE: [ceph-users] RBD: Failed to map rbd device with data pool
> enabled.
>
> Fio has a direct RBD engine which uses librbd. I've just had a quick look at
> the code and I can't see an option in the latest fio to specify datapool, but
> I'm not sure if librbd handles this all behind the scenes. Might be worth a
> try.
>
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> Of Aravind Ramesh
> Sent: 07 December 2016 19:47
> To: n...@fisk.me.uk<mailto:n...@fisk.me.uk>;
> ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] RBD: Failed to map rbd device with data pool
> enabled.
>
> Thanks Nick,
>
> I have not tried using rbd-nbd, I will give it a try. Rbd mapping is failing
> for the image which was created with -data-pool <ec-pool> option, so I can't
> run fio or any IO on it.
>
> Aravind
>
> From: Nick Fisk [mailto:n...@fisk.me.uk]
> Sent: Wednesday, December 07, 2016 6:23 PM
> To: Aravind Ramesh
> <aravind.ram...@sandisk.com<mailto:aravind.ram...@sandisk.com>>;
> ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
> Subject: RE: RBD: Failed to map rbd device with data pool enabled.
>
> Hi Aravind,
>
> I've also seen this merge on Monday and tried to create a RBD on an ecpool
> and also failed. Although I ended up with all my OSD's crashing and refusing
> to restart. I'm going to rebuild the cluster and try again.
>
> Have you tried using the rbd-nbd driver or benchmarking directly with fio for
> the time being so you don't have to disable the image features?
>
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> Of Aravind Ramesh
> Sent: 07 December 2016 11:07
> To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
> Subject: [ceph-users] RBD: Failed to map rbd device with data pool enabled.
>
> Hi,
>
> I am seeing this failure when I try to map a rbd device with -data-pool set
> to a EC Pool. This is a newly merged feature, I am not sure if it is expected
> to work yet or I need to do something more.
> Same issue is seen while mapping a rbd image from replicated pool but after
> disabling the new features, I was able to map it and create a filesystem on
> it, but rbd devices created with the option -data-pool <ec pool), I am not
> able to disable the features also.
> =============================
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ ./bin/rbd create
> rbdimg_EC1 --size 1024 --pool rep_pool --data-pool aravecpool32
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
>
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ rbd ls
> rbdimg_EC1
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
>
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ sudo rbd map rbdimg_EC1
> rbd: sysfs write failed
> rbd: map failed: (6) No such device or address
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$
>
> ems@rack9-ems-5:~/ec-rbd/master/ceph/build$ rbd feature disable
> rbdimg_EC1 exclusive-lock deep-flatten fast-diff object-map
> 2016-12-07 16:19:42.710095 7f6594ff9700 -1
> librbd::image::RefreshRequest: Image uses unsupported features: 128
> 2016-12-07 16:19:42.710185 7f6587fff700 -1 librbd::image::OpenRequest:
> failed to refresh image: (38) Function not implemented
> 2016-12-07 16:19:42.785027 7f6587fff700 -1 librbd::ImageState: failed
> to open image: (38) Function not implemented
> rbd: error opening image rbdimg_EC1: (38) Function not implemented
> ems@rack9-ems-5:~/ ==================================
> PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If the
> reader of this message is not the intended recipient, you are hereby notified
> that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by telephone or e-mail (as shown above) immediately and destroy
> any and all copies of this message in your possession (whether hard copies or
> electronically stored copies).
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
