Hi Ganesha developers, I've been playing with the RGW FSAL (ganesha-2.4.1) on a fairly recent Jewel-based (10.2.3-493-gb4c314a) Ceph cluster and wanted to share some observations.
In a read-only scenario, the FSAL appears to function well! Great work!
However, I am experiencing some problems when attempting writes. I am curious
if anyone else is testing writes with the RGW FSAL and if the below are already
known issues? If not, I'd be happy to help debug further.
Thanks in advance!
0. Bucket/Object creation via S3
No issues on the S3 side, however my NFS mount will not see the new buckets
or objects until ganesha is restarted.
1. `echo` or `touch` a new file
File creation via `echo` or `touch` (createmode=1, offset=0, length=N) has
so far been successful in my test environment.
2. Append to an existing file using `echo`
On an append via `echo`, the behaviour is not consistent.
i) offset=0, length=N
- A full overwrite via `echo` has shown to be successful.
ii) offset=M, length=(N - M)
- The librgw:rgw_write() code path ultimately rejects this write and
(-5) bubbles back up to ganesha. This makes sense as RGW objects need
to be completely overwritten.
In the case of (2ii), I've seen inconsistency between the S3 object and the
NFS representation. The file will have an updated size as if the append
succeeded, however S3 shows that the object has not been modified. Trying to
then read the file from NFS would generate I/O errors. At this point, a restart
of ganesha was needed.
Also, it isn't clear to me why nfs4_write() interprets one `echo` invocation as
an overwrite, and another identical `echo` as an append? Given the rgw_write()
offset limitations, it seems file modification/appends are not yet well
supported?
Log snippets of (2i) and (2ii) are below.
3. Create a new file using an editor (vi, pico)
Editors are problematic.
New files (even with content) end up as empty objects on the S3 side. vi
and pico fail in slightly different ways. Below is a log snippet caputred
while using pico, showing a double write being made as a result of a single
user induced save operation on the editor. The resulting S3 object contains
no data.
4. Append to an existing file using an editor (vi, pico)
Using vi generates a backup file and object (foo~) with the original contents.
The original object is empty, and again there is a similar inconsistency as
in 2ii. pico fails as well, but leaves the original object intact. Again,
a restart of ganesha is needed at this point.
LOGS:
-----
2i)
...
fsal_open2 :FSAL :F_DBG :KM fsal_open2(): name=(null) openflags=2 createmode=0
fsal_open2 :FSAL :F_DBG :KM fsal_open2():1855 name=(null) openflags=2
createmode=0
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=(null) openflags=2
posix_flags=0 createmode=0
rgw_fsal_open2 :FSAL :F_DBG :rgw_fsal_open2 enter obj_hdl 0x7feb58010650 state
(nil)
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=(null) openflags=2
posix_flags=1 createmode=0
rgw_fsal_open2 :RW LOCK :F_DBG :Got write lock on 0x7feb58010860
(&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:785
rgw_fsal_open2 :RW LOCK :F_DBG :Unlocked 0x7feb58010860 (&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:802
nfs4_write :NFS4 :F_DBG :offset = 0 length = 16 stable = 2
nfs4_write :NFS4 :F_DBG :offset = 0 length = 16
fsal_write2 :FSAL :F_DBG :KM fsal_write2()
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2() openflags=2
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 enter obj_hdl 0x7feb58010650
state 0x7feb6c051970
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2():1361 openflags=6
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 post obj_hdl 0x7feb58010650 state
0x7feb6c051970 returned 0
fsal_write2 :FSAL :F_DBG :inode/direct: io_size=16, bytes_moved=16, offset=0
nfs4_op_commit :NFS4 :F_DBG :Commit order over offset = 0, size = 0
rgw_fsal_commit2 :FSAL :F_DBG :rgw_fsal_commit2 enter obj_hdl 0x7feb58010650
offset 0 length 0
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 enter obj_hdl 0x7feb58010650
state 0x7feb6c051970
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 called w/open_state 0x7feb6c051970
rgw_fsal_close2 :RW LOCK :F_DBG :Got write lock on 0x7feb58010860
(&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1481
rgw_fsal_close2 :RW LOCK :F_DBG :Unlocked 0x7feb58010860 (&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1487
...
2ii)
...
fsal_open2 :FSAL :F_DBG :KM fsal_open2(): name=(null) openflags=2 createmode=0
fsal_open2 :FSAL :F_DBG :KM fsal_open2():1855 name=(null) openflags=2
createmode=0
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=(null) openflags=2
posix_flags=0 createmode=0
rgw_fsal_open2 :FSAL :F_DBG :rgw_fsal_open2 enter obj_hdl 0x7feb58010650 state
(nil)
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=(null) openflags=2
posix_flags=1 createmode=0
rgw_fsal_open2 :RW LOCK :F_DBG :Got write lock on 0x7feb58010860
(&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:785
rgw_fsal_open2 :RW LOCK :F_DBG :Unlocked 0x7feb58010860 (&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:802
nfs4_write :NFS4 :F_DBG :offset = 8 length = 8 stable = 2
nfs4_write :NFS4 :F_DBG :offset = 8 length = 8
fsal_write2 :FSAL :F_DBG :KM fsal_write2()
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2() openflags=2
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 enter obj_hdl 0x7feb58010650
state 0x7feb5c11c080
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2():1361 openflags=6
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 post obj_hdl 0x7feb58010650 state
0x7feb5c11c080 returned -5
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 enter obj_hdl 0x7feb58010650
state 0x7feb5c11c080
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 called w/open_state 0x7feb5c11c080
rgw_fsal_close2 :RW LOCK :F_DBG :Got write lock on 0x7feb58010860
(&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1481
rgw_fsal_close2 :RW LOCK :F_DBG :Unlocked 0x7feb58010860 (&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1487
...
3)
...
fsal_open2 :FSAL :F_DBG :KM fsal_open2(): name=foobar-create-via-pico
openflags=2 createmode=1
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=foobar-create-via-pico
openflags=2 posix_flags=0 createmode=1
rgw_fsal_open2 :FSAL :F_DBG :rgw_fsal_open2 enter obj_hdl 0x20abb90 state (nil)
rgw_fsal_open2 :FSAL :F_DBG :attrs set attributes Mask = 00000080 mode=0644
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=foobar-create-via-pico
openflags=2 posix_flags=1 createmode=1
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=foobar-create-via-pico
openflags=2 posix_flags=1 createmode=1
rgw_fsal_open2 :FSAL :F_DBG :KM rgw_fsal_open2(): name=foobar-create-via-pico
openflags=2 posix_flags=193 createmode=1
rgw_fsal_open2 :RW LOCK :F_DBG :Got write lock on 0x7f46b0012ea0
(&(*new_obj)->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1097
rgw_fsal_open2 :RW LOCK :F_DBG :Unlocked 0x7f46b0012ea0 (&(*new_obj)->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1102
nfs4_write :NFS4 :F_DBG :offset = 0 length = 18 stable = 2
nfs4_write :NFS4 :F_DBG :offset = 0 length = 18
fsal_write2 :FSAL :F_DBG :KM fsal_write2()
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2() openflags=2
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 enter obj_hdl 0x7f46b0012c90
state 0x7f46b0001c10
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2():1361 openflags=6
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 post obj_hdl 0x7f46b0012c90 state
0x7f46b0001c10 returned 0
fsal_write2 :FSAL :F_DBG :inode/direct: io_size=18, bytes_moved=18, offset=0
nfs4_op_commit :NFS4 :F_DBG :Commit order over offset = 0, size = 0
rgw_fsal_commit2 :FSAL :F_DBG :rgw_fsal_commit2 enter obj_hdl 0x7f46b0012c90
offset 0 length 0
nfs4_write :NFS4 :F_DBG :offset = 0 length = 18 stable = 2
nfs4_write :NFS4 :F_DBG :offset = 0 length = 18
fsal_write2 :FSAL :F_DBG :KM fsal_write2()
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2() openflags=2
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 enter obj_hdl 0x7f46b0012c90
state 0x7f46b0001c10
rgw_fsal_write2 :FSAL :F_DBG :KM rgw_fsal_write2():1361 openflags=6
rgw_fsal_write2 :FSAL :F_DBG :rgw_fsal_write2 post obj_hdl 0x7f46b0012c90 state
0x7f46b0001c10 returned -5
rgw_fsal_setattr2 :FSAL :F_DBG :rgw_fsal_setattr2 enter obj_hdl 0x7f46b0012c90
state (nil)
rgw_fsal_setattr2 :FSAL :F_DBG :attrs set attributes Mask = 00400000
mtime=SERVER
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 enter obj_hdl 0x7f46b0012c90
state 0x7f46b0001c10
rgw_fsal_close2 :FSAL :F_DBG :rgw_fsal_close2 called w/open_state 0x7f46b0001c10
rgw_fsal_close2 :RW LOCK :F_DBG :Got write lock on 0x7f46b0012ea0
(&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1481
rgw_fsal_close2 :RW LOCK :F_DBG :Unlocked 0x7f46b0012ea0 (&obj_hdl->lock) at
/home/abuild/rpmbuild/BUILD/nfs-ganesha-2.4.1+git.1479343782.64abb87/FSAL/FSAL_RGW/handle.c:1487
...
--
Regards,
Karol
signature.asc
Description: Digital signature
------------------------------------------------------------------------------
_______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
