Re: [ceph-users] CephFS dropping data with rsync?

2018-06-15 Thread Hector Martin 
On 2018-06-16 13:04, Hector Martin wrote:
> I'm at a loss as to what happened here.

Okay, I just realized CephFS has a default 1TB file size... that
explains what triggered the problem. I just bumped it to 10TB. What that
doesn't explain is why rsync didn't complain about anything. Maybe when
ceph-fuse stumbles upon that limit, it just drops the file and ignores
all future I/O from a given client PID, instead of actively returning
I/O errors? If so, that sounds like a bug...

-- 
Hector Martin (hec...@marcansoft.com)
Public Key: https://mrcn.st/pub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS dropping data with rsync?

2018-06-15 Thread Hector Martin 
I'm at a loss as to what happened here.

I'm testing a single-node Ceph "cluster" as a replacement for RAID and
traditional filesystems. 9 4TB HDDs, one single (underpowered) server.
Running Luminous 12.2.5 with BlueStore OSDs.

I set up CephFS on a k=6,m=2 EC pool, mounted it via FUSE, and ran an
rsync from a traditional FS storing a few terabytes of data, including
some huge (>1TB) files. It all was going well, but then this happened:

https://mrcn.st/t/cephfs_dataloss.png

At around 9:13, throughput crashed to 0 and cluster usage dropped by
~1.3TB. This apparently happened while rsync was copying a large (1.1TB)
file. With EC overhead, the file would've been ~1.5TB of cluster
storage, so it seems this happened while the file was being copied, and
then whatever progress had been made was lost/deleted and the objects
dropped. rsync didn't print any errors, though, and kept on merrily
going throuh other files, but anything from this point onward never made
it to the cluster. It's as if everything rsync did might as well gone to
/dev/null from that point on. I didn't realize this until later, though.

At 9:26 I got an alert that the server was running out of RAM (not
unexpected, with the default BlueStore cache size), so I halved that and
restarted all OSDs, but I don't think this is related. By then the
cluster usage was already low and throughput was 0, so the problem had
already occurred. This was probably just a side-effect of all the object
deletion activity. This was a preemptive monitoring alert; nothing ever
actually OOMed or crashed, I restarted the OSDs and fixed the problem
before anything bad could happen.

At 12:41 I figured out something was wrong and straced the running rsync
and confirmed that it was definitely issuing write() syscalls, but the
cluster saw nothing - top showed only rsync consuming CPU (and maybe
ceph-fuse? I forget, possibly that too). All the work from that point
onwards seemed to just vanish into thin air. I checked the rsync verbose
output against the actual cephfs contents, and everything prior to the
1.1TB file was there, while everything after just wasn't (no files; some
empty directories, but I think rsync pre-creates those ahead of time).

I restarted rsync and it just started right back at that big file. The
cluster throughput is back up again so I assume writes are making it to
the cluster now. I have not restarted/remounted ceph-fuse.

I've checked syslog and all the Ceph log but I cannot find anything
relevant. The only evidence of the problem is increased RocksDB
compaction activity during the 9:13-9:28 interval when all the objects
for the huge file were apparently being removed. No errors, nothing in
syslog or dmesg, nothing in the ceph-fuse log
(/var/log/ceph/ceph-client.cephfs.log) either. There are no weird
cronjobs that might've done something strange. It's like the cephfs fuse
client decided to shadowban the rsync process without leaving any
evidence behind.

Any ideas what might've happened here? If this happens again / is
reproducible I'll try to see if I can do some more debugging...

-- 
Hector Martin (hec...@marcansoft.com)
Public Key: https://mrcn.st/pub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PM1633a

2018-06-15 Thread Paul Emmerich 
Hi,

we've evaluated them but they were worse than the SM863a in the usual quick
sync write IOPS benchmark.
Not saying that it's a bad disk (10k IOPS with one thread, ~20k with more
threads), we haven't run any long-term tests.

Paul


2018-06-15 21:02 GMT+02:00 Brian : :

> Hello List - anyone using these drives and have any good  / bad things
> to say about them?
>
> Thanks!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] PM1633a

2018-06-15 Thread Brian : 
Hello List - anyone using these drives and have any good  / bad things
to say about them?

Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI to a Ceph node with 2 network adapters - how to ?

2018-06-15 Thread Wladimir Mutel 

Jason Dillaman wrote:


[1] http://docs.ceph.com/docs/master/rbd/iscsi-initiator-win/



 I don't use either MPIO or MCS on Windows 2008 R2 or Windows 10
initiator (not Win2016 but hope there is no much difference). I try to make
it work with a single session first. Also, right now I only have a single
iSCSI gateway/portal (single host, single IP, single port).
Or is MPIO mandatory to use with Ceph target ?



It's mandatory even if you only have a single path since MPIO is
responsible for activating the paths.


Who would know ? I installed MPIO, enabled it for iSCSI
(required Windows reboot), set MPIO policy to 'Failover only',
and now my iSCSI target is readable !
Thanks a lot for your help !
Probably this should be written with bigger and redder letters
in Ceph docs.

Next question, would it be possible for iPXE loader to boot
from such iSCSI volumes ? I am going to experiment with that
but if the answer is known in advance, it would be good to know.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] move rbd image (with snapshots) to different pool

2018-06-15 Thread Jason Dillaman 
On Fri, Jun 15, 2018 at 12:15 PM, Steve Taylor
 wrote:
> I have done this with Luminous by deep-flattening a clone in a different 
> pool. It seemed to do what I wanted, but the RBD appeared to lose its 
> sparseness in the process.

Hmm, Luminous librbd clients should have kept object-sized sparseness
during the flatten operation [1] since it will not copy-up a zeroed,
object-sized block from the parent to the child image.  Jewel clients
would not preserve object-level sparseness, so a flatten operation
would result in a fully allocated image [2].

> Can anyone verify that and/or comment on whether Mimic's "rbd deep copy" does 
> the same?

The "rbd export --export-format 2" and "rbd deep copy" actions both
preserve the sparseness of the source image since they rely on RADOS
providing the -- not restricted to object-sized sparseness like a
copy-up operation.

> Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation
> 380 Data Drive Suite 300 |?Draper | Utah | 84020
> Office: 801.871.2799 |
>
> If you are not the intended recipient of this message or received it 
> erroneously, please notify the sender and delete it, together with any 
> attachments, and be advised that any dissemination or copying of this message 
> is prohibited.
>
>
>
> -Original Message-
> From: ceph-users  On Behalf Of Jason 
> Dillaman
> Sent: Friday, June 15, 2018 7:45 AM
> To: Marc Roos 
> Cc: ceph-users 
> Subject: Re: [ceph-users] move rbd image (with snapshots) to different pool
>
> The "rbd clone" command will just create a copy-on-write cloned child of the 
> source image. It will not copy any snapshots from the original image to the 
> clone.
>
> With the Luminous release, you can use "rbd export --export-format 2 
>  - | rbd import --export-format 2 - " to export / 
> import an image (and all its snapshots) to a different pool.
> Additionally, with the Mimic release, you can run "rbd deep copy" to copy an 
> image (and all its snapshots) to a different pool.
>
> On Fri, Jun 15, 2018 at 3:26 AM, Marc Roos  wrote:
>>
>> If I would like to copy/move an rbd image, this is the only option I
>> have? (Want to move an image from a hdd pool to an ssd pool)
>>
>> rbd clone mypool/parent@snap otherpool/child
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Jason
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[1] 
https://github.com/ceph/ceph/blob/luminous/src/test/cli-integration/rbd/formatted-output.t#L888
[2] 
https://github.com/ceph/ceph/blob/jewel/src/test/cli-integration/rbd/formatted-output.t#L855

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] move rbd image (with snapshots) to different pool

2018-06-15 Thread Steve Taylor 
I have done this with Luminous by deep-flattening a clone in a different pool. 
It seemed to do what I wanted, but the RBD appeared to lose its sparseness in 
the process. Can anyone verify that and/or comment on whether Mimic's "rbd deep 
copy" does the same?


 
Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation
380 Data Drive Suite 300 |?Draper | Utah | 84020
Office: 801.871.2799 | 
 
If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.

 

-Original Message-
From: ceph-users  On Behalf Of Jason Dillaman
Sent: Friday, June 15, 2018 7:45 AM
To: Marc Roos 
Cc: ceph-users 
Subject: Re: [ceph-users] move rbd image (with snapshots) to different pool

The "rbd clone" command will just create a copy-on-write cloned child of the 
source image. It will not copy any snapshots from the original image to the 
clone.

With the Luminous release, you can use "rbd export --export-format 2 
 - | rbd import --export-format 2 - " to export / 
import an image (and all its snapshots) to a different pool.
Additionally, with the Mimic release, you can run "rbd deep copy" to copy an 
image (and all its snapshots) to a different pool.

On Fri, Jun 15, 2018 at 3:26 AM, Marc Roos  wrote:
>
> If I would like to copy/move an rbd image, this is the only option I 
> have? (Want to move an image from a hdd pool to an ssd pool)
>
> rbd clone mypool/parent@snap otherpool/child
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSDs too slow to start

2018-06-15 Thread Alfredo Daniel Rezinovsky 

Too long is 120 seconds

The DB is in SSD devices. The devices are fast. The process OSD reads 
about 800Mb but I cannot be sure from where.



On 13/06/18 11:36, Gregory Farnum wrote:

How long is “too long”? 800MB on an SSD should only be a second or three.
I’m not sure if that’s a reasonable amount of data; you could try 
compacting the rocksdb instance etc. But if reading 800MB is 
noticeable I would start wondering about the quality of your disks as 
a journal or rocksdb device.

-Greg

On Tue, Jun 12, 2018 at 2:23 PM Alfredo Daniel Rezinovsky 
> wrote:


I migrated my OSDs from filestore to bluestore.

Each node now has 1 SSD with the OS and the BlockDBs and 3 HDDs with
bluestore data.

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdd  8:48   0   2.7T  0 disk
|-sdd2   8:50   0   2.7T  0 part
`-sdd1   8:49   0   100M  0 part /var/lib/ceph/osd/ceph-2
sdb  8:16   0   3.7T  0 disk
|-sdb2   8:18   0   3.7T  0 part
`-sdb1   8:17   0   100M  0 part /var/lib/ceph/osd/ceph-0
sdc  8:32   0   3.7T  0 disk
|-sdc2   8:34   0   3.7T  0 part
`-sdc1   8:33   0   100M  0 part /var/lib/ceph/osd/ceph-1
sda  8:0    0 223.6G  0 disk
|-sda4   8:4    0 1G  0 part
|-sda2   8:2    0  37.3G  0 part /
|-sda5   8:5    0 1G  0 part
|-sda3   8:3    0 1G  0 part
`-sda1   8:1    0   953M  0 part /boot/efi

Now the I/O works better, and I never saw again a slow response
(OSD not
MDS) warning.

But when I reboot a ceph node the OSDs takes too long to get up. With
filestore it was almost inmediate.

Monitoring /proc/$(pidod ceph-osd)/io I could see that each OSD reads
about 800 MBytes before getting up (My block.db partitions are 1G).

Does the OSDs start re-process all the block.db when booting up?

There's any way to accelerate the OSD availability after a reboot?


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MDS: journaler.pq decode error

2018-06-15 Thread John Spray 
On Fri, Jun 15, 2018 at 2:55 PM, Benjeman Meekhof  wrote:
> Have seen some posts and issue trackers related to this topic in the
> past but haven't been able to put it together to resolve the issue I'm
> having.  All on Luminous 12.2.5 (upgraded over time from past
> releases).  We are going to upgrade to Mimic near future if that would
> somehow resolve the issue.
>
> Summary:
>
> 1.  We have a CephFS data pool which has steadily and slowly grown in
> size without corresponding writes to the directory placed on it - a
> plot of usage over a few hours shows a very regular upward rate of
> increase.   The pool is now 300TB vs 16TB of actual space used in
> directory.
>
> 2.  Reading through some email posts and issue trackers led me to
> disabling 'standby replay' though we are not and have not ever used
> snapshots.   Disabling that feature on our 3 MDS stopped the steady
> climb.  However the pool remains with 300TB of unaccounted for space
> usage.  http://tracker.ceph.com/issues/19593 and
> http://tracker.ceph.com/issues/21551

This is pretty strange -- if you were already on 12.2.5 then the
http://tracker.ceph.com/issues/19593 should have been fixed and
switching standby replays on/off shouldn't make a difference (unless
there's some similar bug that crept back into luminous).

> 3.   I've never had any issue starting the MDS or with filesystem
> functionality but looking through the mds logs I see a single
> 'journaler.pg(rw) _decode error from assimilate_prefetch' at every
> startup.  A log snippet with context is below with debug_mds and
> debug_journaler at 20.

This message suggests that the purge queue has been corrupted, but the
MDS is ignoring this -- something is wrong with the error handling.
The MDS should be marked damaged when something like this happens, but
in this case PurgeQueue is apparently dropping the error on the floor
after it gets logged by Journaler.  I've opened a ticket+PR for the
error handling here: http://tracker.ceph.com/issues/24533 (however,
the loading path in PurgeQueue::_recover *does* have error handling so
I'm not clear why that isn't happening in your case).

I believe cephfs-journal-tool in mimic was enhanced to be able to
optionally operate on the purge queue as well as the metadata journal
(they use the same underlying format), so upgrading to mimic would
give you better tooling for debugging this.

John


> As noted, there is at least one past email thread on the topic but I'm
> not quite having the same issue as this person and I couldn't glean
> any information as to what I should do to repair this error and get
> stale objects purged from this pool (if that is in fact the issue):
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021379.html
>
> Any thoughts on troubleshooting steps I could try next?
>
> Here is the log snippet:
>
> 2018-06-15 09:14:50.746831 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.746835 7fb47251b700 10 mds.0.journaler.pq(rw)
> append_entry len 81 to 88121773~101
> 2018-06-15 09:14:50.746838 7fb47251b700 10 mds.0.journaler.pq(rw) _prefetch
> 2018-06-15 09:14:50.746863 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.746864 7fb47251b700 10 mds.0.journaler.pq(rw)
> append_entry len 81 to 88121874~101
> 2018-06-15 09:14:50.746867 7fb47251b700 10 mds.0.journaler.pq(rw) _prefetch
> 2018-06-15 09:14:50.746901 7fb46fd16700 10 mds.0.journaler.pq(rw)
> _finish_read got 6822392~1566216
> 2018-06-15 09:14:50.746909 7fb46fd16700 10 mds.0.journaler.pq(rw)
> _assimilate_prefetch 6822392~1566216
> 2018-06-15 09:14:50.746911 7fb46fd16700 10 mds.0.journaler.pq(rw)
> _assimilate_prefetch gap of 4194304 from received_pos 8388608 to first
> prefetched buffer 12582912
> 2018-06-15 09:14:50.746913 7fb46fd16700 10 mds.0.journaler.pq(rw)
> _assimilate_prefetch read_buf now 6822392~1566216, read pointers
> 6822392/8388608/50331648
>
> === error here ===> 2018-06-15 09:14:50.746965 7fb46fd16700 -1
> mds.0.journaler.pq(rw) _decode error from assimilate_prefetch
>
> 2018-06-15 09:14:50.746994 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.746998 7fb47251b700 10 mds.0.journaler.pq(rw)
> append_entry len 81 to 88121975~101
> 2018-06-15 09:14:50.747007 7fb47251b700 10 mds.0.journaler.pq(rw)
> wait_for_readable at 6822392 onreadable 0x557ee0f58300
> 2018-06-15 09:14:50.747042 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.747043 7fb47251b700 10 mds.0.journaler.pq(rw)
> append_entry len 81 to 88122076~101
> 2018-06-15 09:14:50.747063 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.747064 7fb47251b700 10 mds.0.journaler.pq(rw)
> append_entry len 81 to 88122177~101
> 2018-06-15 09:14:50.747113 7fb47251b700 20 mds.0.journaler.pq(rw)
> write_buf_throttle get, delta 101
> 2018-06-15 09:14:50.747114 7fb47251b700

[ceph-users] MDS: journaler.pq decode error

2018-06-15 Thread Benjeman Meekhof 
Have seen some posts and issue trackers related to this topic in the
past but haven't been able to put it together to resolve the issue I'm
having.  All on Luminous 12.2.5 (upgraded over time from past
releases).  We are going to upgrade to Mimic near future if that would
somehow resolve the issue.

Summary:

1.  We have a CephFS data pool which has steadily and slowly grown in
size without corresponding writes to the directory placed on it - a
plot of usage over a few hours shows a very regular upward rate of
increase.   The pool is now 300TB vs 16TB of actual space used in
directory.

2.  Reading through some email posts and issue trackers led me to
disabling 'standby replay' though we are not and have not ever used
snapshots.   Disabling that feature on our 3 MDS stopped the steady
climb.  However the pool remains with 300TB of unaccounted for space
usage.  http://tracker.ceph.com/issues/19593 and
http://tracker.ceph.com/issues/21551

3.   I've never had any issue starting the MDS or with filesystem
functionality but looking through the mds logs I see a single
'journaler.pg(rw) _decode error from assimilate_prefetch' at every
startup.  A log snippet with context is below with debug_mds and
debug_journaler at 20.

As noted, there is at least one past email thread on the topic but I'm
not quite having the same issue as this person and I couldn't glean
any information as to what I should do to repair this error and get
stale objects purged from this pool (if that is in fact the issue):
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021379.html

Any thoughts on troubleshooting steps I could try next?

Here is the log snippet:

2018-06-15 09:14:50.746831 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.746835 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88121773~101
2018-06-15 09:14:50.746838 7fb47251b700 10 mds.0.journaler.pq(rw) _prefetch
2018-06-15 09:14:50.746863 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.746864 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88121874~101
2018-06-15 09:14:50.746867 7fb47251b700 10 mds.0.journaler.pq(rw) _prefetch
2018-06-15 09:14:50.746901 7fb46fd16700 10 mds.0.journaler.pq(rw)
_finish_read got 6822392~1566216
2018-06-15 09:14:50.746909 7fb46fd16700 10 mds.0.journaler.pq(rw)
_assimilate_prefetch 6822392~1566216
2018-06-15 09:14:50.746911 7fb46fd16700 10 mds.0.journaler.pq(rw)
_assimilate_prefetch gap of 4194304 from received_pos 8388608 to first
prefetched buffer 12582912
2018-06-15 09:14:50.746913 7fb46fd16700 10 mds.0.journaler.pq(rw)
_assimilate_prefetch read_buf now 6822392~1566216, read pointers
6822392/8388608/50331648

=== error here ===> 2018-06-15 09:14:50.746965 7fb46fd16700 -1
mds.0.journaler.pq(rw) _decode error from assimilate_prefetch

2018-06-15 09:14:50.746994 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.746998 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88121975~101
2018-06-15 09:14:50.747007 7fb47251b700 10 mds.0.journaler.pq(rw)
wait_for_readable at 6822392 onreadable 0x557ee0f58300
2018-06-15 09:14:50.747042 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.747043 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88122076~101
2018-06-15 09:14:50.747063 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.747064 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88122177~101
2018-06-15 09:14:50.747113 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
2018-06-15 09:14:50.747114 7fb47251b700 10 mds.0.journaler.pq(rw)
append_entry len 81 to 88122278~101
2018-06-15 09:14:50.747136 7fb47251b700 20 mds.0.journaler.pq(rw)
write_buf_throttle get, delta 101
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI to a Ceph node with 2 network adapters - how to ?

2018-06-15 Thread Jason Dillaman 
On Fri, Jun 15, 2018 at 6:19 AM, Wladimir Mutel  wrote:
> Jason Dillaman wrote:
>
>>> чер 13 08:38:14 p10s tcmu-runner[54121]: 2018-06-13 08:38:14.726 54121
>>> [DEBUG] dbus_name_acquired:461: name org.kernel.TCMUService1 acquired
>>> чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.521 54121
>>> [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 1a
>>> 0 3f 0 c0 0
>>> чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.523 54121
>>> [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 1a
>>> 0 3f 0 c0 0
>>> чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.543 54121
>>> [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e
>>> 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0
>>> чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.550 54121
>>> [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e
>>> 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0
>>> чер 13 08:38:47 p10s tcmu-runner[54121]: 2018-06-13 08:38:47.944 54121
>>> [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e
>>> 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0
>
>
>>>  Wikipedia says that 1A is 'mode sense' and 9e is 'service action
>>> in'.
>>>  These records are logged when I try to put the disk online or
>>> initialize it with GPT/MBR partition table in Windows Disk Management (and
>>> Windows report errors after that)
>>>  What to check next ? Any importance of missing 'SSD' device
>>> class ?
>
>
>> Did you configure MPIO within Windows [1]? Any errors recorded in the
>> Windows Event Viewer?
>
>
>> The "SSD" device class isn't important -- it's just a way to describe
>> the LUN as being backed by non-rotational media (e.g. VMware will show
>> a different icon).
>
>
>> [1] http://docs.ceph.com/docs/master/rbd/iscsi-initiator-win/
>
>
> I don't use either MPIO or MCS on Windows 2008 R2 or Windows 10
> initiator (not Win2016 but hope there is no much difference). I try to make
> it work with a single session first. Also, right now I only have a single
> iSCSI gateway/portal (single host, single IP, single port).
> Or is MPIO mandatory to use with Ceph target ?

It's mandatory even if you only have a single path since MPIO is
responsible for activating the paths.

>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] move rbd image (with snapshots) to different pool

2018-06-15 Thread Jason Dillaman 
The "rbd clone" command will just create a copy-on-write cloned child
of the source image. It will not copy any snapshots from the original
image to the clone.

With the Luminous release, you can use "rbd export --export-format 2
 - | rbd import --export-format 2 - " to
export / import an image (and all its snapshots) to a different pool.
Additionally, with the Mimic release, you can run "rbd deep copy" to
copy an image (and all its snapshots) to a different pool.

On Fri, Jun 15, 2018 at 3:26 AM, Marc Roos  wrote:
>
> If I would like to copy/move an rbd image, this is the only option I
> have? (Want to move an image from a hdd pool to an ssd pool)
>
> rbd clone mypool/parent@snap otherpool/child
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] move rbd image (with snapshots) to different pool

2018-06-15 Thread Marc Roos 


If I would like to copy/move an rbd image, this is the only option I 
have? (Want to move an image from a hdd pool to an ssd pool)

rbd clone mypool/parent@snap otherpool/child


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RGW Dynamic bucket index resharding keeps resharding all buckets

2018-06-15 Thread Sander van Schie / True 
Hello,

We're into some problems with dynamic bucket index resharding. After an upgrade 
from Ceph 12.2.2 to 12.2.5, which fixed an issue with the resharding when using 
tenants (which we do), the cluster was busy resharding for 2 days straight, 
resharding the same buckets over and over again.

After disabling it and re-enabling it a while later, it resharded all buckets 
again and then kept quiet for a bit. Later on it started resharding buckets 
over and over again, even buckets which didn't have any data added in the 
meantime. In the reshard list it always says 'old_num_shards: 1' for every 
bucket, even though I can confirm with 'bucket stats' there's already the 
desired amount of shards present. It looks like the background process which 
scans buckets doesn't properly recognize the amount of shards a bucket 
currently has. When I manually add a reshard job, it does properly recognize 
the current amount of shards.

On a sidenote, we had two buckets in the reshard list which were removed a long 
while ago. We were unable to cancel the reshard job for those buckets. After 
recreating the users and buckets we were able to remove them from the list 
though, so they are no longer present. Probably not relevant, but you never 
know.

Are we missing something, or are we running into a bug?

Thanks,

Sander
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI to a Ceph node with 2 network adapters - how to ?

2018-06-15 Thread Wladimir Mutel 

Jason Dillaman wrote:


чер 13 08:38:14 p10s tcmu-runner[54121]: 2018-06-13 08:38:14.726 54121 [DEBUG] 
dbus_name_acquired:461: name org.kernel.TCMUService1 acquired
чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.521 54121 
[DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 1a 0 
3f 0 c0 0
чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.523 54121 
[DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 1a 0 
3f 0 c0 0
чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.543 54121 
[DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e 10 
0 0 0 0 0 0 0 0 0 0 0 c 0 0
чер 13 08:38:30 p10s tcmu-runner[54121]: 2018-06-13 08:38:30.550 54121 
[DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e 10 
0 0 0 0 0 0 0 0 0 0 0 c 0 0
чер 13 08:38:47 p10s tcmu-runner[54121]: 2018-06-13 08:38:47.944 54121 
[DEBUG_SCSI_CMD] tcmu_print_cdb_info:1205 rbd/libvirt.tower-prime-e-3tb: 9e 10 
0 0 0 0 0 0 0 0 0 0 0 c 0 0



 Wikipedia says that 1A is 'mode sense' and 9e is 'service action in'.
 These records are logged when I try to put the disk online or 
initialize it with GPT/MBR partition table in Windows Disk Management (and 
Windows report errors after that)
 What to check next ? Any importance of missing 'SSD' device class ?



Did you configure MPIO within Windows [1]? Any errors recorded in the
Windows Event Viewer?



The "SSD" device class isn't important -- it's just a way to describe
the LUN as being backed by non-rotational media (e.g. VMware will show
a different icon).



[1] http://docs.ceph.com/docs/master/rbd/iscsi-initiator-win/


	I don't use either MPIO or MCS on Windows 2008 R2 or Windows 10 
initiator (not Win2016 but hope there is no much difference). I try to 
make it work with a single session first. Also, right now I only have a 
single iSCSI gateway/portal (single host, single IP, single port).

Or is MPIO mandatory to use with Ceph target ?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_op_threads appears to be removed from the settings

2018-06-15 Thread Matthew Stroud 
Thanks for the info

From: Piotr Dalek 
Date: Friday, June 15, 2018 at 12:33 AM
To: Matthew Stroud , ceph-users 

Subject: RE: osd_op_threads appears to be removed from the settings

No, it’s no longer valid.

--
Piotr Dałek
piotr.da...@corp.ovh.com
https://ovhcloud.com/

From: ceph-users  On Behalf Of Matthew Stroud
Sent: Friday, June 15, 2018 8:11 AM
To: ceph-users 
Subject: [ceph-users] osd_op_threads appears to be removed from the settings

So I’m trying to update the osd_op_threads setting that was in jewel, that now 
doesn’t appear to be in luminous. What’s more confusing is that the docs state 
that is a valid option. Is osd_op_threads still valid?

I’m currently running ceph 12.2.2

Thanks,
Matthew Stroud



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_op_threads appears to be removed from the settings

2018-06-15 Thread Piotr Dalek 
No, it’s no longer valid.

--
Piotr Dałek
piotr.da...@corp.ovh.com
https://ovhcloud.com/

From: ceph-users  On Behalf Of Matthew Stroud
Sent: Friday, June 15, 2018 8:11 AM
To: ceph-users 
Subject: [ceph-users] osd_op_threads appears to be removed from the settings

So I’m trying to update the osd_op_threads setting that was in jewel, that now 
doesn’t appear to be in luminous. What’s more confusing is that the docs state 
that is a valid option. Is osd_op_threads still valid?

I’m currently running ceph 12.2.2

Thanks,
Matthew Stroud



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd_op_threads appears to be removed from the settings

2018-06-15 Thread Matthew Stroud 
So I’m trying to update the osd_op_threads setting that was in jewel, that now 
doesn’t appear to be in luminous. What’s more confusing is that the docs state 
that is a valid option. Is osd_op_threads still valid?

I’m currently running ceph 12.2.2

Thanks,
Matthew Stroud



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com