Re: [ceph-users] rbd cache did not help improve performance

[Tom Christensen]

If you are mapping the RBD with the kernel driver then you're not using
librbd so these settings will have no effect I believe.  The kernel driver
does its own caching but I don't believe there are any settings to change
its default behavior.


On Mon, Feb 29, 2016 at 9:36 PM, Shinobu Kinjo  wrote:

> You may want to set "ioengine=rbd", I guess.
>
> Cheers,
>
> - Original Message -
> From: "min fang" 
> To: "ceph-users" 
> Sent: Tuesday, March 1, 2016 1:28:54 PM
> Subject: [ceph-users]  rbd cache did not help improve performance
>
> Hi, I set the following parameters in ceph.conf
>
> [client]
> rbd cache=true
> rbd cache size= 25769803776
> rbd readahead disable after byte=0
>
>
> map a rbd image to a rbd device then run fio testing on 4k read as the
> command
> ./fio -filename=/dev/rbd4 -direct=1 -iodepth 64 -thread -rw=read
> -ioengine=aio -bs=4K -size=500G -numjobs=32 -runtime=300 -group_reporting
> -name=mytest2
>
> Compared the result with setting rbd cache=false and enable cache model, I
> did not see performance improved by librbd cache.
>
> Is my setting not right, or it is true that ceph librbd cache will not
> have benefit on 4k seq read?
>
> thanks.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Replacing OSD drive without rempaping pg's

[Lindsay Mathieson]

I was looking at replacing an osd drive in place as per the procedure here:

http://www.spinics.net/lists/ceph-users/msg05959.html

   "If you are going to replace the drive immediately, set the “noout”
   flag.  Take the OSD “down” and replace drive.  Assuming it is
   mounted in the same place as the bad drive, bring the OSD back up.
 This will replicate exactly the same PGs the bad drive held back
   to the replacement drive."



But the new drive mount will be blank - what happens with the journal, 
keyring etc? does starting the OSD process recreate them automatically?



thanks,

--
Lindsay Mathieson

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd cache did not help improve performance

[Shinobu Kinjo]

You may want to set "ioengine=rbd", I guess.

Cheers,

- Original Message -
From: "min fang" 
To: "ceph-users" 
Sent: Tuesday, March 1, 2016 1:28:54 PM
Subject: [ceph-users]  rbd cache did not help improve performance

Hi, I set the following parameters in ceph.conf 

[client] 
rbd cache=true 
rbd cache size= 25769803776 
rbd readahead disable after byte=0 


map a rbd image to a rbd device then run fio testing on 4k read as the command 
./fio -filename=/dev/rbd4 -direct=1 -iodepth 64 -thread -rw=read -ioengine=aio 
-bs=4K -size=500G -numjobs=32 -runtime=300 -group_reporting -name=mytest2 

Compared the result with setting rbd cache=false and enable cache model, I did 
not see performance improved by librbd cache. 

Is my setting not right, or it is true that ceph librbd cache will not have 
benefit on 4k seq read? 

thanks. 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rbd cache did not help improve performance

[min fang]

Hi, I set the following parameters in ceph.conf

[client]
rbd cache=true
rbd cache size= 25769803776
rbd readahead disable after byte=0


map a rbd image to a rbd device then run fio testing on 4k read as the
command
./fio -filename=/dev/rbd4 -direct=1 -iodepth 64 -thread -rw=read
-ioengine=aio -bs=4K -size=500G -numjobs=32 -runtime=300 -group_reporting
-name=mytest2

Compared the result with setting rbd cache=false and enable cache model, I
did not see performance improved by librbd cache.

Is my setting not right, or it is true that ceph librbd cache will not have
benefit on 4k seq read?

thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cannot mount cephfs after some disaster recovery

[10000]

Hi,
I meet a trouble on mount the cephfs after doing some disaster recovery 
introducing by official 
document(http://docs.ceph.com/docs/master/cephfs/disaster-recovery).
Now when I try to mount the cephfs, I get "mount error 5 = Input/output 
error". 
When run "ceph -s" on clusters, it print like this:
 cluster 15935dde-1d19-486e-9e1c-67414f9927f6
 health HEALTH_OK
 monmap e1: 4 mons at 
{HK-IDC1-10-1-72-151=172.17.17.151:6789/0,HK-IDC1-10-1-72-152=172.17.17.152:6789/0,HK-IDC1-10-1-72-153=172.17.17.153:6789/0,HK-IDC1-10-1-72-160=10.1.72.160:6789/0}
election epoch 528, quorum 0,1,2,3 
HK-IDC1-10-1-72-160,HK-IDC1-10-1-72-151,HK-IDC1-10-1-72-152,HK-IDC1-10-1-72-153
 mdsmap e21038: 1/1/0 up {0=HK-IDC1-10-1-72-160=up:active}
 osdmap e10536: 108 osds: 108 up, 108 in
flags sortbitwise
  pgmap v424957: 6564 pgs, 3 pools, 3863 GB data, 67643 kobjects
8726 GB used, 181 TB / 189 TB avail
6560 active+clean
   3 active+clean+scrubbing+deep
   1 active+clean+scrubbing
 
 It seems there should be "1/1/1 up" at mdsmap instead of "1/1/0 up" and I 
really don't know what the last number mean.
 And there is cephfs if I run "ceph fs ls" which print this:
 
 name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
 
 I try my best to Google such problem however i get nothing. And I still 
want to know if i can bring the cephfs back. So does any one have ideas?


 Oh, I do the disaster recovery because I get "mdsmap e21012: 0/1/1 up, 1 
up:standby, 1 damaged" at first. And to bring the fs back to work, I do 
"JOURNAL TRUNCATION", "MDS TABLE WIPES", "MDS MAP RESET". However I think there 
must exist (and most) files that their metadata have been saved at OSDs 
(metadata pool, in RADOS). I just want to get them.
 
  Thanks.


Yingdi Guo___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hammer : rbd info/Status : operation not supported (95) (EC+RBD tier pools)

[Christian Balzer]

Hello,

just to close this up as far as my upgrade is concerned. 

I choose not to go for 0.94.6 at this point in time, too fresh off the
boat for my taste.

Phasing in the cache tier literally caused 10 Minutes of Terror (tm
pending), including the first ever "wrongly marked down" OSDs I've seen in
production. 
However after the 8000 hottest objects had been promoted, it stabilized
quickly. 
After 3 days we're now at 84k objects in the cache pool, 13% of its
capacity. 
Of those I'd consider about 50k to be hot or warm, the rest are from VM
installs and reboot and thus mostly cold.

Activity on the base pool (promotions) is nearly 0, so I'm taking this
golden opportunity to finally add the new HDD based storage node and its
OSDs. ^o^

Regards,

Christian

On Wed, 24 Feb 2016 22:44:47 -0700 Robert LeBlanc wrote:

> We have not seen this issue, but we don't run EC pools yet (we are
> waiting for multiple layers to be available). We are not running 0.94.6
> in production yet either. We have adopted the policy to only run released
> versions in production unless there is a really pressing need to have a
> patch. We are running 0.94.6 through our alpha and staging clusters and
> hoping to do the upgrade in the next couple of weeks. We won't know how
> much the recency fix will help until then because we have not been able
> to replicate our workload with fio accurately enough to get good test
> results. Unfortunately we will probably be swapping out our M600s with
> S3610s. We've burned through 30% of the life in 2 months and they have
> 8x the op latency. Due to the 10 Minutes of Terror, we are going to have
> to do both at the same time to reduce the impact. Luckily, when you have
> weighted out OSDs or empty ones, it is much less impactful. If you get
> your upgrade done before ours, I'd like to know how it went. I'll be
> posting the results from ours when it is done.
> 
> Sent from a mobile device, please excuse any typos.
> On Feb 24, 2016 5:43 PM, "Christian Balzer"  wrote:
> 
> >
> > Hello Jason (Ceph devs et al),
> >
> > On Wed, 24 Feb 2016 13:15:34 -0500 (EST) Jason Dillaman wrote:
> >
> > > If you run "rados -p  ls | grep "rbd_id." and
> > > don't see that object, you are experiencing that issue [1].
> > >
> > > You can attempt to work around this issue by running "rados -p
> > > irfu-virt setomapval rbd_id. dummy value" to
> > > force-promote the object to the cache pool.  I haven't tested /
> > > verified that will alleviate the issue, though.
> > >
> > > [1] http://tracker.ceph.com/issues/14762
> > >
> >
> > This concerns me greatly, as I'm about to phase in a cache tier this
> > weekend into a very busy, VERY mission critical Ceph cluster.
> > That is on top of a replicated pool, Hammer.
> >
> > That issue and the related git blurb are less than crystal clear, so
> > for my and everybody else's benefit could you elaborate a bit more on
> > this?
> >
> > 1. Does this only affect EC base pools?
> > 2. Is this a regressions of sorts and when came it about?
> >I have a hard time imagining people not running into this earlier,
> >unless that problem is very hard to trigger.
> > 3. One assumes that this isn't fixed in any released version of Ceph,
> >correct?
> >
> > Robert, sorry for CC'ing you, but AFAICT your cluster is about the
> > closest approximation in terms of busyness to mine here.
> > And I a assume that you're neither using EC pools (since you need
> > performance, not space) and haven't experienced this bug all?
> >
> > Also, would you consider the benefits of the recency fix (thanks for
> > that) being worth risk of being an early adopter of 0.94.6?
> > In other words, are you eating your own dog food already and 0.94.6
> > hasn't eaten your data babies yet? ^o^
> >
> > Regards,
> >
> > Christian
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com   Global OnLine Japan/Rakuten Communications
> > http://www.gol.com/
> >


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] User Interface

[Vlad Blando]

Hi,

We already have a user interface that is admin facing (ex. calamari,
kraken, ceph-dash), how about a client facing interface, that can cater for
both block and object store. For object store I can use Swift via Horizon
dashboard, but for block-store, I'm not sure how.

Thanks.


/Vlad
ᐧ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd suddenly down / connect claims to be / heartbeat_check: no reply

[Christian Balzer]

Hello,

googling for "ceph wrong node" gives us this insightful thread:
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg09960.html

I suggest reading through it, more below:

On Mon, 29 Feb 2016 15:30:41 +0100 Oliver Dzombic wrote:

> Hi,
> 
> i face here some trouble with the cluster.
> 
> Suddenly "random" OSD's are getting marked out.
> 
> After restarting the OSD on the specific node, its working again.
> 
Matches the scenario mentioned above.

> This happens usually during activated scrubbing/deep scrubbing.
>
I guess your cluster is very much overloaded on some level, use atop or
similar tools to find out what needs improvement.

Also, as always, versions of all SW/kernel, a HW description, output of
"ceph -s" etc. will help people identify possible problem spots or to
correlate this to other things.
 
Christian

> In the logs i can see:
> 
> 2016-02-29 06:08:58.130376 7fd5dae75700  0 -- 10.0.1.2:0/36459 >>
> 10.0.0.4:6807/9051245 pipe(0x27488000 sd=58 :60473 s=1 pgs=0 cs=0 l=1
> c=0x28b39440).connect claims to be 10.0.0.4:6807/12051245 not
> 10.0.0.4:6807/9051245 - wrong node!
> 2016-02-29 06:08:58.130417 7fd5d9961700  0 -- 10.0.1.2:0/36459 >>
> 10.0.1.4:6803/6002429 pipe(0x2a6c9000 sd=75 :37736 s=1 pgs=0 cs=0 l=1
> c=0x2420be40).connect claims to be 10.0.1.4:6803/10002429 not
> 10.0.1.4:6803/6002429 - wrong node!
> 2016-02-29 06:08:58.130918 7fd5b1c17700  0 -- 10.0.1.2:0/36459 >>
> 10.0.0.1:6800/8050402 pipe(0x26834000 sd=74 :37605 s=1 pgs=0 cs=0 l=1
> c=0x1f7a9020).connect claims to be 10.0.0.1:6800/9050770 not
> 10.0.0.1:6800/8050402 - wrong node!
> 2016-02-29 06:08:58.131266 7fd5be141700  0 -- 10.0.1.2:0/36459 >>
> 10.0.0.3:6806/9059302 pipe(0x27f07000 sd=76 :48347 s=1 pgs=0 cs=0 l=1
> c=0x2371adc0).connect claims to be 10.0.0.3:6806/11059302 not
> 10.0.0.3:6806/9059302 - wrong node!
> 2016-02-29 06:08:58.131299 7fd5c1914700  0 -- 10.0.1.2:0/36459 >>
> 10.0.1.4:6801/9051245 pipe(0x2d288000 sd=100 :33848 s=1 pgs=0 cs=0 l=1
> c=0x28b37760).connect claims to be 10.0.1.4:6801/12051245 not
> 10.0.1.4:6801/9051245 - wrong node!
> 
> and
> 
> 2016-02-29 06:08:59.230754 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
> no reply from osd.0 since back 2016-02-29 05:55:26.351951 front
> 2016-02-29 05:55:26.351951 (cutoff 2016-02-29 06:08:39.230753)
> 2016-02-29 06:08:59.230761 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
> no reply from osd.1 since back 2016-02-29 05:41:59.191341 front
> 2016-02-29 05:41:59.191341 (cutoff 2016-02-29 06:08:39.230753)
> 2016-02-29 06:08:59.230765 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
> no reply from osd.2 since back 2016-02-29 05:41:59.191341 front
> 2016-02-29 05:41:59.191341 (cutoff 2016-02-29 06:08:39.230753)
> 2016-02-29 06:08:59.230769 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
> no reply from osd.4 since back 2016-02-29 05:55:30.452505 front
> 2016-02-29 05:55:30.452505 (cutoff 2016-02-29 06:08:39.230753)
> 2016-02-29 06:08:59.230773 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
> no reply from osd.7 since back 2016-02-29 05:41:52.790422 front
> 2016-02-29 05:41:52.790422 (cutoff 2016-02-29 06:08:39.230753)
> 
> 
> Any idea what could be the trouble of the cluster ?
> 
> Thank you !
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] systemd & sysvinit scripts mix ?

[Christian Balzer]

Hello,

definitely not going to drag this into the systemd versus SysV cesspit (I
have very strong feelings about that matter, but that's neither here nor
now).

For the record, I installed my clusters on Jessie starting with Firefly.
Which had crap systemd support, so I couldn't (re)start individual Ceph
components, etc.

Since Debian offers a choice I re-enabled SysV init and was/am a happy
camper.
Things work as expected and I intend to keep it this way unless forced. 
So yes, please keep supporting the init scripts.

Christian

On Mon, 29 Feb 2016 18:22:44 +0100 Florent B wrote:

> init script can stay, but have to be disabled on systemd when systemd is
> used...
> 
> On 02/29/2016 05:58 PM, c...@jack.fr.eu.org wrote:
> > Well, if they are harmless, please let them
> > All people do not (and will not) use systemd nor upstart, please do
> > make life harder for these
> >
> > On 29/02/2016 17:56, Vasu Kulkarni wrote:
> >> +1
> >>
> >> On Mon, Feb 29, 2016 at 8:36 AM, Ken Dreyer 
> >> wrote:
> >>
> >>> I recommend we simply drop the init scripts from the master branch.
> >>> All our supported platforms (CentOS 7 or newer, and Ubuntu Trusty or
> >>> newer) use upstart or systemd.
> >>>
> >>> - Ken
> >>>
> >>> On Mon, Feb 29, 2016 at 3:44 AM, Florent B 
> >>> wrote:
>  Hi everyone,
> 
>  On a few servers, updated from Hammer to Infernalis, and from Debian
>  Wheezy to Jessie, I can see that it seems to have some mixes
>  between old sysvinit "ceph" script and the new ones on systemd.
> 
>  I always have an /etc/init.d/ceph old script, converted as a
>  service by systemd :
> 
>  # systemctl status ceph
>  ● ceph.service - LSB: Start Ceph distributed file system daemons at
>  boot time
> Loaded: loaded (/etc/init.d/ceph)
> Active: active (running) since Mon 2016-01-25 13:48:31 CET; 1
>  months 4 days ago
> CGroup: /system.slice/ceph.service
> └─13458 /usr/bin/python /usr/sbin/ceph-create-keys
>  --cluster ceph ...
> 
> 
>  And some new systemd services as "ceph.target" are inactive &
>  disabled :
> 
>  # systemctl status ceph.target
>  ● ceph.target - ceph target allowing to start/stop all
>  ceph*@.service instances at once
> Loaded: loaded (/lib/systemd/system/ceph.target; disabled)
> Active: inactive (dead)
> 
>  But others are loaded & enabled :
> 
>  # systemctl status ceph-osd@*
>  ● ceph-osd@0.service - Ceph object storage daemon
> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
> Active: active (running) since Thu 2015-12-03 17:51:55 CET; 2
>  months 26 days ago
>   Main PID: 13350 (ceph-osd)
> CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
> └─13350 /usr/bin/ceph-osd -f --cluster ceph --id 0
>  --setuser ceph ...
> 
>  ● ceph-osd@1.service - Ceph object storage daemon
> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
> Active: active (running) since Thu 2015-12-03 17:55:02 CET; 2
>  months 26 days ago
>   Main PID: 57626 (ceph-osd)
> CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
> └─57626 /usr/bin/ceph-osd -f --cluster ceph --id 1
>  --setuser ceph ...
> 
> 
>  Isn't there any misconfiguration there ? I think "/etc/init.d/ceph"
>  script should have been deleted on upgrade by Infernalis, isn't it ?
> 
>  What are the official recommendations about this ? Should I have to
>  delete old "ceph" script myself and enable all new services ? (and
>  why does it have to be done manually ?)
> 
>  Thank you.
> 
>  Florent
> 
>  ___
>  ceph-users mailing list
>  ceph-users@lists.ceph.com
>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> ___
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>
> >>
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] ext4 external journal - anyone tried this?

[Lindsay Mathieson]

On 2/05/2015 6:53 PM, Matthew Monaco wrote:

It looks like you can get a pretty good performance benefit from using ext4 with
an "external" SSD journal. Has anyone tried this with ceph? Take, for example, a
system with a 3:1 HDD to SSD ratio. What are some of your thoughts?



Did you ever get a reply to this Mathew or try it yourself?

Interested in trying it as an alternative to the various SSD caching 
options.


--
Lindsay Mathieson

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] s3 bucket creation time

[Robin H. Johnson]

On Mon, Feb 29, 2016 at 04:58:07PM +, Luis Periquito wrote:
> Hi all,
> 
> I have a biggish ceph environment and currently creating a bucket in
> radosgw can take as long as 20s.
> 
> What affects the time a bucket takes to be created? How can I improve that 
> time?
> 
> I've tried to create in several "bucket-location" with different
> backing pools (some of them empty) and the time was the same.
How many shards do you have configured for the bucket index?

I was recently benchmarking different bucket index shard values, and
also saw a notable increase relative to the number of shards.

Plus a concerning increase directly correlated to number of keys in the
bucket, but I need more data before I post to the lists about it.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail : robb...@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: List of SSDs

[Heath Albritton]

> Did you just do these tests or did you also do the "suitable for Ceph"
> song and dance, as in sync write speed?

These were done with libaio, so async.  I can do a sync test if that
helps.  My goal for testing wasn't specifically suitability with ceph,
but overall suitability in my environment, much of which uses async
IO.


>> SM863 Pro (default over-provisioning) ~7k IOPS per thread (4 threads,
>> QD32) Intel S3710 ~10k IOPS per thread
>> 845DC Pro ~12k IOPS per thread
>> SM863 (28% over-provisioning) ~18k IOPS per thread
>>
> Very interesting.
> To qualify your values up there, could you provide us with the exact
> models, well size of the SSD will do.

SM863 was 960GB, I've many of these and the 1.92TB models deployed
845DC Pro, 800GB
S3710, 800GB

> Also did you test with a S3700 (I find the 3710s to be a slight regression
> in some ways)?
> And for kicks, did you try over-provisioning with an Intel SSD to see the
> effects there?

These tests were performed mid-2015.  I requested an S3700, but at
that point, I could only get the S3710.  I didn't test the Intel with
increased over-provisioning.  I suspect it wouldn't have performed
much better as it was already over-provisioned by 28% or thereabouts.

It's easy to guess at these sort of things.  The total capacity of
flash is in some power of two and the advertised capacity is some
power of ten.  Manufacturer's use the difference to buy themselves
some space for garbage collection.  So, a terabyte worth of flash is
1099511627776 bytes.  800GB is 8e+11 bytes with the difference of
about 299GB, which is the space they've set aside for GC.

Again, if there's some tests you'd like to see done, let me know.
It's relatively easy for me to get samples and the tests are a benefit
to me as much as any other.


>> I'm seeing the S3710s at ~$1.20/GB and the SM863 around $.63/GB.  As
>> such, I'm buying quite a lot of the latter.
>
> I assume those numbers are before over-provisioning the SM863, still quite
> a difference indeed.

Yes, that's correct.  Here's some current pricing:  Newegg has the
SM863 960GB at $565 or ~$.59/GB raw.  With 28% OP, that yields around
800GB and around $.71/GB

>> I've not had them deployed
>> for very long, so I can't attest to anything beyond my synthetic
>> benchmarks.  I'm using the LSI 3008 based HBA as well and I've had to
>> use updated firmware and kernel module for it.  I haven't checked the
>> kernel that comes with EL7.2, but 7.1 still had problems with the
>> included driver.
>>
> Now THIS is really interesting.
> As you may know several people on this ML including me have issues with
> LSI 3008s and SSDs, including Samsung ones.
>
> Can you provide all the details here, as in:
> IT or IR mode (IT I presume)
> Firmware version
> Kernel driver version

When initially deployed about a year ago, I had problems with SSDs and
spinning disks.  Not sure about any problems specific to Samsung SSDs,
but I've been on the upgrade train.

I think the stock kernel module is 4.x something or other and LSA, now
Avago has released P9 through P12 in the past year.  When I first
started using them, I was on the P9 firmware and kernel module, which
I built from the sources they supply.  At this point most of my infra
is on the P10 version.  I've not tested the later versions.

Everything is IT mode where possible.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph and Google Summer of Code

[Patrick McGarry]

Hey cephers,

As many of you may have seen by now, Ceph was accepted back for
another year of GSoC. I’m asking all of you to make sure that any
applicable students that you know consider working with Ceph this
year.

We’re happy to accept proposals from our ideas list [0], or any custom
proposal that you and they might dream up. This also applies to
mentors. While we have a great group of initial mentors, if you are
interested in mentoring and have a student work with you to create a
proposal, I can add you as a mentor all the way through the end of the
application period.

If you have questions or comments, please feel free to reach out to me
directly. Thanks!

[0] http://ceph.com/gsoc2016

-- 

Best Regards,

Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com  ||  http://community.redhat.com
@scuttlemonkey || @ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and Google Summer of Code

[Patrick McGarry]

Hey Wido,

That’s a great idea, I’ll add it to the ideas list (and you as a
mentor). You, or anyone, should feel free to solicit a student to
submit the proposal if you know any. I think this would be a great
summer project. Thanks.


On Mon, Feb 29, 2016 at 4:53 PM, Wido den Hollander  wrote:
> A long wanted feature is mail storage in RADOS:
> http://tracker.ceph.com/issues/12430
>
> Would that be a good idea? I'd be more than happy to mentor this one.
>
> I will probably lack the technical C++ skills, but e-mail storage itself is
> something I'm very familiar with.
>
> Wido
>
>> Op 29 februari 2016 om 22:12 schreef Patrick McGarry :
>>
>>
>> Hey cephers,
>>
>> As many of you may have seen by now, Ceph was accepted back for
>> another year of GSoC. I’m asking all of you to make sure that any
>> applicable students that you know consider working with Ceph this
>> year.
>>
>> We’re happy to accept proposals from our ideas list [0], or any custom
>> proposal that you and they might dream up. This also applies to
>> mentors. While we have a great group of initial mentors, if you are
>> interested in mentoring and have a student work with you to create a
>> proposal, I can add you as a mentor all the way through the end of the
>> application period.
>>
>> If you have questions or comments, please feel free to reach out to me
>> directly. Thanks!
>>
>> [0] http://ceph.com/gsoc2016
>>
>> --
>>
>> Best Regards,
>>
>> Patrick McGarry
>> Director Ceph Community || Red Hat
>> http://ceph.com  ||  http://community.redhat.com
>> @scuttlemonkey || @ceph
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 

Best Regards,

Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com  ||  http://community.redhat.com
@scuttlemonkey || @ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and Google Summer of Code

[Shinobu Kinjo]

Yeah, sounds good. +1

Cheers,

- Original Message -
From: "David" 
To: "Wido den Hollander" 
Cc: "Ceph Devel" , "Ceph-User" 
, bo...@lists.ceph.com
Sent: Tuesday, March 1, 2016 7:22:31 AM
Subject: Re: [ceph-users] Ceph and Google Summer of Code

Great idea! +1

David Majchrzak

> 29 feb. 2016 kl. 22:53 skrev Wido den Hollander :
> 
> A long wanted feature is mail storage in RADOS:
> http://tracker.ceph.com/issues/12430
> 
> Would that be a good idea? I'd be more than happy to mentor this one.
> 
> I will probably lack the technical C++ skills, but e-mail storage itself is
> something I'm very familiar with.
> 
> Wido
> 
>> Op 29 februari 2016 om 22:12 schreef Patrick McGarry :
>> 
>> 
>> Hey cephers,
>> 
>> As many of you may have seen by now, Ceph was accepted back for
>> another year of GSoC. I’m asking all of you to make sure that any
>> applicable students that you know consider working with Ceph this
>> year.
>> 
>> We’re happy to accept proposals from our ideas list [0], or any custom
>> proposal that you and they might dream up. This also applies to
>> mentors. While we have a great group of initial mentors, if you are
>> interested in mentoring and have a student work with you to create a
>> proposal, I can add you as a mentor all the way through the end of the
>> application period.
>> 
>> If you have questions or comments, please feel free to reach out to me
>> directly. Thanks!
>> 
>> [0] http://ceph.com/gsoc2016
>> 
>> -- 
>> 
>> Best Regards,
>> 
>> Patrick McGarry
>> Director Ceph Community || Red Hat
>> http://ceph.com  ||  http://community.redhat.com
>> @scuttlemonkey || @ceph
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Shinobu Kinjo]

> Probably not (unless they reveal themselves extremely unreliable with
> Ceph OSD usage patterns which would be surprising to me).

Thank you for letting me know your thought.
That does make sense.

Cheers,

- Original Message -
From: "Lionel Bouton" 
To: "Shinobu Kinjo" 
Cc: "Mario Giammarco" , ceph-users@lists.ceph.com
Sent: Tuesday, March 1, 2016 6:56:05 AM
Subject: Re: [ceph-users] Help: pool not responding

Le 29/02/2016 22:50, Shinobu Kinjo a écrit :
>> the fact that they are optimized for benchmarks and certainly not
>> Ceph OSD usage patterns (with or without internal journal).
> Are you assuming that SSHD is causing the issue?
> If you could elaborate on this more, it would be helpful.

Probably not (unless they reveal themselves extremely unreliable with
Ceph OSD usage patterns which would be surprising to me).

For incomplete PG the documentation seems good enough for what should be
done :
http://docs.ceph.com/docs/master/rados/operations/pg-states/

The relevant text:

/Incomplete/
Ceph detects that a placement group is missing information about
writes that may have occurred, or does not have any healthy copies.
If you see this state, try to start any failed OSDs that may contain
the needed information or temporarily adjust min_size to allow recovery.

We don't have the full history but the most probable cause of these
incomplete PGs is that min_size is set to 2 or 3 and at some time the 4
incomplete pgs didn't have as many replica as the min_size value. So if
setting min_size to 2 isn't enough setting it to 1 should unfreeze them.

Lionel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Nmz]

In my free time I`m trying to understand how CEPH tries to detect corrupted 
data.
You can look here 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007680.html

Can you try to do md5sum on stucks PG from all OSD?

> Oliver Dzombic  writes:


>> Hi,

>> i dont know, but as it seems to me:

>> incomplete = not enough data

>> the only solution would be to drop it ( delete )

>> so the cluster get in active healthy state.

>> How many copies do you do from each data ?



> Do you mean dropping the pg not working or the entire pool?

> It is a pool with replication=3 and I had alway at least two osd on.

> Is replication=3 not enough?


> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and Google Summer of Code

[David]

Great idea! +1

David Majchrzak

> 29 feb. 2016 kl. 22:53 skrev Wido den Hollander :
> 
> A long wanted feature is mail storage in RADOS:
> http://tracker.ceph.com/issues/12430
> 
> Would that be a good idea? I'd be more than happy to mentor this one.
> 
> I will probably lack the technical C++ skills, but e-mail storage itself is
> something I'm very familiar with.
> 
> Wido
> 
>> Op 29 februari 2016 om 22:12 schreef Patrick McGarry :
>> 
>> 
>> Hey cephers,
>> 
>> As many of you may have seen by now, Ceph was accepted back for
>> another year of GSoC. I’m asking all of you to make sure that any
>> applicable students that you know consider working with Ceph this
>> year.
>> 
>> We’re happy to accept proposals from our ideas list [0], or any custom
>> proposal that you and they might dream up. This also applies to
>> mentors. While we have a great group of initial mentors, if you are
>> interested in mentoring and have a student work with you to create a
>> proposal, I can add you as a mentor all the way through the end of the
>> application period.
>> 
>> If you have questions or comments, please feel free to reach out to me
>> directly. Thanks!
>> 
>> [0] http://ceph.com/gsoc2016
>> 
>> -- 
>> 
>> Best Regards,
>> 
>> Patrick McGarry
>> Director Ceph Community || Red Hat
>> http://ceph.com  ||  http://community.redhat.com
>> @scuttlemonkey || @ceph
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and Google Summer of Code

[Wido den Hollander]

A long wanted feature is mail storage in RADOS:
http://tracker.ceph.com/issues/12430

Would that be a good idea? I'd be more than happy to mentor this one.

I will probably lack the technical C++ skills, but e-mail storage itself is
something I'm very familiar with.

Wido

> Op 29 februari 2016 om 22:12 schreef Patrick McGarry :
> 
> 
> Hey cephers,
> 
> As many of you may have seen by now, Ceph was accepted back for
> another year of GSoC. I’m asking all of you to make sure that any
> applicable students that you know consider working with Ceph this
> year.
> 
> We’re happy to accept proposals from our ideas list [0], or any custom
> proposal that you and they might dream up. This also applies to
> mentors. While we have a great group of initial mentors, if you are
> interested in mentoring and have a student work with you to create a
> proposal, I can add you as a mentor all the way through the end of the
> application period.
> 
> If you have questions or comments, please feel free to reach out to me
> directly. Thanks!
> 
> [0] http://ceph.com/gsoc2016
> 
> -- 
> 
> Best Regards,
> 
> Patrick McGarry
> Director Ceph Community || Red Hat
> http://ceph.com  ||  http://community.redhat.com
> @scuttlemonkey || @ceph
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Lionel Bouton]

Le 29/02/2016 22:50, Shinobu Kinjo a écrit :
>> the fact that they are optimized for benchmarks and certainly not
>> Ceph OSD usage patterns (with or without internal journal).
> Are you assuming that SSHD is causing the issue?
> If you could elaborate on this more, it would be helpful.

Probably not (unless they reveal themselves extremely unreliable with
Ceph OSD usage patterns which would be surprising to me).

For incomplete PG the documentation seems good enough for what should be
done :
http://docs.ceph.com/docs/master/rados/operations/pg-states/

The relevant text:

/Incomplete/
Ceph detects that a placement group is missing information about
writes that may have occurred, or does not have any healthy copies.
If you see this state, try to start any failed OSDs that may contain
the needed information or temporarily adjust min_size to allow recovery.

We don't have the full history but the most probable cause of these
incomplete PGs is that min_size is set to 2 or 3 and at some time the 4
incomplete pgs didn't have as many replica as the min_size value. So if
setting min_size to 2 isn't enough setting it to 1 should unfreeze them.

Lionel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Shinobu Kinjo]

> the fact that they are optimized for benchmarks and certainly not
> Ceph OSD usage patterns (with or without internal journal).

Are you assuming that SSHD is causing the issue?
If you could elaborate on this more, it would be helpful.

Cheers,
Shinobu

- Original Message -
From: "Lionel Bouton" 
To: "Mario Giammarco" , ceph-users@lists.ceph.com
Sent: Tuesday, March 1, 2016 5:29:38 AM
Subject: Re: [ceph-users] Help: pool not responding

Le 29/02/2016 20:43, Mario Giammarco a écrit :
> [...]
> I said SSHD that is a standard hdd with ssd cache. It is 7200rpms but in
> benchmarks it is better than a 1rpm disk.

Lies, damn lies and benchmarks...
SSHD usually have very small flash caches (16GB or less for 500GB of
data or more) and AFAIK there's no distribution supporting cache hints
or to be of any use here Ceph OSD cache hint support : the drive makes
the decisions about when to use the cache and you can trust only one
thing: the fact that they are optimized for benchmarks and certainly not
Ceph OSD usage patterns (with or without internal journal).

There are probably 2 kinds of optimizations that SSHD can perform :
- buffering random writes with a writeback cache algorithm targeting
random writes. With only 8 to 16GB of flash this would probably mean
that under heavy random write usage (typical for OSD) the flash will die
very fast which would kill the entire drive or lose data and so it's
unlikely that this is what they use.
- write the most used data (what is first loaded on system boot and what
is most used) to the flash cache to speed up the OS boot sequence and
access to the most used applications or data. As OSDs don't have any
recognizable pattern this is useless in most cases.

So SSHD for OSD are almost certainly useless. You are better off saving
money by buying more ordinary HDD SATA drives or as many HDD and a few
good SSDs for journal if you can afford them.

In fact if the SSHD tries to cache writes and doesn't die early in the
process you may get even worse performance than a pure HDD setup because
most consumer-level SSD (and probably SSHD) are absolute crap for the
type of access Ceph OSD do with journals (see
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
for the horror stories).

Best regards,

Lionel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Mario Giammarco]

Oliver Dzombic  writes:

> 
> Hi,
> 
> i dont know, but as it seems to me:
> 
> incomplete = not enough data
> 
> the only solution would be to drop it ( delete )
> 
> so the cluster get in active healthy state.
> 
> How many copies do you do from each data ?
> 


Do you mean dropping the pg not working or the entire pool?

It is a pool with replication=3 and I had alway at least two osd on.

Is replication=3 not enough?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Oliver Dzombic]

Hi,

i dont know, but as it seems to me:

incomplete = not enough data

the only solution would be to drop it ( delete )

so the cluster get in active healthy state.

How many copies do you do from each data ?

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 29.02.2016 um 20:56 schrieb Mario Giammarco:
> Mario Giammarco  writes:
> 
> Sorry 
> ceph health detail is:
> 
> 
> HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean
> pg 0.0 is stuck inactive for 4836623.776873, current state incomplete, last
> acting [0,1,3]
> pg 0.40 is stuck inactive for 2773379.028048, current state incomplete, last
> acting [1,0,3]
> pg 0.3f is stuck inactive for 4836763.332907, current state incomplete, last
> acting [0,3,1]
> pg 0.3b is stuck inactive for 4836777.230337, current state incomplete, last
> acting [0,3,1]
> pg 0.0 is stuck unclean for 4850437.633464, current state incomplete, last
> acting [0,1,3]
> pg 0.40 is stuck unclean for 4850437.633467, current state incomplete, last
> acting [1,0,3]
> pg 0.3f is stuck unclean for 4850456.399217, current state incomplete, last
> acting [0,3,1]
> pg 0.3b is stuck unclean for 4850490.534154, current state incomplete, last
> acting [0,3,1]
> pg 0.40 is incomplete, acting [1,0,3]
> pg 0.3f is incomplete, acting [0,3,1]
> pg 0.3b is incomplete, acting [0,3,1]
> pg 0.0 is incomplete, acting [0,1,3]
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Developer Monthly this Wed!

[Patrick McGarry]

Hey cephers,

Just a reminder, the monthly dev meeting [0] for ceph developers is
this Wed at 9p EST (we are on an APAC-friendly month). If you are
currently working on commits to ceph, or would like to be, please join
us for a quick rundown of work in progress.

If you are able, it would be greatly appreciated if you added your
work to the wiki page [1] so that we know what to expect for
disucssions. Thanks!


[0] http://tracker.ceph.com/projects/ceph/wiki/Planning
[1] http://tracker.ceph.com/projects/ceph/wiki/CDM_02-MAR-2016

-- 

Best Regards,

Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com  ||  http://community.redhat.com
@scuttlemonkey || @ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Mario Giammarco]

Mario Giammarco  writes:

Sorry 
ceph health detail is:


HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean
pg 0.0 is stuck inactive for 4836623.776873, current state incomplete, last
acting [0,1,3]
pg 0.40 is stuck inactive for 2773379.028048, current state incomplete, last
acting [1,0,3]
pg 0.3f is stuck inactive for 4836763.332907, current state incomplete, last
acting [0,3,1]
pg 0.3b is stuck inactive for 4836777.230337, current state incomplete, last
acting [0,3,1]
pg 0.0 is stuck unclean for 4850437.633464, current state incomplete, last
acting [0,1,3]
pg 0.40 is stuck unclean for 4850437.633467, current state incomplete, last
acting [1,0,3]
pg 0.3f is stuck unclean for 4850456.399217, current state incomplete, last
acting [0,3,1]
pg 0.3b is stuck unclean for 4850490.534154, current state incomplete, last
acting [0,3,1]
pg 0.40 is incomplete, acting [1,0,3]
pg 0.3f is incomplete, acting [0,3,1]
pg 0.3b is incomplete, acting [0,3,1]
pg 0.0 is incomplete, acting [0,1,3]



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Mario Giammarco]

Thank you for your time.
Dimitar Boichev  writes:

> 
> I am sure that I speak for the majority of people reading this, when I say
that I didn't get anything from your emails.
> Could you provide more debug information ?
> Like (but not limited to):
> ceph -s 
> ceph health details
> ceph osd tree

I asked infact what I need to provide because honestly I do not know.

Here is ceph -s:

cluster ac7bc476-3a02-453d-8e5c-606ab6f022ca
 health HEALTH_WARN
4 pgs incomplete
4 pgs stuck inactive
4 pgs stuck unclean
 monmap e8: 3 mons at
{0=10.1.0.12:6789/0,1=10.1.0.14:6789/0,2=10.1.0.17:6789/0}
election epoch 832, quorum 0,1,2 0,1,2
 osdmap e2400: 3 osds: 3 up, 3 in
  pgmap v5883297: 288 pgs, 4 pools, 391 GB data, 100 kobjects
1090 GB used, 4481 GB / 5571 GB avail
 284 active+clean
   4 incomplete

ceph health detail:

cluster ac7bc476-3a02-453d-8e5c-606ab6f022ca
 health HEALTH_WARN
4 pgs incomplete
4 pgs stuck inactive
4 pgs stuck unclean
 monmap e8: 3 mons at
{0=10.1.0.12:6789/0,1=10.1.0.14:6789/0,2=10.1.0.17:6789/0}
election epoch 832, quorum 0,1,2 0,1,2
 osdmap e2400: 3 osds: 3 up, 3 in
  pgmap v5883297: 288 pgs, 4 pools, 391 GB data, 100 kobjects
1090 GB used, 4481 GB / 5571 GB avail
 284 active+clean
   4 incomplete

ceph osd tree:

ID WEIGHT  TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 5.42999 root default 
-2 1.81000 host proxmox-quad3   
 0 1.81000 osd.0   up  1.0  1.0 
-3 1.81000 host proxmox-zotac   
 1 1.81000 osd.1   up  1.0  1.0 
-4 1.81000 host proxmox-hp  
 3 1.81000 osd.3   up  1.0  1.0 


> 
> I am really having a bad time trying to decode the exact problems.
> First you had network issues, then osd failed (in the same time or after?),
> Then the cluser did not have enough free space to recover I suppose  ?
> 
It is a three server/osd test/evaluation system with Ceph and Proxmox PVE.
The load is very light and there is a lot of free space.

So:

- I NEVER had network issues. People TOLD me that I must have network
problems. I changed cables and switches just in case but nothing improved. 
- One disk had bad sectors. So I added another disk/osd and then removed the
osd. Following official documentation. After that the cluster runned ok for
two months. So there was enough free space and the cluster has recovered.
- Then one day I discovered that proxmox backup was hanged and I see that it
was because ceph was not responding.


> Regarding the slow SSD disks, what disks are you using ?

I said SSHD that is a standard hdd with ssd cache. It is 7200rpms but in
benchmarks it is better than a 1rpm disk.

Thanks again,
Mario


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

Hm, I should be a bit more updated now. At least for 
{debian,rpm}-{hammer,infernalis,testing}

/Josef
> On 29 Feb 2016, at 19:19, Wido den Hollander  wrote:
> 
> 
>> Op 29 februari 2016 om 18:22 schreef Austin Johnson 
>> :
>> 
>> 
>> All,
>> 
>> I agree that rsync is down on download.ceph.com. I get a connection timeout
>> as well. Which makes it seem like an issue of the firewall silently
>> dropping packets.
>> 
> 
> Remember that download.ceph.com is THE source of all data. eu.ceph.com syncs
> from there as well.
> 
> That needs to be fixed since there is no other source.
> 
> Wido
> 
>> It has been down for at least a few weeks, forcing me to sync from eu,
>> which seems out of date.
>> 
>> Tyler - Is there any way that beyondhosting.net could turn rsync up for its
>> mirror?
>> 
>> Thanks,
>> Austin
>> 
>> On Mon, Feb 29, 2016 at 7:19 AM, Florent B  wrote:
>> 
>>> I would like to inform you that I have difficulties to set-up a mirror.
>>> 
>>> rsync on download.ceph.com is down
>>> 
>>> # rsync download.ceph.com::
>>> rsync: failed to connect to download.ceph.com (173.236.253.173):
>>> Connection timed out (110)
>>> 
>>> And eu.ceph.com is out of sync for a few weeks.
>>> 
>>> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
 Hi,
 
 My PR was merged with a script to mirror Ceph properly:
 https://github.com/ceph/ceph/tree/master/mirroring
 
 Currently there are 3 (official) locations where you can get Ceph:
 
 - download.ceph.com (Dreamhost, US)
 - eu.ceph.com (PCextreme, Netherlands)
 - au.ceph.com (Digital Pacific, Australia)
 
 I'm looking for more mirrors to become official mirrors so we can easily
 distribute Ceph.
 
 Mirrors do go down and it's always nice to have a mirror local to you.
 
 I'd like to have one or more mirrors in Asia, Africa and/or South
 Ameirca if possible. Anyone able to host there? Other locations are
 welcome as well!
 
 A few things which are required:
 
 - 1Gbit connection or more
 - Native IPv4 and IPv6
 - HTTP access
 - rsync access
 - 2TB of storage or more
 - Monitoring of the mirror/source
 
 You can easily mirror Ceph yourself with this script I wrote:
 https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh
 
 eu.ceph.com and au.ceph.com use it to sync from download.ceph.com. If
 you want to mirror Ceph locally, please pick a mirror local to you.
 
 Please refer to these guidelines:
 https://github.com/ceph/ceph/tree/master/mirroring#guidelines
 
>>> 
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Wido den Hollander]

> Op 29 februari 2016 om 18:22 schreef Austin Johnson :
> 
> 
> All,
> 
> I agree that rsync is down on download.ceph.com. I get a connection timeout
> as well. Which makes it seem like an issue of the firewall silently
> dropping packets.
> 

Remember that download.ceph.com is THE source of all data. eu.ceph.com syncs
from there as well.

That needs to be fixed since there is no other source.

Wido

> It has been down for at least a few weeks, forcing me to sync from eu,
> which seems out of date.
> 
> Tyler - Is there any way that beyondhosting.net could turn rsync up for its
> mirror?
> 
> Thanks,
> Austin
> 
> On Mon, Feb 29, 2016 at 7:19 AM, Florent B  wrote:
> 
> > I would like to inform you that I have difficulties to set-up a mirror.
> >
> > rsync on download.ceph.com is down
> >
> > # rsync download.ceph.com::
> > rsync: failed to connect to download.ceph.com (173.236.253.173):
> > Connection timed out (110)
> >
> > And eu.ceph.com is out of sync for a few weeks.
> >
> > On 01/30/2016 03:14 PM, Wido den Hollander wrote:
> > > Hi,
> > >
> > > My PR was merged with a script to mirror Ceph properly:
> > > https://github.com/ceph/ceph/tree/master/mirroring
> > >
> > > Currently there are 3 (official) locations where you can get Ceph:
> > >
> > > - download.ceph.com (Dreamhost, US)
> > > - eu.ceph.com (PCextreme, Netherlands)
> > > - au.ceph.com (Digital Pacific, Australia)
> > >
> > > I'm looking for more mirrors to become official mirrors so we can easily
> > > distribute Ceph.
> > >
> > > Mirrors do go down and it's always nice to have a mirror local to you.
> > >
> > > I'd like to have one or more mirrors in Asia, Africa and/or South
> > > Ameirca if possible. Anyone able to host there? Other locations are
> > > welcome as well!
> > >
> > > A few things which are required:
> > >
> > > - 1Gbit connection or more
> > > - Native IPv4 and IPv6
> > > - HTTP access
> > > - rsync access
> > > - 2TB of storage or more
> > > - Monitoring of the mirror/source
> > >
> > > You can easily mirror Ceph yourself with this script I wrote:
> > > https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh
> > >
> > > eu.ceph.com and au.ceph.com use it to sync from download.ceph.com. If
> > > you want to mirror Ceph locally, please pick a mirror local to you.
> > >
> > > Please refer to these guidelines:
> > > https://github.com/ceph/ceph/tree/master/mirroring#guidelines
> > >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Dan van der Ster]

If it can help, it's really very little work for me to send the hammer
SRPM to our Koji build system.

I think the real work will come if people starting asking for jewel
builds on el6 and other old platforms. In that case, if a reputable
organisation offers to maintain the builds (+ deps), then IMHO
ceph.com should just link to them. Copying to ceph.com might give a
false sense of security in this case.

-- Dan


On Mon, Feb 29, 2016 at 5:10 PM, Loic Dachary  wrote:
> I've created a pad at http://pad.ceph.com/p/development-releases for the next 
> CDM ( see http://tracker.ceph.com/projects/ceph/wiki/Planning for details).
>
> On 29/02/2016 22:49, Nathan Cutler wrote:
>> The basic idea is to copy the packages that are build by gitbuilders or by 
>> the buildpackage teuthology task in a central place. Because these packages 
>> are built, for development versions as well as stable versions[2]. And they 
>> are tested via teuthology. The packages that are published on 
>> http://ceph.com/ are rebuilt from scratch, using the process that Alfredo 
>> described. This is fine for the supported platforms and for the stable 
>> releases. But for the development releases and the platforms that are no 
>> longer supported but still built by gibuilders, we could just copy the 
>> packages over.
>
> --
> Loïc Dachary, Artisan Logiciel Libre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Austin Johnson]

All,

I agree that rsync is down on download.ceph.com. I get a connection timeout
as well. Which makes it seem like an issue of the firewall silently
dropping packets.

It has been down for at least a few weeks, forcing me to sync from eu,
which seems out of date.

Tyler - Is there any way that beyondhosting.net could turn rsync up for its
mirror?

Thanks,
Austin

On Mon, Feb 29, 2016 at 7:19 AM, Florent B  wrote:

> I would like to inform you that I have difficulties to set-up a mirror.
>
> rsync on download.ceph.com is down
>
> # rsync download.ceph.com::
> rsync: failed to connect to download.ceph.com (173.236.253.173):
> Connection timed out (110)
>
> And eu.ceph.com is out of sync for a few weeks.
>
> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
> > Hi,
> >
> > My PR was merged with a script to mirror Ceph properly:
> > https://github.com/ceph/ceph/tree/master/mirroring
> >
> > Currently there are 3 (official) locations where you can get Ceph:
> >
> > - download.ceph.com (Dreamhost, US)
> > - eu.ceph.com (PCextreme, Netherlands)
> > - au.ceph.com (Digital Pacific, Australia)
> >
> > I'm looking for more mirrors to become official mirrors so we can easily
> > distribute Ceph.
> >
> > Mirrors do go down and it's always nice to have a mirror local to you.
> >
> > I'd like to have one or more mirrors in Asia, Africa and/or South
> > Ameirca if possible. Anyone able to host there? Other locations are
> > welcome as well!
> >
> > A few things which are required:
> >
> > - 1Gbit connection or more
> > - Native IPv4 and IPv6
> > - HTTP access
> > - rsync access
> > - 2TB of storage or more
> > - Monitoring of the mirror/source
> >
> > You can easily mirror Ceph yourself with this script I wrote:
> > https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh
> >
> > eu.ceph.com and au.ceph.com use it to sync from download.ceph.com. If
> > you want to mirror Ceph locally, please pick a mirror local to you.
> >
> > Please refer to these guidelines:
> > https://github.com/ceph/ceph/tree/master/mirroring#guidelines
> >
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

Got rpm-infernalis in now, and I’m updating debian-infernalis as well.

/Josef
> On 29 Feb 2016, at 15:44, Josef Johansson  wrote:
> 
> Syncing now.
>> On 29 Feb 2016, at 15:38, Josef Johansson > > wrote:
>> 
>> I’ll check if I can mirror it though http.
>>> On 29 Feb 2016, at 15:31, Josef Johansson >> > wrote:
>>> 
>>> Then we’re all in the same boat.
>>> 
 On 29 Feb 2016, at 15:30, Florent B > wrote:
 
 Hi and thank you. But for me, you are out of sync as eu.ceph.com 
 . Can't find Infernalis 9.2.1 on your mirror :(
 
 On 02/29/2016 03:21 PM, Josef Johansson wrote:
> You could sync from me instead @ se.ceph.com  
> As a start.
> 
> Regards
> /Josef
> 
>> On 29 Feb 2016, at 15:19, Florent B < 
>> flor...@coppint.com 
>> > wrote:
>> 
>> I would like to inform you that I have difficulties to set-up a mirror.
>> 
>> rsync on download.ceph.com  is down
>> 
>> # rsync download.ceph.com ::
>> rsync: failed to connect to download.ceph.com 
>>  (173.236.253.173):
>> Connection timed out (110)
>> 
>> And eu.ceph.com  is out of sync for a few weeks.
>> 
>> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
>>> Hi,
>>> 
>>> My PR was merged with a script to mirror Ceph properly:
>>> https://github.com/ceph/ceph/tree/master/mirroring 
>>> 
>>> 
>>> Currently there are 3 (official) locations where you can get Ceph:
>>> 
>>> - download.ceph.com  (Dreamhost, US)
>>> - eu.ceph.com  (PCextreme, Netherlands)
>>> - au.ceph.com  (Digital Pacific, Australia)
>>> 
>>> I'm looking for more mirrors to become official mirrors so we can easily
>>> distribute Ceph.
>>> 
>>> Mirrors do go down and it's always nice to have a mirror local to you.
>>> 
>>> I'd like to have one or more mirrors in Asia, Africa and/or South
>>> Ameirca if possible. Anyone able to host there? Other locations are
>>> welcome as well!
>>> 
>>> A few things which are required:
>>> 
>>> - 1Gbit connection or more
>>> - Native IPv4 and IPv6
>>> - HTTP access
>>> - rsync access
>>> - 2TB of storage or more
>>> - Monitoring of the mirror/source
>>> 
>>> You can easily mirror Ceph yourself with this script I wrote:
>>> https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh 
>>> 
>>> 
>>> eu.ceph.com  and au.ceph.com  
>>> use it to sync from download.ceph.com . If
>>> you want to mirror Ceph locally, please pick a mirror local to you.
>>> 
>>> Please refer to these guidelines:
>>> https://github.com/ceph/ceph/tree/master/mirroring#guidelines 
>>> 
>>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>> 
> 
 
>>> 
>> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] s3 bucket creation time

[Luis Periquito]

Hi all,

I have a biggish ceph environment and currently creating a bucket in
radosgw can take as long as 20s.

What affects the time a bucket takes to be created? How can I improve that time?

I've tried to create in several "bucket-location" with different
backing pools (some of them empty) and the time was the same.

thanks,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] systemd & sysvinit scripts mix ?

[ceph]

Well, if they are harmless, please let them
All people do not (and will not) use systemd nor upstart, please do make
life harder for these

On 29/02/2016 17:56, Vasu Kulkarni wrote:
> +1
> 
> On Mon, Feb 29, 2016 at 8:36 AM, Ken Dreyer  wrote:
> 
>> I recommend we simply drop the init scripts from the master branch.
>> All our supported platforms (CentOS 7 or newer, and Ubuntu Trusty or
>> newer) use upstart or systemd.
>>
>> - Ken
>>
>> On Mon, Feb 29, 2016 at 3:44 AM, Florent B  wrote:
>>> Hi everyone,
>>>
>>> On a few servers, updated from Hammer to Infernalis, and from Debian
>>> Wheezy to Jessie, I can see that it seems to have some mixes between old
>>> sysvinit "ceph" script and the new ones on systemd.
>>>
>>> I always have an /etc/init.d/ceph old script, converted as a service by
>>> systemd :
>>>
>>> # systemctl status ceph
>>> ● ceph.service - LSB: Start Ceph distributed file system daemons at boot
>>> time
>>>Loaded: loaded (/etc/init.d/ceph)
>>>Active: active (running) since Mon 2016-01-25 13:48:31 CET; 1 months
>>> 4 days ago
>>>CGroup: /system.slice/ceph.service
>>>└─13458 /usr/bin/python /usr/sbin/ceph-create-keys --cluster
>>> ceph ...
>>>
>>>
>>> And some new systemd services as "ceph.target" are inactive & disabled :
>>>
>>> # systemctl status ceph.target
>>> ● ceph.target - ceph target allowing to start/stop all ceph*@.service
>>> instances at once
>>>Loaded: loaded (/lib/systemd/system/ceph.target; disabled)
>>>Active: inactive (dead)
>>>
>>> But others are loaded & enabled :
>>>
>>> # systemctl status ceph-osd@*
>>> ● ceph-osd@0.service - Ceph object storage daemon
>>>Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
>>>Active: active (running) since Thu 2015-12-03 17:51:55 CET; 2 months
>>> 26 days ago
>>>  Main PID: 13350 (ceph-osd)
>>>CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
>>>└─13350 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser
>>> ceph ...
>>>
>>> ● ceph-osd@1.service - Ceph object storage daemon
>>>Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
>>>Active: active (running) since Thu 2015-12-03 17:55:02 CET; 2 months
>>> 26 days ago
>>>  Main PID: 57626 (ceph-osd)
>>>CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
>>>└─57626 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser
>>> ceph ...
>>>
>>>
>>> Isn't there any misconfiguration there ? I think "/etc/init.d/ceph"
>>> script should have been deleted on upgrade by Infernalis, isn't it ?
>>>
>>> What are the official recommendations about this ? Should I have to
>>> delete old "ceph" script myself and enable all new services ? (and why
>>> does it have to be done manually ?)
>>>
>>> Thank you.
>>>
>>> Florent
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] systemd & sysvinit scripts mix ?

[Vasu Kulkarni]

+1

On Mon, Feb 29, 2016 at 8:36 AM, Ken Dreyer  wrote:

> I recommend we simply drop the init scripts from the master branch.
> All our supported platforms (CentOS 7 or newer, and Ubuntu Trusty or
> newer) use upstart or systemd.
>
> - Ken
>
> On Mon, Feb 29, 2016 at 3:44 AM, Florent B  wrote:
> > Hi everyone,
> >
> > On a few servers, updated from Hammer to Infernalis, and from Debian
> > Wheezy to Jessie, I can see that it seems to have some mixes between old
> > sysvinit "ceph" script and the new ones on systemd.
> >
> > I always have an /etc/init.d/ceph old script, converted as a service by
> > systemd :
> >
> > # systemctl status ceph
> > ● ceph.service - LSB: Start Ceph distributed file system daemons at boot
> > time
> >Loaded: loaded (/etc/init.d/ceph)
> >Active: active (running) since Mon 2016-01-25 13:48:31 CET; 1 months
> > 4 days ago
> >CGroup: /system.slice/ceph.service
> >└─13458 /usr/bin/python /usr/sbin/ceph-create-keys --cluster
> > ceph ...
> >
> >
> > And some new systemd services as "ceph.target" are inactive & disabled :
> >
> > # systemctl status ceph.target
> > ● ceph.target - ceph target allowing to start/stop all ceph*@.service
> > instances at once
> >Loaded: loaded (/lib/systemd/system/ceph.target; disabled)
> >Active: inactive (dead)
> >
> > But others are loaded & enabled :
> >
> > # systemctl status ceph-osd@*
> > ● ceph-osd@0.service - Ceph object storage daemon
> >Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
> >Active: active (running) since Thu 2015-12-03 17:51:55 CET; 2 months
> > 26 days ago
> >  Main PID: 13350 (ceph-osd)
> >CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
> >└─13350 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser
> > ceph ...
> >
> > ● ceph-osd@1.service - Ceph object storage daemon
> >Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
> >Active: active (running) since Thu 2015-12-03 17:55:02 CET; 2 months
> > 26 days ago
> >  Main PID: 57626 (ceph-osd)
> >CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
> >└─57626 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser
> > ceph ...
> >
> >
> > Isn't there any misconfiguration there ? I think "/etc/init.d/ceph"
> > script should have been deleted on upgrade by Infernalis, isn't it ?
> >
> > What are the official recommendations about this ? Should I have to
> > delete old "ceph" script myself and enable all new services ? (and why
> > does it have to be done manually ?)
> >
> > Thank you.
> >
> > Florent
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] systemd & sysvinit scripts mix ?

[Ken Dreyer]

I recommend we simply drop the init scripts from the master branch.
All our supported platforms (CentOS 7 or newer, and Ubuntu Trusty or
newer) use upstart or systemd.

- Ken

On Mon, Feb 29, 2016 at 3:44 AM, Florent B  wrote:
> Hi everyone,
>
> On a few servers, updated from Hammer to Infernalis, and from Debian
> Wheezy to Jessie, I can see that it seems to have some mixes between old
> sysvinit "ceph" script and the new ones on systemd.
>
> I always have an /etc/init.d/ceph old script, converted as a service by
> systemd :
>
> # systemctl status ceph
> ● ceph.service - LSB: Start Ceph distributed file system daemons at boot
> time
>Loaded: loaded (/etc/init.d/ceph)
>Active: active (running) since Mon 2016-01-25 13:48:31 CET; 1 months
> 4 days ago
>CGroup: /system.slice/ceph.service
>└─13458 /usr/bin/python /usr/sbin/ceph-create-keys --cluster
> ceph ...
>
>
> And some new systemd services as "ceph.target" are inactive & disabled :
>
> # systemctl status ceph.target
> ● ceph.target - ceph target allowing to start/stop all ceph*@.service
> instances at once
>Loaded: loaded (/lib/systemd/system/ceph.target; disabled)
>Active: inactive (dead)
>
> But others are loaded & enabled :
>
> # systemctl status ceph-osd@*
> ● ceph-osd@0.service - Ceph object storage daemon
>Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
>Active: active (running) since Thu 2015-12-03 17:51:55 CET; 2 months
> 26 days ago
>  Main PID: 13350 (ceph-osd)
>CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
>└─13350 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser
> ceph ...
>
> ● ceph-osd@1.service - Ceph object storage daemon
>Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled)
>Active: active (running) since Thu 2015-12-03 17:55:02 CET; 2 months
> 26 days ago
>  Main PID: 57626 (ceph-osd)
>CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
>└─57626 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser
> ceph ...
>
>
> Isn't there any misconfiguration there ? I think "/etc/init.d/ceph"
> script should have been deleted on upgrade by Infernalis, isn't it ?
>
> What are the official recommendations about this ? Should I have to
> delete old "ceph" script myself and enable all new services ? (and why
> does it have to be done manually ?)
>
> Thank you.
>
> Florent
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Loic Dachary]

On 29/02/2016 22:49, Nathan Cutler wrote:
>> The basic idea is to copy the packages that are build by gitbuilders or by 
>> the buildpackage teuthology task in a central place. Because these packages 
>> are built, for development versions as well as stable versions[2]. And they 
>> are tested via teuthology. The packages that are published on 
>> http://ceph.com/ are rebuilt from scratch, using the process that Alfredo 
>> described. This is fine for the supported platforms and for the stable 
>> releases. But for the development releases and the platforms that are no 
>> longer supported but still built by gibuilders, we could just copy the 
>> packages over.
>>
>> Does that sound sensible ?
> 
> Hi Loic:
> 
> Community packages for "deprecated" platforms ("deprecated" in the sense that 
> the Ceph developers are no longer testing on them) would be welcomed by many, 
> I imagine. And the additional workload for the Stable Releases team is not 
> large. The question is, where will the packages be copied *to*?
> 

How does archive.ceph.com sound ?

-- 
Loïc Dachary, Artisan Logiciel Libre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Loic Dachary]

I've created a pad at http://pad.ceph.com/p/development-releases for the next 
CDM ( see http://tracker.ceph.com/projects/ceph/wiki/Planning for details).

On 29/02/2016 22:49, Nathan Cutler wrote:
> The basic idea is to copy the packages that are build by gitbuilders or by 
> the buildpackage teuthology task in a central place. Because these packages 
> are built, for development versions as well as stable versions[2]. And they 
> are tested via teuthology. The packages that are published on 
> http://ceph.com/ are rebuilt from scratch, using the process that Alfredo 
> described. This is fine for the supported platforms and for the stable 
> releases. But for the development releases and the platforms that are no 
> longer supported but still built by gibuilders, we could just copy the 
> packages over.

-- 
Loïc Dachary, Artisan Logiciel Libre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Sage Weil]

The intention was to continue building stable releases (0.94.x) on the old 
list of supported platforms (which inclues 12.04 and el6).  I think it was 
just an oversight that they weren't built this time around.  I the 
overhead to doing so is just keeping a 12.04 and el6 jenkins build slave 
around.

Doing this builds in the existing environment sounds much better than 
trying to pull in externally built binaries...

sage


On Mon, 29 Feb 2016, Loic Dachary wrote:

> Hi Dan & al,
> 
> I think it would be relatively simple to have these binaries published as 
> part of the current "Stable release" team effort[1]. Essentially doing what 
> you did and electing a central place to store these binaries. The trick is to 
> find a sustainable way to do this which means having a process that is simple 
> to maintain and understand as well as enough of us to maintain that in the 
> long run.
> 
> The basic idea is to copy the packages that are build by gitbuilders or by 
> the buildpackage teuthology task in a central place. Because these packages 
> are built, for development versions as well as stable versions[2]. And they 
> are tested via teuthology. The packages that are published on 
> http://ceph.com/ are rebuilt from scratch, using the process that Alfredo 
> described. This is fine for the supported platforms and for the stable 
> releases. But for the development releases and the platforms that are no 
> longer supported but still built by gibuilders, we could just copy the 
> packages over.
> 
> Does that sound sensible ?
> 
> [1] http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO
> [2] http://gitbuilder.ceph.com/
> 
> 
> On 29/02/2016 20:28, Dan van der Ster wrote:
> > On Mon, Feb 29, 2016 at 12:30 PM, Odintsov Vladislav  
> > wrote:
> >> Can you please provide right way for building rpm packages?
> > 
> > It's documented here:
> > http://docs.ceph.com/docs/master/install/build-ceph/#rpm-package-manager
> > 
> > For 0.94.6 you need to change the .spec file to use .tar.gz (because
> > there was no .bz2 published for some reason). And then also grab
> > init-ceph.in-fedora.patch from here:
> > https://raw.githubusercontent.com/ceph/ceph/master/rpm/init-ceph.in-fedora.patch
> > 
> > BTW, I've put our build here:
> > http://linuxsoft.cern.ch/internal/repos/ceph6-stable/x86_64/os/
> > These are unsigned, untested and come with no warranty, no guarantees
> > of any sort. And IMHO, no third party build would ever to give that
> > warm fuzzy trust-it-with-my-data feeling like a ceph.com build would
> > ;)
> > 
> > Moving forward, it would be great if the required community effort
> > could be put to work to get ceph.com el6 (and other) builds. For el6
> > in particular there is also the option to help out the Centos Storage
> > SIG to produce builds. I don't have a good feeling which direction is
> > better ... maybe both.
> > 
> > -- Dan
> > CERN IT Storage Group
> > 
> > 
> >> 
> >> Regards,
> >>
> >> Vladislav Odintsov
> >>
> >> 
> >> From: Shinobu Kinjo 
> >> Sent: Monday, February 29, 2016 14:11
> >> To: Odintsov Vladislav
> >> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; 
> >> ceph-users; Sage Weil
> >> Subject: Re: [ceph-users] v0.94.6 Hammer released
> >>
> >> Can we make any kind of general procedure to make packages so that almost 
> >> everyone in community build packages by themselves and reduce developers 
> >> work load caused by too much requirement -;
> >>
> >> Cheers,
> >> Shinobu
> >>
> >> - Original Message -
> >> From: "Odintsov Vladislav" 
> >> To: "Franklin M. Siler" , "Xiaoxi Chen" 
> >> 
> >> Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage 
> >> Weil" 
> >> Sent: Monday, February 29, 2016 6:04:02 PM
> >> Subject: Re: [ceph-users] v0.94.6 Hammer released
> >>
> >> Hi all,
> >>
> >> should we build el6 packages ourself or, it's hoped that these packages 
> >> would be built officially by community?
> >>
> >> 
> >> Regards,
> >>
> >> Vladislav Odintsov
> >>
> >> 
> >> From: ceph-devel-ow...@vger.kernel.org  
> >> on behalf of Franklin M. Siler 
> >> Sent: Friday, February 26, 2016 05:03
> >> To: Xiaoxi Chen
> >> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
> >> ceph-users
> >> Subject: Re: [ceph-users] v0.94.6 Hammer released
> >>
> >> On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:
> >>
> >>> Will we build package for ubuntu 12.04 (Precise)?
> >>> Seems it also doesnt show in the repo
> >>
> >> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
> >> publishes its own packages.
> >>
> >> http://download.ceph.com/debian-hammer/dists/precise/

Re: [ceph-users] v0.94.6 Hammer released

[Nathan Cutler]

The basic idea is to copy the packages that are build by gitbuilders or by the 
buildpackage teuthology task in a central place. Because these packages are 
built, for development versions as well as stable versions[2]. And they are 
tested via teuthology. The packages that are published on http://ceph.com/ are 
rebuilt from scratch, using the process that Alfredo described. This is fine 
for the supported platforms and for the stable releases. But for the 
development releases and the platforms that are no longer supported but still 
built by gibuilders, we could just copy the packages over.

Does that sound sensible ?


Hi Loic:

Community packages for "deprecated" platforms ("deprecated" in the sense 
that the Ceph developers are no longer testing on them) would be 
welcomed by many, I imagine. And the additional workload for the Stable 
Releases team is not large. The question is, where will the packages be 
copied *to*?


--
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Loic Dachary]

Hi Dan & al,

I think it would be relatively simple to have these binaries published as part 
of the current "Stable release" team effort[1]. Essentially doing what you did 
and electing a central place to store these binaries. The trick is to find a 
sustainable way to do this which means having a process that is simple to 
maintain and understand as well as enough of us to maintain that in the long 
run.

The basic idea is to copy the packages that are build by gitbuilders or by the 
buildpackage teuthology task in a central place. Because these packages are 
built, for development versions as well as stable versions[2]. And they are 
tested via teuthology. The packages that are published on http://ceph.com/ are 
rebuilt from scratch, using the process that Alfredo described. This is fine 
for the supported platforms and for the stable releases. But for the 
development releases and the platforms that are no longer supported but still 
built by gibuilders, we could just copy the packages over.

Does that sound sensible ?

[1] http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO
[2] http://gitbuilder.ceph.com/


On 29/02/2016 20:28, Dan van der Ster wrote:
> On Mon, Feb 29, 2016 at 12:30 PM, Odintsov Vladislav  
> wrote:
>> Can you please provide right way for building rpm packages?
> 
> It's documented here:
> http://docs.ceph.com/docs/master/install/build-ceph/#rpm-package-manager
> 
> For 0.94.6 you need to change the .spec file to use .tar.gz (because
> there was no .bz2 published for some reason). And then also grab
> init-ceph.in-fedora.patch from here:
> https://raw.githubusercontent.com/ceph/ceph/master/rpm/init-ceph.in-fedora.patch
> 
> BTW, I've put our build here:
> http://linuxsoft.cern.ch/internal/repos/ceph6-stable/x86_64/os/
> These are unsigned, untested and come with no warranty, no guarantees
> of any sort. And IMHO, no third party build would ever to give that
> warm fuzzy trust-it-with-my-data feeling like a ceph.com build would
> ;)
> 
> Moving forward, it would be great if the required community effort
> could be put to work to get ceph.com el6 (and other) builds. For el6
> in particular there is also the option to help out the Centos Storage
> SIG to produce builds. I don't have a good feeling which direction is
> better ... maybe both.
> 
> -- Dan
> CERN IT Storage Group
> 
> 
>> 
>> Regards,
>>
>> Vladislav Odintsov
>>
>> 
>> From: Shinobu Kinjo 
>> Sent: Monday, February 29, 2016 14:11
>> To: Odintsov Vladislav
>> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; ceph-users; 
>> Sage Weil
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>
>> Can we make any kind of general procedure to make packages so that almost 
>> everyone in community build packages by themselves and reduce developers 
>> work load caused by too much requirement -;
>>
>> Cheers,
>> Shinobu
>>
>> - Original Message -
>> From: "Odintsov Vladislav" 
>> To: "Franklin M. Siler" , "Xiaoxi Chen" 
>> 
>> Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage 
>> Weil" 
>> Sent: Monday, February 29, 2016 6:04:02 PM
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>
>> Hi all,
>>
>> should we build el6 packages ourself or, it's hoped that these packages 
>> would be built officially by community?
>>
>> 
>> Regards,
>>
>> Vladislav Odintsov
>>
>> 
>> From: ceph-devel-ow...@vger.kernel.org  on 
>> behalf of Franklin M. Siler 
>> Sent: Friday, February 26, 2016 05:03
>> To: Xiaoxi Chen
>> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
>> ceph-users
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>
>> On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:
>>
>>> Will we build package for ubuntu 12.04 (Precise)?
>>> Seems it also doesnt show in the repo
>>
>> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
>> publishes its own packages.
>>
>> http://download.ceph.com/debian-hammer/dists/precise/
>>
>> so repo lines for sources.list would be, I think:
>>
>> deb http://download.ceph.com/debian-hammer/ precise main
>> deb-src http://download.ceph.com/debian-hammer/ precise main
>>
>>
>> Cheers,
>>
>> Frank Siler
>> Siler Industrial Analytics
>> 314.799.9405--
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users 

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

Syncing now.
> On 29 Feb 2016, at 15:38, Josef Johansson  wrote:
> 
> I’ll check if I can mirror it though http.
>> On 29 Feb 2016, at 15:31, Josef Johansson > > wrote:
>> 
>> Then we’re all in the same boat.
>> 
>>> On 29 Feb 2016, at 15:30, Florent B >> > wrote:
>>> 
>>> Hi and thank you. But for me, you are out of sync as eu.ceph.com 
>>> . Can't find Infernalis 9.2.1 on your mirror :(
>>> 
>>> On 02/29/2016 03:21 PM, Josef Johansson wrote:
 You could sync from me instead @ se.ceph.com  
 As a start.
 
 Regards
 /Josef
 
> On 29 Feb 2016, at 15:19, Florent B < 
> flor...@coppint.com 
> > wrote:
> 
> I would like to inform you that I have difficulties to set-up a mirror.
> 
> rsync on download.ceph.com  is down
> 
> # rsync download.ceph.com ::
> rsync: failed to connect to download.ceph.com  
> (173.236.253.173):
> Connection timed out (110)
> 
> And eu.ceph.com  is out of sync for a few weeks.
> 
> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
>> Hi,
>> 
>> My PR was merged with a script to mirror Ceph properly:
>> https://github.com/ceph/ceph/tree/master/mirroring 
>> 
>> 
>> Currently there are 3 (official) locations where you can get Ceph:
>> 
>> - download.ceph.com  (Dreamhost, US)
>> - eu.ceph.com  (PCextreme, Netherlands)
>> - au.ceph.com  (Digital Pacific, Australia)
>> 
>> I'm looking for more mirrors to become official mirrors so we can easily
>> distribute Ceph.
>> 
>> Mirrors do go down and it's always nice to have a mirror local to you.
>> 
>> I'd like to have one or more mirrors in Asia, Africa and/or South
>> Ameirca if possible. Anyone able to host there? Other locations are
>> welcome as well!
>> 
>> A few things which are required:
>> 
>> - 1Gbit connection or more
>> - Native IPv4 and IPv6
>> - HTTP access
>> - rsync access
>> - 2TB of storage or more
>> - Monitoring of the mirror/source
>> 
>> You can easily mirror Ceph yourself with this script I wrote:
>> https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh 
>> 
>> 
>> eu.ceph.com  and au.ceph.com  
>> use it to sync from download.ceph.com . If
>> you want to mirror Ceph locally, please pick a mirror local to you.
>> 
>> Please refer to these guidelines:
>> https://github.com/ceph/ceph/tree/master/mirroring#guidelines 
>> 
>> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
 
>>> 
>> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

I’ll check if I can mirror it though http.
> On 29 Feb 2016, at 15:31, Josef Johansson  wrote:
> 
> Then we’re all in the same boat.
> 
>> On 29 Feb 2016, at 15:30, Florent B > > wrote:
>> 
>> Hi and thank you. But for me, you are out of sync as eu.ceph.com 
>> . Can't find Infernalis 9.2.1 on your mirror :(
>> 
>> On 02/29/2016 03:21 PM, Josef Johansson wrote:
>>> You could sync from me instead @ se.ceph.com  
>>> As a start.
>>> 
>>> Regards
>>> /Josef
>>> 
 On 29 Feb 2016, at 15:19, Florent B < 
 flor...@coppint.com 
 > wrote:
 
 I would like to inform you that I have difficulties to set-up a mirror.
 
 rsync on download.ceph.com  is down
 
 # rsync download.ceph.com ::
 rsync: failed to connect to download.ceph.com  
 (173.236.253.173):
 Connection timed out (110)
 
 And eu.ceph.com  is out of sync for a few weeks.
 
 On 01/30/2016 03:14 PM, Wido den Hollander wrote:
> Hi,
> 
> My PR was merged with a script to mirror Ceph properly:
> https://github.com/ceph/ceph/tree/master/mirroring 
> 
> 
> Currently there are 3 (official) locations where you can get Ceph:
> 
> - download.ceph.com  (Dreamhost, US)
> - eu.ceph.com  (PCextreme, Netherlands)
> - au.ceph.com  (Digital Pacific, Australia)
> 
> I'm looking for more mirrors to become official mirrors so we can easily
> distribute Ceph.
> 
> Mirrors do go down and it's always nice to have a mirror local to you.
> 
> I'd like to have one or more mirrors in Asia, Africa and/or South
> Ameirca if possible. Anyone able to host there? Other locations are
> welcome as well!
> 
> A few things which are required:
> 
> - 1Gbit connection or more
> - Native IPv4 and IPv6
> - HTTP access
> - rsync access
> - 2TB of storage or more
> - Monitoring of the mirror/source
> 
> You can easily mirror Ceph yourself with this script I wrote:
> https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh 
> 
> 
> eu.ceph.com  and au.ceph.com  
> use it to sync from download.ceph.com . If
> you want to mirror Ceph locally, please pick a mirror local to you.
> 
> Please refer to these guidelines:
> https://github.com/ceph/ceph/tree/master/mirroring#guidelines 
> 
> 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 
>>> 
>> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

Then we’re all in the same boat.

> On 29 Feb 2016, at 15:30, Florent B  wrote:
> 
> Hi and thank you. But for me, you are out of sync as eu.ceph.com. Can't find 
> Infernalis 9.2.1 on your mirror :(
> 
> On 02/29/2016 03:21 PM, Josef Johansson wrote:
>> You could sync from me instead @ se.ceph.com  
>> As a start.
>> 
>> Regards
>> /Josef
>> 
>>> On 29 Feb 2016, at 15:19, Florent B < 
>>> flor...@coppint.com 
>>> > wrote:
>>> 
>>> I would like to inform you that I have difficulties to set-up a mirror.
>>> 
>>> rsync on download.ceph.com  is down
>>> 
>>> # rsync download.ceph.com ::
>>> rsync: failed to connect to download.ceph.com  
>>> (173.236.253.173):
>>> Connection timed out (110)
>>> 
>>> And eu.ceph.com  is out of sync for a few weeks.
>>> 
>>> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
 Hi,
 
 My PR was merged with a script to mirror Ceph properly:
 https://github.com/ceph/ceph/tree/master/mirroring 
 
 
 Currently there are 3 (official) locations where you can get Ceph:
 
 - download.ceph.com (Dreamhost, US)
 - eu.ceph.com (PCextreme, Netherlands)
 - au.ceph.com (Digital Pacific, Australia)
 
 I'm looking for more mirrors to become official mirrors so we can easily
 distribute Ceph.
 
 Mirrors do go down and it's always nice to have a mirror local to you.
 
 I'd like to have one or more mirrors in Asia, Africa and/or South
 Ameirca if possible. Anyone able to host there? Other locations are
 welcome as well!
 
 A few things which are required:
 
 - 1Gbit connection or more
 - Native IPv4 and IPv6
 - HTTP access
 - rsync access
 - 2TB of storage or more
 - Monitoring of the mirror/source
 
 You can easily mirror Ceph yourself with this script I wrote:
 https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh 
 
 
 eu.ceph.com and au.ceph.com use it to sync from download.ceph.com. If
 you want to mirror Ceph locally, please pick a mirror local to you.
 
 Please refer to these guidelines:
 https://github.com/ceph/ceph/tree/master/mirroring#guidelines 
 
 
>>> 
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>>> 
>> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd suddenly down / connect claims to be / heartbeat_check: no reply

[Oliver Dzombic]

Hi,

i face here some trouble with the cluster.

Suddenly "random" OSD's are getting marked out.

After restarting the OSD on the specific node, its working again.

This happens usually during activated scrubbing/deep scrubbing.

In the logs i can see:

2016-02-29 06:08:58.130376 7fd5dae75700  0 -- 10.0.1.2:0/36459 >>
10.0.0.4:6807/9051245 pipe(0x27488000 sd=58 :60473 s=1 pgs=0 cs=0 l=1
c=0x28b39440).connect claims to be 10.0.0.4:6807/12051245 not
10.0.0.4:6807/9051245 - wrong node!
2016-02-29 06:08:58.130417 7fd5d9961700  0 -- 10.0.1.2:0/36459 >>
10.0.1.4:6803/6002429 pipe(0x2a6c9000 sd=75 :37736 s=1 pgs=0 cs=0 l=1
c=0x2420be40).connect claims to be 10.0.1.4:6803/10002429 not
10.0.1.4:6803/6002429 - wrong node!
2016-02-29 06:08:58.130918 7fd5b1c17700  0 -- 10.0.1.2:0/36459 >>
10.0.0.1:6800/8050402 pipe(0x26834000 sd=74 :37605 s=1 pgs=0 cs=0 l=1
c=0x1f7a9020).connect claims to be 10.0.0.1:6800/9050770 not
10.0.0.1:6800/8050402 - wrong node!
2016-02-29 06:08:58.131266 7fd5be141700  0 -- 10.0.1.2:0/36459 >>
10.0.0.3:6806/9059302 pipe(0x27f07000 sd=76 :48347 s=1 pgs=0 cs=0 l=1
c=0x2371adc0).connect claims to be 10.0.0.3:6806/11059302 not
10.0.0.3:6806/9059302 - wrong node!
2016-02-29 06:08:58.131299 7fd5c1914700  0 -- 10.0.1.2:0/36459 >>
10.0.1.4:6801/9051245 pipe(0x2d288000 sd=100 :33848 s=1 pgs=0 cs=0 l=1
c=0x28b37760).connect claims to be 10.0.1.4:6801/12051245 not
10.0.1.4:6801/9051245 - wrong node!

and

2016-02-29 06:08:59.230754 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
no reply from osd.0 since back 2016-02-29 05:55:26.351951 front
2016-02-29 05:55:26.351951 (cutoff 2016-02-29 06:08:39.230753)
2016-02-29 06:08:59.230761 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
no reply from osd.1 since back 2016-02-29 05:41:59.191341 front
2016-02-29 05:41:59.191341 (cutoff 2016-02-29 06:08:39.230753)
2016-02-29 06:08:59.230765 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
no reply from osd.2 since back 2016-02-29 05:41:59.191341 front
2016-02-29 05:41:59.191341 (cutoff 2016-02-29 06:08:39.230753)
2016-02-29 06:08:59.230769 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
no reply from osd.4 since back 2016-02-29 05:55:30.452505 front
2016-02-29 05:55:30.452505 (cutoff 2016-02-29 06:08:39.230753)
2016-02-29 06:08:59.230773 7fd5c5425700 -1 osd.3 14877 heartbeat_check:
no reply from osd.7 since back 2016-02-29 05:41:52.790422 front
2016-02-29 05:41:52.790422 (cutoff 2016-02-29 06:08:39.230753)


Any idea what could be the trouble of the cluster ?

Thank you !

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mirrors wanted!

[Josef Johansson]

You could sync from me instead @ se.ceph.com  
As a start.

Regards
/Josef

> On 29 Feb 2016, at 15:19, Florent B  wrote:
> 
> I would like to inform you that I have difficulties to set-up a mirror.
> 
> rsync on download.ceph.com is down
> 
> # rsync download.ceph.com::
> rsync: failed to connect to download.ceph.com (173.236.253.173):
> Connection timed out (110)
> 
> And eu.ceph.com is out of sync for a few weeks.
> 
> On 01/30/2016 03:14 PM, Wido den Hollander wrote:
>> Hi,
>> 
>> My PR was merged with a script to mirror Ceph properly:
>> https://github.com/ceph/ceph/tree/master/mirroring
>> 
>> Currently there are 3 (official) locations where you can get Ceph:
>> 
>> - download.ceph.com (Dreamhost, US)
>> - eu.ceph.com (PCextreme, Netherlands)
>> - au.ceph.com (Digital Pacific, Australia)
>> 
>> I'm looking for more mirrors to become official mirrors so we can easily
>> distribute Ceph.
>> 
>> Mirrors do go down and it's always nice to have a mirror local to you.
>> 
>> I'd like to have one or more mirrors in Asia, Africa and/or South
>> Ameirca if possible. Anyone able to host there? Other locations are
>> welcome as well!
>> 
>> A few things which are required:
>> 
>> - 1Gbit connection or more
>> - Native IPv4 and IPv6
>> - HTTP access
>> - rsync access
>> - 2TB of storage or more
>> - Monitoring of the mirror/source
>> 
>> You can easily mirror Ceph yourself with this script I wrote:
>> https://github.com/ceph/ceph/blob/master/mirroring/mirror-ceph.sh
>> 
>> eu.ceph.com and au.ceph.com use it to sync from download.ceph.com. If
>> you want to mirror Ceph locally, please pick a mirror local to you.
>> 
>> Please refer to these guidelines:
>> https://github.com/ceph/ceph/tree/master/mirroring#guidelines
>> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Josef Johansson]

Maybe the reverse is possible, where we as a community lend out computing 
resources that the central build system could use.

> On 29 Feb 2016, at 14:38, Josef Johansson  wrote:
> 
> Hi,
> 
> There is also https://github.com/jordansissel/fpm/wiki 
> 
> 
> I find it quite useful for building deb/rpm.
> 
> What would be useful for the community per se would be if you made a 
> Dockerfile for each type of combination, i.e. Ubuntu trusty / 10.0.3 and so 
> forth.
> 
> That way anyone could just docker run ceph/compile-ubuntu-trusty-10.0.3 and 
> that would be it.
> 
> I don’t think that would even be tough to do.
> 
> I’m unsure how well you can test that it’s not tampered with, but I assume 
> it’s possible to solve, or at least set up trusts between a contributor and 
> the repo. 
> 
> Regards,
> Josef
> 
>> On 29 Feb 2016, at 14:28, Dan van der Ster > > wrote:
>> 
>> On Mon, Feb 29, 2016 at 12:30 PM, Odintsov Vladislav > > wrote:
>>> Can you please provide right way for building rpm packages?
>> 
>> It's documented here:
>> http://docs.ceph.com/docs/master/install/build-ceph/#rpm-package-manager 
>> 
>> 
>> For 0.94.6 you need to change the .spec file to use .tar.gz (because
>> there was no .bz2 published for some reason). And then also grab
>> init-ceph.in-fedora.patch from here:
>> https://raw.githubusercontent.com/ceph/ceph/master/rpm/init-ceph.in-fedora.patch
>>  
>> 
>> 
>> BTW, I've put our build here:
>> http://linuxsoft.cern.ch/internal/repos/ceph6-stable/x86_64/os/ 
>> 
>> These are unsigned, untested and come with no warranty, no guarantees
>> of any sort. And IMHO, no third party build would ever to give that
>> warm fuzzy trust-it-with-my-data feeling like a ceph.com  
>> build would
>> ;)
>> 
>> Moving forward, it would be great if the required community effort
>> could be put to work to get ceph.com  el6 (and other) 
>> builds. For el6
>> in particular there is also the option to help out the Centos Storage
>> SIG to produce builds. I don't have a good feeling which direction is
>> better ... maybe both.
>> 
>> -- Dan
>> CERN IT Storage Group
>> 
>> 
>>> 
>>> Regards,
>>> 
>>> Vladislav Odintsov
>>> 
>>> 
>>> From: Shinobu Kinjo >
>>> Sent: Monday, February 29, 2016 14:11
>>> To: Odintsov Vladislav
>>> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org 
>>> ; ceph-users; Sage Weil
>>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>> 
>>> Can we make any kind of general procedure to make packages so that almost 
>>> everyone in community build packages by themselves and reduce developers 
>>> work load caused by too much requirement -;
>>> 
>>> Cheers,
>>> Shinobu
>>> 
>>> - Original Message -
>>> From: "Odintsov Vladislav" >
>>> To: "Franklin M. Siler" >, 
>>> "Xiaoxi Chen" >
>>> Cc: ceph-de...@vger.kernel.org , 
>>> "ceph-users" >, "Sage 
>>> Weil" >
>>> Sent: Monday, February 29, 2016 6:04:02 PM
>>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>> 
>>> Hi all,
>>> 
>>> should we build el6 packages ourself or, it's hoped that these packages 
>>> would be built officially by community?
>>> 
>>> 
>>> Regards,
>>> 
>>> Vladislav Odintsov
>>> 
>>> 
>>> From: ceph-devel-ow...@vger.kernel.org 
>>>  >> > on behalf of Franklin M. Siler 
>>> >
>>> Sent: Friday, February 26, 2016 05:03
>>> To: Xiaoxi Chen
>>> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org 
>>> ; ceph-users
>>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>>> 
>>> On Feb 25, 2016, at 1839, Xiaoxi Chen >> > wrote:
>>> 
 Will we build package for ubuntu 12.04 (Precise)?
 Seems it also doesnt show in the repo
>>> 
>>> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
>>> publishes its own packages.
>>> 
>>> http://download.ceph.com/debian-hammer/dists/precise/ 
>>> 

Re: [ceph-users] v0.94.6 Hammer released

[Josef Johansson]

Hi,

There is also https://github.com/jordansissel/fpm/wiki 


I find it quite useful for building deb/rpm.

What would be useful for the community per se would be if you made a Dockerfile 
for each type of combination, i.e. Ubuntu trusty / 10.0.3 and so forth.

That way anyone could just docker run ceph/compile-ubuntu-trusty-10.0.3 and 
that would be it.

I don’t think that would even be tough to do.

I’m unsure how well you can test that it’s not tampered with, but I assume it’s 
possible to solve, or at least set up trusts between a contributor and the 
repo. 

Regards,
Josef

> On 29 Feb 2016, at 14:28, Dan van der Ster  wrote:
> 
> On Mon, Feb 29, 2016 at 12:30 PM, Odintsov Vladislav  > wrote:
>> Can you please provide right way for building rpm packages?
> 
> It's documented here:
> http://docs.ceph.com/docs/master/install/build-ceph/#rpm-package-manager 
> 
> 
> For 0.94.6 you need to change the .spec file to use .tar.gz (because
> there was no .bz2 published for some reason). And then also grab
> init-ceph.in-fedora.patch from here:
> https://raw.githubusercontent.com/ceph/ceph/master/rpm/init-ceph.in-fedora.patch
>  
> 
> 
> BTW, I've put our build here:
> http://linuxsoft.cern.ch/internal/repos/ceph6-stable/x86_64/os/ 
> 
> These are unsigned, untested and come with no warranty, no guarantees
> of any sort. And IMHO, no third party build would ever to give that
> warm fuzzy trust-it-with-my-data feeling like a ceph.com  
> build would
> ;)
> 
> Moving forward, it would be great if the required community effort
> could be put to work to get ceph.com  el6 (and other) 
> builds. For el6
> in particular there is also the option to help out the Centos Storage
> SIG to produce builds. I don't have a good feeling which direction is
> better ... maybe both.
> 
> -- Dan
> CERN IT Storage Group
> 
> 
>> 
>> Regards,
>> 
>> Vladislav Odintsov
>> 
>> 
>> From: Shinobu Kinjo 
>> Sent: Monday, February 29, 2016 14:11
>> To: Odintsov Vladislav
>> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; ceph-users; 
>> Sage Weil
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>> 
>> Can we make any kind of general procedure to make packages so that almost 
>> everyone in community build packages by themselves and reduce developers 
>> work load caused by too much requirement -;
>> 
>> Cheers,
>> Shinobu
>> 
>> - Original Message -
>> From: "Odintsov Vladislav" 
>> To: "Franklin M. Siler" , "Xiaoxi Chen" 
>> 
>> Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage 
>> Weil" 
>> Sent: Monday, February 29, 2016 6:04:02 PM
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>> 
>> Hi all,
>> 
>> should we build el6 packages ourself or, it's hoped that these packages 
>> would be built officially by community?
>> 
>> 
>> Regards,
>> 
>> Vladislav Odintsov
>> 
>> 
>> From: ceph-devel-ow...@vger.kernel.org  on 
>> behalf of Franklin M. Siler 
>> Sent: Friday, February 26, 2016 05:03
>> To: Xiaoxi Chen
>> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
>> ceph-users
>> Subject: Re: [ceph-users] v0.94.6 Hammer released
>> 
>> On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:
>> 
>>> Will we build package for ubuntu 12.04 (Precise)?
>>> Seems it also doesnt show in the repo
>> 
>> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
>> publishes its own packages.
>> 
>> http://download.ceph.com/debian-hammer/dists/precise/
>> 
>> so repo lines for sources.list would be, I think:
>> 
>> deb http://download.ceph.com/debian-hammer/ precise main
>> deb-src http://download.ceph.com/debian-hammer/ precise main
>> 
>> 
>> Cheers,
>> 
>> Frank Siler
>> Siler Industrial Analytics
>> 314.799.9405--
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

Re: [ceph-users] v0.94.6 Hammer released

[Dan van der Ster]

On Mon, Feb 29, 2016 at 12:30 PM, Odintsov Vladislav  wrote:
> Can you please provide right way for building rpm packages?

It's documented here:
http://docs.ceph.com/docs/master/install/build-ceph/#rpm-package-manager

For 0.94.6 you need to change the .spec file to use .tar.gz (because
there was no .bz2 published for some reason). And then also grab
init-ceph.in-fedora.patch from here:
https://raw.githubusercontent.com/ceph/ceph/master/rpm/init-ceph.in-fedora.patch

BTW, I've put our build here:
http://linuxsoft.cern.ch/internal/repos/ceph6-stable/x86_64/os/
These are unsigned, untested and come with no warranty, no guarantees
of any sort. And IMHO, no third party build would ever to give that
warm fuzzy trust-it-with-my-data feeling like a ceph.com build would
;)

Moving forward, it would be great if the required community effort
could be put to work to get ceph.com el6 (and other) builds. For el6
in particular there is also the option to help out the Centos Storage
SIG to produce builds. I don't have a good feeling which direction is
better ... maybe both.

-- Dan
CERN IT Storage Group


> 
> Regards,
>
> Vladislav Odintsov
>
> 
> From: Shinobu Kinjo 
> Sent: Monday, February 29, 2016 14:11
> To: Odintsov Vladislav
> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; ceph-users; 
> Sage Weil
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> Can we make any kind of general procedure to make packages so that almost 
> everyone in community build packages by themselves and reduce developers work 
> load caused by too much requirement -;
>
> Cheers,
> Shinobu
>
> - Original Message -
> From: "Odintsov Vladislav" 
> To: "Franklin M. Siler" , "Xiaoxi Chen" 
> 
> Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage 
> Weil" 
> Sent: Monday, February 29, 2016 6:04:02 PM
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> Hi all,
>
> should we build el6 packages ourself or, it's hoped that these packages would 
> be built officially by community?
>
> 
> Regards,
>
> Vladislav Odintsov
>
> 
> From: ceph-devel-ow...@vger.kernel.org  on 
> behalf of Franklin M. Siler 
> Sent: Friday, February 26, 2016 05:03
> To: Xiaoxi Chen
> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
> ceph-users
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:
>
>> Will we build package for ubuntu 12.04 (Precise)?
>> Seems it also doesnt show in the repo
>
> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
> publishes its own packages.
>
> http://download.ceph.com/debian-hammer/dists/precise/
>
> so repo lines for sources.list would be, I think:
>
> deb http://download.ceph.com/debian-hammer/ precise main
> deb-src http://download.ceph.com/debian-hammer/ precise main
>
>
> Cheers,
>
> Frank Siler
> Siler Industrial Analytics
> 314.799.9405--
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Alfredo Deza]

On Mon, Feb 29, 2016 at 6:30 AM, Odintsov Vladislav  wrote:
> Can you please provide right way for building rpm packages?

Building binaries is tricky. CI has a few steps to be able to get
binaries at the end of the process. The actual RPM building is mainly
this portion: 
https://github.com/ceph/ceph-build/blob/master/ceph-build/build/build_rpm#L87-L110

But before doing that you need to install dependencies and run
autogen.sh and configure, etc... as this script does:

https://github.com/ceph/ceph-build/blob/master/ceph-setup/build/build
>
> 
> Regards,
>
> Vladislav Odintsov
>
> 
> From: Shinobu Kinjo 
> Sent: Monday, February 29, 2016 14:11
> To: Odintsov Vladislav
> Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; ceph-users; 
> Sage Weil
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> Can we make any kind of general procedure to make packages so that almost 
> everyone in community build packages by themselves and reduce developers work 
> load caused by too much requirement -;

This is also very tricky to do. All I can offer is to point you (and
others) to the tools that we have and use to create binaries in hopes
that they
can be generalized a bit. For building Ceph we have 3 main steps:

tagging: done with ansible ->
https://github.com/ceph/ceph-build/blob/master/ansible/roles/ceph-release/tasks/main.yml
setup: https://github.com/ceph/ceph-build/blob/master/ceph-setup/build/build
building: depending on the binary type to build, broken up into a few
scripts: https://github.com/ceph/ceph-build/tree/master/ceph-build/build

Once these are built, they are pushed to an HTTP binary API where they
can be consumed later https://chacra.ceph.com/

If a community/generalized effort would be in place I would think a
separate instance of this API should exist so that binary-building
chores
could be split (e.g. by release/distro) and be pushed independently to
this community-owned service.

The service is configured to create the repositories when a new binary
is POSTed so once new ones exist, consuming the repos is trivial.

Everything related to the service is very well documented here:
https://github.com/ceph/chacra#chacra

The most important thing on that service would be the need for space
as repos can get quite large. I wouldn't mind helping out setting up
an instance.

>
> Cheers,
> Shinobu
>
> - Original Message -
> From: "Odintsov Vladislav" 
> To: "Franklin M. Siler" , "Xiaoxi Chen" 
> 
> Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage 
> Weil" 
> Sent: Monday, February 29, 2016 6:04:02 PM
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> Hi all,
>
> should we build el6 packages ourself or, it's hoped that these packages would 
> be built officially by community?
>
> 
> Regards,
>
> Vladislav Odintsov
>
> 
> From: ceph-devel-ow...@vger.kernel.org  on 
> behalf of Franklin M. Siler 
> Sent: Friday, February 26, 2016 05:03
> To: Xiaoxi Chen
> Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
> ceph-users
> Subject: Re: [ceph-users] v0.94.6 Hammer released
>
> On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:
>
>> Will we build package for ubuntu 12.04 (Precise)?
>> Seems it also doesnt show in the repo
>
> The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
> publishes its own packages.
>
> http://download.ceph.com/debian-hammer/dists/precise/
>
> so repo lines for sources.list would be, I think:
>
> deb http://download.ceph.com/debian-hammer/ precise main
> deb-src http://download.ceph.com/debian-hammer/ precise main
>
>
> Cheers,
>
> Frank Siler
> Siler Industrial Analytics
> 314.799.9405--
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Dimitar Boichev]

I am sure that I speak for the majority of people reading this, when I say that 
I didn't get anything from your emails.
Could you provide more debug information ?
Like (but not limited to):
ceph -s 
ceph health details
ceph osd tree
...

I am really having a bad time trying to decode the exact problems.
First you had network issues, then osd failed (in the same time or after?),
Then the cluser did not have enough free space to recover I suppose  ?

Regarding the slow SSD disks, what disks are you using ?
The majority of the issues with SSD disks are because people are using consumer 
grade disks that are not optimized for the load that ceph is producing.

Regards.

Dimitar Boichev
SysAdmin Team Lead
AXSMarine Sofia
Phone: +359 889 22 55 42
Skype: dimitar.boichev.axsmarine
E-mail: dimitar.boic...@axsmarine.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Shinobu Kinjo
Sent: Monday, February 29, 2016 1:32 PM
To: Mario Giammarco
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Help: pool not responding

> What can I do now? How can I debug?

I also would like to know more specific procedure to fix the issue under this 
situation.

Cheers,
Shinobu

- Original Message -
From: "Mario Giammarco" 
To: ceph-users@lists.ceph.com
Sent: Monday, February 29, 2016 6:39:16 PM
Subject: Re: [ceph-users] Help: pool not responding

Ferhat Ozkasgarli  writes:


> 1-) One of the OSD nodes has network problem.
> 2-) Disk failure
> 3-) Not enough resource for OSD nodes
> 4-) Slow OSD Disks

I have replaced cables and switches. I am sure that there are no network 
problems. Disks are SSHD and so they are fast. Nodes memory is empty. I have a 
simple cluster with three nodes just to experiment. One disk brand new has 
failed some time ago and so I added a new osd and deleted the old one using 
official procedure in documentation.

What can I do now? How can I debug?

Thanks again,
Mario

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hammer : rbd info/Status : operation not supported (95) (EC+RBD tier pools)

[Adrien Gillard]

Nope, I have not, as we didn't face the issue for some time now and the
less promotion happening, the better for us : this is a cluster for backups
and the same disks are used for cache and EC pools at the moment.

I will try this if the bug happens again.

On Mon, Feb 29, 2016 at 11:40 AM, Christian Balzer  wrote:

>
> Hello,
>
> On Mon, 29 Feb 2016 11:14:28 +0100 Adrien Gillard wrote:
>
> > We are likely facing the same kind of issue in our infernalis cluster
> > with EC.
> >
> Have you tried what Nick Fisk suggested (and which makes perfect sense to
> me, but I can't test it, no EC pools here)?
>
> That is setting the recency values to 0, which should always force
> promotion.
>
> Christian
>
> > From times to times some of our volumes mounted via the RBD kernel
> > module, will start to "freeze". I can still browse the volume, but the
> > (backup) application using it hangs. I guess it's because it tries to
> > access an object from the EC pool (tracker.ceph.com seems down at the
> > moment so I can't access the details).
> >
> > I can't map / unmap the affected volumes (it rarely concerns all the
> > volumes at the same time). Running 'rbd -p ec-pool info volume-1' gets me
> > the same errors as Frederic ((95) Operation not supported). The sloppy
> > workaround I found is running 'rbd -p ec-pool ls -l' a couple of times.
> > It "magically" gets the volumes in order and they become usable again.
> >
> > Adrien
> >
> > On Sat, Feb 27, 2016 at 12:14 PM, SCHAER Frederic
> >  wrote:
> >
> > > Hi,
> > >
> > > Many thanks.
> > > Just tested : I could see the rbd_id object in the EC pool, and after
> > > promoting it I could see it in the SSD cache pool and could
> > > successfully list the image information, indeed.
> > >
> > > Cheers
> > >
> > > -Message d'origine-
> > > De : Jason Dillaman [mailto:dilla...@redhat.com]
> > > Envoyé : mercredi 24 février 2016 19:16
> > > À : SCHAER Frederic 
> > > Cc : ceph-us...@ceph.com; HONORE Pierre-Francois <
> > > pierre-francois.hon...@cea.fr>
> > > Objet : Re: [ceph-users] ceph hammer : rbd info/Status : operation not
> > > supported (95) (EC+RBD tier pools)
> > >
> > > If you run "rados -p  ls | grep "rbd_id." and
> > > don't see that object, you are experiencing that issue [1].
> > >
> > > You can attempt to work around this issue by running "rados -p
> > > irfu-virt setomapval rbd_id. dummy value" to force-promote
> > > the object to the cache pool.  I haven't tested / verified that will
> > > alleviate the issue, though.
> > >
> > > [1] http://tracker.ceph.com/issues/14762
> > >
> > > --
> > >
> > > Jason Dillaman
> > >
> > > - Original Message -
> > >
> > > > From: "SCHAER Frederic" 
> > > > To: ceph-us...@ceph.com
> > > > Cc: "HONORE Pierre-Francois" 
> > > > Sent: Wednesday, February 24, 2016 12:56:48 PM
> > > > Subject: [ceph-users] ceph hammer : rbd info/Status : operation not
> > > supported
> > > > (95) (EC+RBD tier pools)
> > >
> > > > Hi,
> > >
> > > > I just started testing VMs inside ceph this week, ceph-hammer 0.94-5
> > > here.
> > >
> > > > I built several pools, using pool tiering:
> > > > - A small replicated SSD pool (5 SSDs only, but I thought it’d be
> > > > better
> > > for
> > > > IOPS, I intend to test the difference with disks only)
> > > > - Overlaying a larger EC pool
> > >
> > > > I just have 2 VMs in Ceph… and one of them is breaking something.
> > > > The VM that is not breaking was migrated using qemu-img for creating
> > > > the
> > > ceph
> > > > volume, then migrating the data. Its rbd format is 1 :
> > > > rbd image 'xxx-disk1':
> > > > size 20480 MB in 5120 objects
> > > > order 22 (4096 kB objects)
> > > > block_name_prefix: rb.0.83a49.3d1b58ba
> > > > format: 1
> > >
> > > > The VM that’s failing has a rbd format 2
> > > > this is what I had before things started breaking :
> > > > rbd image 'yyy-disk1':
> > > > size 10240 MB in 2560 objects
> > > > order 22 (4096 kB objects)
> > > > block_name_prefix: rbd_data.8ae1f47398c89
> > > > format: 2
> > > > features: layering, striping
> > > > flags:
> > > > stripe unit: 4096 kB
> > > > stripe count: 1
> > >
> > > > The VM started behaving weirdly with a huge IOwait % during its
> > > > install (that’s to say it did not take long to go wrong ;) )
> > > > Now, this is the only thing that I can get
> > >
> > > > [root@ceph0 ~]# rbd -p irfu-virt info yyy-disk1
> > > > 2016-02-24 18:30:33.213590 7f00e6f6d7c0 -1 librbd::ImageCtx: error
> > > reading
> > > > image id: (95) Operation not supported
> > > > rbd: error opening image yyy-disk1: (95) Operation not supported
> > >
> > > > One thing to note : the VM * IS STILL * working : I can still do disk
> > > > operations, apparently.
> > > > During the VM installation, I realized I wrongly set the target SSD
> > > caching
> > > > size to 100Mbytes, instead of 100Gbytes, and ceph complained it was
> > > almost

Re: [ceph-users] Help: pool not responding

[Shinobu Kinjo]

> What can I do now? How can I debug?

I also would like to know more specific procedure to fix the issue under this 
situation.

Cheers,
Shinobu

- Original Message -
From: "Mario Giammarco" 
To: ceph-users@lists.ceph.com
Sent: Monday, February 29, 2016 6:39:16 PM
Subject: Re: [ceph-users] Help: pool not responding

Ferhat Ozkasgarli  writes:


> 1-) One of the OSD nodes has network problem.
> 2-) Disk failure
> 3-) Not enough resource for OSD nodes
> 4-) Slow OSD Disks

I have replaced cables and switches. I am sure that there are no network
problems. Disks are SSHD and so they are fast. Nodes memory is empty. I have
a simple cluster with three nodes just to experiment. One disk brand new has
failed some time ago and so I added a new osd and deleted the old one using
official procedure in documentation.

What can I do now? How can I debug?

Thanks again,
Mario

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Odintsov Vladislav]

Can you please provide right way for building rpm packages?


Regards,

Vladislav Odintsov


From: Shinobu Kinjo 
Sent: Monday, February 29, 2016 14:11
To: Odintsov Vladislav
Cc: Franklin M. Siler; Xiaoxi Chen; ceph-de...@vger.kernel.org; ceph-users; 
Sage Weil
Subject: Re: [ceph-users] v0.94.6 Hammer released

Can we make any kind of general procedure to make packages so that almost 
everyone in community build packages by themselves and reduce developers work 
load caused by too much requirement -;

Cheers,
Shinobu

- Original Message -
From: "Odintsov Vladislav" 
To: "Franklin M. Siler" , "Xiaoxi Chen" 

Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage Weil" 

Sent: Monday, February 29, 2016 6:04:02 PM
Subject: Re: [ceph-users] v0.94.6 Hammer released

Hi all,

should we build el6 packages ourself or, it's hoped that these packages would 
be built officially by community?


Regards,

Vladislav Odintsov


From: ceph-devel-ow...@vger.kernel.org  on 
behalf of Franklin M. Siler 
Sent: Friday, February 26, 2016 05:03
To: Xiaoxi Chen
Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
ceph-users
Subject: Re: [ceph-users] v0.94.6 Hammer released

On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:

> Will we build package for ubuntu 12.04 (Precise)?
> Seems it also doesnt show in the repo

The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
publishes its own packages.

http://download.ceph.com/debian-hammer/dists/precise/

so repo lines for sources.list would be, I think:

deb http://download.ceph.com/debian-hammer/ precise main
deb-src http://download.ceph.com/debian-hammer/ precise main


Cheers,

Frank Siler
Siler Industrial Analytics
314.799.9405--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph and systemd

[zorg]

Hi can someone can just explain how ceph is organize with systemd

I can see tha tfor each osd there a unit file in like this
/etc/systemd/system/ceph.target.wants/ceph-osd@0.service
wich is a symblink to /lib/systemd/system/ceph-osd@.service

but there is other service start for ceph like
sys-devices-pci:00-:00:01.0-:04:00.0-host0-target0:0:0-0:0:0:1-block-sdb-sdb1.device 
loaded active plugged   LOGICAL_VOLUME ceph\x20data
sys-devices-pci:00-:00:01.0-:04:00.0-host0-target0:0:0-0:0:0:1-block-sdb-sdb2.device 
loaded active plugged   LOGICAL_VOLUME ceph\x20journal


and
var-lib-ceph-osd-ceph\x2d0.mount loaded active mounted   
/var/lib/ceph/osd/ceph-0


That in can't find anywhere in my system and don't really understand how 
they are start




Thanks for your explanation

--
probeSys - spécialiste GNU/Linux
site web : http://www.probesys.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Shinobu Kinjo]

Can we make any kind of general procedure to make packages so that almost 
everyone in community build packages by themselves and reduce developers work 
load caused by too much requirement -;

Cheers,
Shinobu

- Original Message -
From: "Odintsov Vladislav" 
To: "Franklin M. Siler" , "Xiaoxi Chen" 

Cc: ceph-de...@vger.kernel.org, "ceph-users" , "Sage Weil" 

Sent: Monday, February 29, 2016 6:04:02 PM
Subject: Re: [ceph-users] v0.94.6 Hammer released

Hi all,

should we build el6 packages ourself or, it's hoped that these packages would 
be built officially by community?


Regards,

Vladislav Odintsov


From: ceph-devel-ow...@vger.kernel.org  on 
behalf of Franklin M. Siler 
Sent: Friday, February 26, 2016 05:03
To: Xiaoxi Chen
Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
ceph-users
Subject: Re: [ceph-users] v0.94.6 Hammer released

On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:

> Will we build package for ubuntu 12.04 (Precise)?
> Seems it also doesnt show in the repo

The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
publishes its own packages.

http://download.ceph.com/debian-hammer/dists/precise/

so repo lines for sources.list would be, I think:

deb http://download.ceph.com/debian-hammer/ precise main
deb-src http://download.ceph.com/debian-hammer/ precise main


Cheers,

Frank Siler
Siler Industrial Analytics
314.799.9405--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hammer : rbd info/Status : operation not supported (95) (EC+RBD tier pools)

[Christian Balzer]

Hello,

On Mon, 29 Feb 2016 11:14:28 +0100 Adrien Gillard wrote:

> We are likely facing the same kind of issue in our infernalis cluster
> with EC.
>
Have you tried what Nick Fisk suggested (and which makes perfect sense to
me, but I can't test it, no EC pools here)?

That is setting the recency values to 0, which should always force
promotion.

Christian
 
> From times to times some of our volumes mounted via the RBD kernel
> module, will start to "freeze". I can still browse the volume, but the
> (backup) application using it hangs. I guess it's because it tries to
> access an object from the EC pool (tracker.ceph.com seems down at the
> moment so I can't access the details).
> 
> I can't map / unmap the affected volumes (it rarely concerns all the
> volumes at the same time). Running 'rbd -p ec-pool info volume-1' gets me
> the same errors as Frederic ((95) Operation not supported). The sloppy
> workaround I found is running 'rbd -p ec-pool ls -l' a couple of times.
> It "magically" gets the volumes in order and they become usable again.
> 
> Adrien
> 
> On Sat, Feb 27, 2016 at 12:14 PM, SCHAER Frederic
>  wrote:
> 
> > Hi,
> >
> > Many thanks.
> > Just tested : I could see the rbd_id object in the EC pool, and after
> > promoting it I could see it in the SSD cache pool and could
> > successfully list the image information, indeed.
> >
> > Cheers
> >
> > -Message d'origine-
> > De : Jason Dillaman [mailto:dilla...@redhat.com]
> > Envoyé : mercredi 24 février 2016 19:16
> > À : SCHAER Frederic 
> > Cc : ceph-us...@ceph.com; HONORE Pierre-Francois <
> > pierre-francois.hon...@cea.fr>
> > Objet : Re: [ceph-users] ceph hammer : rbd info/Status : operation not
> > supported (95) (EC+RBD tier pools)
> >
> > If you run "rados -p  ls | grep "rbd_id." and
> > don't see that object, you are experiencing that issue [1].
> >
> > You can attempt to work around this issue by running "rados -p
> > irfu-virt setomapval rbd_id. dummy value" to force-promote
> > the object to the cache pool.  I haven't tested / verified that will
> > alleviate the issue, though.
> >
> > [1] http://tracker.ceph.com/issues/14762
> >
> > --
> >
> > Jason Dillaman
> >
> > - Original Message -
> >
> > > From: "SCHAER Frederic" 
> > > To: ceph-us...@ceph.com
> > > Cc: "HONORE Pierre-Francois" 
> > > Sent: Wednesday, February 24, 2016 12:56:48 PM
> > > Subject: [ceph-users] ceph hammer : rbd info/Status : operation not
> > supported
> > > (95) (EC+RBD tier pools)
> >
> > > Hi,
> >
> > > I just started testing VMs inside ceph this week, ceph-hammer 0.94-5
> > here.
> >
> > > I built several pools, using pool tiering:
> > > - A small replicated SSD pool (5 SSDs only, but I thought it’d be
> > > better
> > for
> > > IOPS, I intend to test the difference with disks only)
> > > - Overlaying a larger EC pool
> >
> > > I just have 2 VMs in Ceph… and one of them is breaking something.
> > > The VM that is not breaking was migrated using qemu-img for creating
> > > the
> > ceph
> > > volume, then migrating the data. Its rbd format is 1 :
> > > rbd image 'xxx-disk1':
> > > size 20480 MB in 5120 objects
> > > order 22 (4096 kB objects)
> > > block_name_prefix: rb.0.83a49.3d1b58ba
> > > format: 1
> >
> > > The VM that’s failing has a rbd format 2
> > > this is what I had before things started breaking :
> > > rbd image 'yyy-disk1':
> > > size 10240 MB in 2560 objects
> > > order 22 (4096 kB objects)
> > > block_name_prefix: rbd_data.8ae1f47398c89
> > > format: 2
> > > features: layering, striping
> > > flags:
> > > stripe unit: 4096 kB
> > > stripe count: 1
> >
> > > The VM started behaving weirdly with a huge IOwait % during its
> > > install (that’s to say it did not take long to go wrong ;) )
> > > Now, this is the only thing that I can get
> >
> > > [root@ceph0 ~]# rbd -p irfu-virt info yyy-disk1
> > > 2016-02-24 18:30:33.213590 7f00e6f6d7c0 -1 librbd::ImageCtx: error
> > reading
> > > image id: (95) Operation not supported
> > > rbd: error opening image yyy-disk1: (95) Operation not supported
> >
> > > One thing to note : the VM * IS STILL * working : I can still do disk
> > > operations, apparently.
> > > During the VM installation, I realized I wrongly set the target SSD
> > caching
> > > size to 100Mbytes, instead of 100Gbytes, and ceph complained it was
> > almost
> > > full :
> > > health HEALTH_WARN
> > > 'ssd-hot-irfu-virt' at/near target max
> >
> > > My question is…… am I facing the bug as reported in this list thread
> > > with title “Possible Cache Tier Bug - Can someone confirm” ?
> > > Or did I do something wrong ?
> >
> > > The libvirt and kvm that are writing into ceph are the following :
> > > libvirt -1.2.17-13.el7_2.3.x86_64
> > > qemu- kvm -1.5.3-105.el7_2.3.x86_64
> >
> > > Any idea how I could recover the VM file, if possible ?
> > > Please note I have no problem with deleting the VM and rebuilding
> > > it, I

Re: [ceph-users] ceph hammer : rbd info/Status : operation not supported (95) (EC+RBD tier pools)

[Adrien Gillard]

We are likely facing the same kind of issue in our infernalis cluster with
EC.

>From times to times some of our volumes mounted via the RBD kernel module,
will start to "freeze". I can still browse the volume, but the (backup)
application using it hangs. I guess it's because it tries to access an
object from the EC pool (tracker.ceph.com seems down at the moment so I
can't access the details).

I can't map / unmap the affected volumes (it rarely concerns all the
volumes at the same time). Running 'rbd -p ec-pool info volume-1' gets me
the same errors as Frederic ((95) Operation not supported). The sloppy
workaround I found is running 'rbd -p ec-pool ls -l' a couple of times. It
"magically" gets the volumes in order and they become usable again.

Adrien

On Sat, Feb 27, 2016 at 12:14 PM, SCHAER Frederic 
wrote:

> Hi,
>
> Many thanks.
> Just tested : I could see the rbd_id object in the EC pool, and after
> promoting it I could see it in the SSD cache pool and could successfully
> list the image information, indeed.
>
> Cheers
>
> -Message d'origine-
> De : Jason Dillaman [mailto:dilla...@redhat.com]
> Envoyé : mercredi 24 février 2016 19:16
> À : SCHAER Frederic 
> Cc : ceph-us...@ceph.com; HONORE Pierre-Francois <
> pierre-francois.hon...@cea.fr>
> Objet : Re: [ceph-users] ceph hammer : rbd info/Status : operation not
> supported (95) (EC+RBD tier pools)
>
> If you run "rados -p  ls | grep "rbd_id." and don't
> see that object, you are experiencing that issue [1].
>
> You can attempt to work around this issue by running "rados -p irfu-virt
> setomapval rbd_id. dummy value" to force-promote the object to
> the cache pool.  I haven't tested / verified that will alleviate the issue,
> though.
>
> [1] http://tracker.ceph.com/issues/14762
>
> --
>
> Jason Dillaman
>
> - Original Message -
>
> > From: "SCHAER Frederic" 
> > To: ceph-us...@ceph.com
> > Cc: "HONORE Pierre-Francois" 
> > Sent: Wednesday, February 24, 2016 12:56:48 PM
> > Subject: [ceph-users] ceph hammer : rbd info/Status : operation not
> supported
> > (95) (EC+RBD tier pools)
>
> > Hi,
>
> > I just started testing VMs inside ceph this week, ceph-hammer 0.94-5
> here.
>
> > I built several pools, using pool tiering:
> > - A small replicated SSD pool (5 SSDs only, but I thought it’d be better
> for
> > IOPS, I intend to test the difference with disks only)
> > - Overlaying a larger EC pool
>
> > I just have 2 VMs in Ceph… and one of them is breaking something.
> > The VM that is not breaking was migrated using qemu-img for creating the
> ceph
> > volume, then migrating the data. Its rbd format is 1 :
> > rbd image 'xxx-disk1':
> > size 20480 MB in 5120 objects
> > order 22 (4096 kB objects)
> > block_name_prefix: rb.0.83a49.3d1b58ba
> > format: 1
>
> > The VM that’s failing has a rbd format 2
> > this is what I had before things started breaking :
> > rbd image 'yyy-disk1':
> > size 10240 MB in 2560 objects
> > order 22 (4096 kB objects)
> > block_name_prefix: rbd_data.8ae1f47398c89
> > format: 2
> > features: layering, striping
> > flags:
> > stripe unit: 4096 kB
> > stripe count: 1
>
> > The VM started behaving weirdly with a huge IOwait % during its install
> > (that’s to say it did not take long to go wrong ;) )
> > Now, this is the only thing that I can get
>
> > [root@ceph0 ~]# rbd -p irfu-virt info yyy-disk1
> > 2016-02-24 18:30:33.213590 7f00e6f6d7c0 -1 librbd::ImageCtx: error
> reading
> > image id: (95) Operation not supported
> > rbd: error opening image yyy-disk1: (95) Operation not supported
>
> > One thing to note : the VM * IS STILL * working : I can still do disk
> > operations, apparently.
> > During the VM installation, I realized I wrongly set the target SSD
> caching
> > size to 100Mbytes, instead of 100Gbytes, and ceph complained it was
> almost
> > full :
> > health HEALTH_WARN
> > 'ssd-hot-irfu-virt' at/near target max
>
> > My question is…… am I facing the bug as reported in this list thread with
> > title “Possible Cache Tier Bug - Can someone confirm” ?
> > Or did I do something wrong ?
>
> > The libvirt and kvm that are writing into ceph are the following :
> > libvirt -1.2.17-13.el7_2.3.x86_64
> > qemu- kvm -1.5.3-105.el7_2.3.x86_64
>
> > Any idea how I could recover the VM file, if possible ?
> > Please note I have no problem with deleting the VM and rebuilding it, I
> just
> > spawned it to test.
> > As a matter of fact, I just “virsh destroyed” the VM, to see if I could
> start
> > it again… and I cant :
>
> > # virsh start yyy
> > error: Failed to start domain yyy
> > error: internal error: process exited while connecting to monitor:
> > 2016-02-24T17:49:59.262170Z qemu-kvm: -drive
> >
> file=rbd:irfu-virt/yyy-disk1:id=irfu-virt:key=***==:auth_supported=cephx\;none:mon_host=_\:6789,if=none,id=drive-virtio-disk0,format=raw:
> > error reading header from yyy-disk1
> > 2016-02-24T17:49:59.263743Z 

Re: [ceph-users] State of Ceph documention

[John Spray]

On Fri, Feb 26, 2016 at 10:49 PM, Nigel Williams
 wrote:
> On Fri, Feb 26, 2016 at 11:28 PM, John Spray  wrote:
>> Some projects have big angry warning banners at the top of their
>> master branch documentation, I think perhaps we should do that too,
>> and at the same time try to find a way to steer google hits to the
>> latest stable branch docs rather than to master.
>
> Are there reasons to "publish" the version-less master? Maybe I've
> missed the explanation for why master is necessary, but could it be
> completely hidden?

Same reason we have packages for it, I guess.  Anybody playing with or
doing development on master would want to see the latest docs.  It
just doesn't need to be the first thing that end users see.

John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help: pool not responding

[Mario Giammarco]

Ferhat Ozkasgarli  writes:


> 1-) One of the OSD nodes has network problem.
> 2-) Disk failure
> 3-) Not enough resource for OSD nodes
> 4-) Slow OSD Disks

I have replaced cables and switches. I am sure that there are no network
problems. Disks are SSHD and so they are fast. Nodes memory is empty. I have
a simple cluster with three nodes just to experiment. One disk brand new has
failed some time ago and so I added a new osd and deleted the old one using
official procedure in documentation.

What can I do now? How can I debug?

Thanks again,
Mario

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Observations with a SSD based pool under Hammer

[Mark Nelson]

On 02/29/2016 02:37 AM, Christian Balzer wrote:

On Mon, 29 Feb 2016 02:15:28 -0500 (EST) Shinobu Kinjo wrote:


Christian,


Ceph: no tuning or significant/relevant config changes, OSD FS is Ext4,
Ceph journal is inline (journal file).


Quick question. Is there any reason you selected Ext4?


https://www.mail-archive.com/ceph-users@lists.ceph.com/msg08619.html

XFS has historically always been slower for me, whenever I tested it.
Now with Ceph there are several optimizations (in the latest versions, not
when I started) for XFS.
However there also were (near-lethal) XFS bugs exposed by Ceph.

Lastly XFS seems to fragment faster than Ext4, definitely when used as OSD
FS.
My badly overloaded old production cluster with 80 files/objects per
OSD has a e4defrag score of 11 (up to 30 is fine) after running for nearly
2 years.
My newer Ext4 OSDs are formatted so that they have LARGE blocks, so the
chance for fragmentation is even lower.
I managed to severely fragment my XFS based test cluster with far less,
synthetic usage.


You may see a lot less fragmentation with the filestore_xfs_extsize set 
to true.  We did back when we were testing it for hammer.  The problem 
is that on one of the test clusters inside RH it was causing a 
sequential write throughput regression vs firefly.  It doesn't really 
make any sense why that would be, but it was pretty clearly causing it 
after bisecting and narrowing it down to the commit that enabled it.


Mark



Now the SSD based OSDs could have been formatted with XFS I suppose as
the last point doesn't apply to them, but I like consistency in my setups.




Christian


Cheers,
Shinobu

- Original Message -
From: "Christian Balzer" 
To: ceph-users@lists.ceph.com
Sent: Thursday, February 25, 2016 12:10:41 PM
Subject: [ceph-users] Observations with a SSD based pool under Hammer


Hello,

For posterity and of course to ask some questions, here are my
experiences with a pure SSD pool.

SW: Debian Jessie, Ceph Hammer 0.94.5.

HW:
2 nodes (thus replication of 2) with each:
2x E5-2623 CPUs
64GB RAM
4x DC S3610 800GB SSDs
Infiniband (IPoIB) network

Ceph: no tuning or significant/relevant config changes, OSD FS is Ext4,
Ceph journal is inline (journal file).

Performance:
A test run with "rados -p cache  bench 30 write -t 32" (4MB blocks) gives
me about 620MB/s, the storage nodes are I/O bound (all SSDs are 100% busy
according to atop) and this meshes nicely with the speeds I saw when
testing the individual SSDs with fio before involving Ceph.

To elaborate on that, an individual SSD of that type can do about 500MB/s
sequential writes, so ideally you would see 1GB/s writes with Ceph
(500*8/2(replication)/2(journal on same disk).
However my experience tells me that other activities (FS journals,
leveldb PG updates, etc) impact things as well.

A test run with "rados -p cache  bench 30 write -t 32 -b 4096" (4KB
blocks) gives me about 7200 IOPS, the SSDs are about 40% busy.
All OSD processes are using about 2 cores and the OS another 2, but that
leaves about 6 cores unused (MHz on all cores scales to max during the
test run).
Closer inspection with all CPUs being displayed in atop shows that no
single core is fully used, they all average around 40% and even the
busiest ones (handling IRQs) still have ample capacity available.
I'm wondering if this an indication of insufficient parallelism or if
it's latency of sorts.
I'm aware of the many tuning settings for SSD based OSDs, however I was
expecting to run into a CPU wall first and foremost.


Write amplification:
10 second rados bench with 4MB blocks, 6348MB written in total.
nand-writes per SSD:118*32MB=3776MB.
30208MB total written to all SSDs.
Amplification:4.75

Very close to what you would expect with a replication of 2 and journal
on same disk.


10 second rados bench with 4KB blocks, 219MB written in total.
nand-writes per SSD:41*32MB=1312MB.
10496MB total written to all SSDs.
Amplification:48!!!

Le ouch.
In my use case with rbd cache on all VMs I expect writes to be rather
large for the most part and not like this extreme example.
But as I wrote the last time I did this kind of testing, this is an area
where caveat emptor most definitely applies when planning and buying
SSDs. And where the Ceph code could probably do with some attention.

Regards,

Christian




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.94.6 Hammer released

[Odintsov Vladislav]

Hi all,

should we build el6 packages ourself or, it's hoped that these packages would 
be built officially by community?


Regards,

Vladislav Odintsov


From: ceph-devel-ow...@vger.kernel.org  on 
behalf of Franklin M. Siler 
Sent: Friday, February 26, 2016 05:03
To: Xiaoxi Chen
Cc: Alfredo Deza; Dan van der Ster; Sage Weil; ceph-de...@vger.kernel.org; 
ceph-users
Subject: Re: [ceph-users] v0.94.6 Hammer released

On Feb 25, 2016, at 1839, Xiaoxi Chen  wrote:

> Will we build package for ubuntu 12.04 (Precise)?
> Seems it also doesnt show in the repo

The Ceph packages provided by Ubuntu are old.  However, the Ceph project 
publishes its own packages.

http://download.ceph.com/debian-hammer/dists/precise/

so repo lines for sources.list would be, I think:

deb http://download.ceph.com/debian-hammer/ precise main
deb-src http://download.ceph.com/debian-hammer/ precise main


Cheers,

Frank Siler
Siler Industrial Analytics
314.799.9405--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Observations with a SSD based pool under Hammer

[Christian Balzer]

On Mon, 29 Feb 2016 02:15:28 -0500 (EST) Shinobu Kinjo wrote:

> Christian,
> 
> > Ceph: no tuning or significant/relevant config changes, OSD FS is Ext4,
> > Ceph journal is inline (journal file).
> 
> Quick question. Is there any reason you selected Ext4?
> 
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg08619.html

XFS has historically always been slower for me, whenever I tested it.
Now with Ceph there are several optimizations (in the latest versions, not
when I started) for XFS.
However there also were (near-lethal) XFS bugs exposed by Ceph.

Lastly XFS seems to fragment faster than Ext4, definitely when used as OSD
FS.
My badly overloaded old production cluster with 80 files/objects per
OSD has a e4defrag score of 11 (up to 30 is fine) after running for nearly
2 years.
My newer Ext4 OSDs are formatted so that they have LARGE blocks, so the
chance for fragmentation is even lower.
I managed to severely fragment my XFS based test cluster with far less,
synthetic usage.

Now the SSD based OSDs could have been formatted with XFS I suppose as
the last point doesn't apply to them, but I like consistency in my setups.

Christian

> Cheers,
> Shinobu
> 
> - Original Message -
> From: "Christian Balzer" 
> To: ceph-users@lists.ceph.com
> Sent: Thursday, February 25, 2016 12:10:41 PM
> Subject: [ceph-users] Observations with a SSD based pool under Hammer
> 
> 
> Hello, 
> 
> For posterity and of course to ask some questions, here are my
> experiences with a pure SSD pool.
> 
> SW: Debian Jessie, Ceph Hammer 0.94.5.
> 
> HW:
> 2 nodes (thus replication of 2) with each: 
> 2x E5-2623 CPUs
> 64GB RAM
> 4x DC S3610 800GB SSDs
> Infiniband (IPoIB) network
> 
> Ceph: no tuning or significant/relevant config changes, OSD FS is Ext4,
> Ceph journal is inline (journal file).
> 
> Performance:
> A test run with "rados -p cache  bench 30 write -t 32" (4MB blocks) gives
> me about 620MB/s, the storage nodes are I/O bound (all SSDs are 100% busy
> according to atop) and this meshes nicely with the speeds I saw when
> testing the individual SSDs with fio before involving Ceph.
> 
> To elaborate on that, an individual SSD of that type can do about 500MB/s
> sequential writes, so ideally you would see 1GB/s writes with Ceph
> (500*8/2(replication)/2(journal on same disk).
> However my experience tells me that other activities (FS journals,
> leveldb PG updates, etc) impact things as well.
> 
> A test run with "rados -p cache  bench 30 write -t 32 -b 4096" (4KB
> blocks) gives me about 7200 IOPS, the SSDs are about 40% busy.
> All OSD processes are using about 2 cores and the OS another 2, but that
> leaves about 6 cores unused (MHz on all cores scales to max during the
> test run). 
> Closer inspection with all CPUs being displayed in atop shows that no
> single core is fully used, they all average around 40% and even the
> busiest ones (handling IRQs) still have ample capacity available.
> I'm wondering if this an indication of insufficient parallelism or if
> it's latency of sorts.
> I'm aware of the many tuning settings for SSD based OSDs, however I was
> expecting to run into a CPU wall first and foremost.
> 
> 
> Write amplification:
> 10 second rados bench with 4MB blocks, 6348MB written in total. 
> nand-writes per SSD:118*32MB=3776MB. 
> 30208MB total written to all SSDs.
> Amplification:4.75
> 
> Very close to what you would expect with a replication of 2 and journal
> on same disk.
> 
> 
> 10 second rados bench with 4KB blocks, 219MB written in total. 
> nand-writes per SSD:41*32MB=1312MB. 
> 10496MB total written to all SSDs.
> Amplification:48!!!
> 
> Le ouch. 
> In my use case with rbd cache on all VMs I expect writes to be rather
> large for the most part and not like this extreme example. 
> But as I wrote the last time I did this kind of testing, this is an area
> where caveat emptor most definitely applies when planning and buying
> SSDs. And where the Ceph code could probably do with some attention.
>  
> Regards,
> 
> Christian


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com