Re: [ceph-users] Erasure Coding - FPGA / Hardware Acceleration

[Brett Niver]

Also the picture I saw at Cephalocon - which could have been
inaccurate, looked to me as if it multiplied the data path.

On Fri, Jun 14, 2019 at 8:27 AM Janne Johansson  wrote:
>
> Den fre 14 juni 2019 kl 13:58 skrev Sean Redmond :
>>
>> Hi Ceph-Uers,
>> I noticed that Soft Iron now have hardware acceleration for Erasure 
>> Coding[1], this is interesting as the CPU overhead can be a problem in 
>> addition to the extra disk I/O required for EC pools.
>> Does anyone know if any other work is ongoing to support generic FPGA 
>> Hardware Acceleration for EC pools, or if this is just a vendor specific 
>> feature.
>>
>> [1] 
>> https://www.theregister.co.uk/2019/05/20/softiron_unleashes_accepherator_an_erasure_coding_accelerator_for_ceph/
>
>
> Are there numbers anywhere to see how "tough" on a CPU it would be to 
> calculate an EC code compared to "writing a sector to
> a disk on a remote server and getting an ack back" ? To my very untrained 
> eye, it seems like a very small part of the whole picture,
> especially if you are meant to buy a ton of cards to do it.
>
> --
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Quota and ACL support

[Brett Niver]

+Ilya

On Mon, Aug 27, 2018 at 10:53 AM Oliver Freyermuth <
freyerm...@physik.uni-bonn.de> wrote:

> Thanks for the replies.
>
> Am 27.08.18 um 19:25 schrieb Patrick Donnelly:
> > On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth
> >  wrote:
> >> These features are critical for us, so right now we use the Fuse
> client. My hope is CentOS 8 will use a recent enough kernel
> >> to get those features automatically, though.
> >
> > Your cluster needs to be running Mimic and Linux v4.17+.
> >
> > See also: https://github.com/ceph/ceph/pull/23728/files
> >
>
> Yes, I know that it's part of the official / vanilla kernel as of 4.17.
> However, I was wondering whether this functionality is also likely to be
> backported to the RedHat-maintained kernel which is also used in CentOS 7?
> Even though the kernel version is "stone-aged", it matches CentOS 7's
> userspace and RedHat is taking good care to implement fixes.
>
> Seeing that even features are backported, it would be really helpful if
> also this functionality would appear as part of CentOS 7.6 / 7.7,
> especially since CentOS 8 still appears to be quite some time away.
>
> Cheers,
> Oliver
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Testing Weekly Tomorrow — With Kubernetes/Install discussion

[Brett Niver]

Great discussion!


On Thu, Aug 23, 2018 at 12:38 PM, Gregory Farnum  wrote:
> And the recording from this session is now available at
> https://www.youtube.com/watch?v=0WHHTjdgarQ
>
> We didn't come to any conclusions but I think we've got a good
> understanding of the motivations and concerns, along with several
> options and some of the constraints of each.
> -Greg
>
> On Tue, Aug 21, 2018 at 5:06 PM, Gregory Farnum  wrote:
>> Hey all,
>> We've had the testing weekly call going on for several months now
>> (URL: https://ceph.com/testing/) and some people have found it useful,
>> but we haven't gotten many new attendees so here's a reminder:
>> Wednesday 8AM Pacific time, there's a BlueJeans session to discuss
>> anything about testing Ceph (particularly, but not exclusively,
>> teuthology). Etherpad containing occasionally-updated notes is at
>> https://pad.ceph.com/p/community-testing-weekly
>>
>> But this week, there's an extra-special bonus topic! We're going to
>> discuss options for updating our test installation strategy so we can
>> include ceph-ansible/DeepSea/Rook testing in our regular suites, and
>> options for working with Kubernetes going forward (inside the tests?
>> Outside them as part of the framework? So many choices!). If you're
>> interested in helping shape how we test Ceph going forward, don't miss
>> it! :)
>> Thanks,
>> -Greg
>>
>> To join the Meeting:
>> https://bluejeans.com/185197632
>> To join via phone :
>> 1)  Dial:
>>  408-915-6466 (United States)
>>  (see all numbers - https://www.redhat.com/en/conference-numbers)
>> 2)  Enter Conference ID : 185197632
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recovery after datacenter outage

[Brett Niver]

+Paul


On Mon, Jun 25, 2018 at 5:14 AM, Christian Zunker
 wrote:
> Hi Jason,
>
> your guesses were correct. Thank you for your support.
>
> Just in case, someone else stumbles upon this thread, some more links:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020722.html
> http://docs.ceph.com/docs/luminous/rados/operations/user-management/#authorization-capabilities
> http://docs.ceph.com/docs/luminous/rbd/rbd-openstack/#setup-ceph-client-authentication
> https://github.com/ceph/ceph/pull/15991
>
> Jason Dillaman  schrieb am Fr., 22. Juni 2018 um 22:58
> Uhr:
>>
>> It sounds like your OpenStack users do not have the correct caps to
>> blacklist dead clients. See step 6 in the upgrade section of Luminous’
>> release notes or (preferably) use the new “profile rbd”-style caps if you
>> don’t use older clients.
>>
>> The reason why repairing the object map seemed to fix everything was
>> because I suspect you performed the op using the admin user, which had the
>> caps necessary to blacklist the dead clients and clean up the dirty
>> exclusive lock on the image.
>>
>> On Fri, Jun 22, 2018 at 4:47 PM Gregory Farnum  wrote:
>>>
>>> On Fri, Jun 22, 2018 at 2:26 AM Christian Zunker
>>>  wrote:

 Hi List,

 we are running a ceph cluster (12.2.5) as backend to our OpenStack
 cloud.

 Yesterday our datacenter had a power outage. As this wouldn't be enough,
 we also had a separated ceph cluster because of networking problems.

 First of all thanks a lot to the ceph developers. After the network was
 back to normal, ceph recovered itself. You saved us from a lot of downtime,
 lack of sleep and insanity.

 Now to our problem/question:
 After ceph recovered, we tried to bring up our VMs. They have cinder
 volumes saved in ceph. All VMs didn't start because of I/O problems during
 start:
 [4.393246] JBD2: recovery failed
 [4.395949] EXT4-fs (vda1): error loading journal
 [4.400811] VFS: Dirty inode writeback failed for block device vda1
 (err=-5).
 mount: mounting /dev/vda1 on /root failed: Input/output error
 done.
 Begin: Running /scripts/local-bottom ... done.
 Begin: Running /scripts/init-bottom ... mount: mounting /dev on
 /root/dev failed: No such file or directory

 We tried to recover the disk with different methods, but all failed
 because of different reasons. What helped us at the end was a rebuild on 
 the
 object map of each image:
 rbd object-map rebuild volumes/

 From what we understood, object-map is a feature for ceph internal
 speedup. How can this lead to I/O errors in our VMs?
 Is this the expected way for a recovery?
 Did we miss something?
 Is there any documentation describing what leads to invalid object-maps
 and how to recover? (We did not find a doc on that topic...)
>>>
>>>
>>> An object map definitely shouldn't lead to IO errors in your VMs; in fact
>>> I thought it auto-repaired itself if necessary. Maybe the RBD guys can chime
>>> in here about probable causes of trouble.
>>>
>>> My *guess* is that perhaps your VMs or QEMU were configured to ignore
>>> barriers or some similar thing, so that when the power failed a write was
>>> "lost" as it got written to a new RBD object but not committed into the
>>> object map, but the FS or database journal recorded it as complete. I can't
>>> be sure about that though.
>>> -Greg
>>>



 regards
 Christian
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> --
>> Jason
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph tech talk on deploy ceph with rook on kubernetes

[Brett Niver]

Is the recording available?  I wasn't able to attend.
Thanks,
Brett


On Thu, May 24, 2018 at 10:04 AM, Sage Weil  wrote:
> Starting now!
>
> https://redhat.bluejeans.com/967991495/
>
> It'll be recorded and go up on youtube shortly as well.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Performance

[Brett Niver]

What is your workload like?  Do you have a single or multiple active
MDS ranks configured?


On Tue, May 9, 2017 at 3:10 PM, Webert de Souza Lima
 wrote:
> That 1gbps link is the only option I have for those servers, unfortunately.
> It's all dedicated server rentals from OVH.
> I don't have information regarding the internals of the vrack.
>
> So by what you said, I understand that one should expect a performance drop
> in comparison to ceph rbd using the same architecture , right?
>
> Thanks.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance benchmark -- metadata intensive

[Brett Niver]

Patrick and I had a related question yesterday, are we able to dynamically
vary cache size to artificially manipulate cache pressure?

On Thu, Aug 11, 2016 at 6:07 AM, John Spray  wrote:

> On Thu, Aug 11, 2016 at 8:29 AM, Xiaoxi Chen 
> wrote:
> > Hi ,
> >
> >
> >  Here is the slide I shared yesterday on performance meeting.
> > Thanks and hoping for inputs.
> >
> >
> > http://www.slideshare.net/XiaoxiChen3/cephfs-jewel-mds-
> performance-benchmark
>
> These are definitely useful results and I encourage everyone working
> with cephfs to go and look at Xiaoxi's slides.
>
> The main thing that this highlighted for me was our lack of testing so
> far on systems with full caches.  Too much of our existing testing is
> done on freshly configured systems that never fill the MDS cache.
>
> Test 2.1 notes that we don't enable directory fragmentation by default
> currently -- this is an issue, and I'm hoping we can switch it on by
> default in Kraken (see thread "Switching on mds_bal_frag by default").
> In the meantime we have the fix that Patrick wrote for Jewel which at
> least prevents people creating dirfrags too large for the OSDs to
> handle.
>
> Test 2.2: since a "failing to respond to cache pressure" bug is
> affecting this, I would guess we see the performance fall off at about
> the point where the *client* caches fill up (so they start trimming
> things even though they're ignore cache pressure).  It would be
> interesting to see this chart with addition lines for some related
> perf counters like mds_log.evtrm and mds.inodes_expired, that might
> make it pretty obvious where the MDS is entering different stages that
> see a decrease in the rate of handling client requests.
>
> We really need to sort out the "failing to respond to cache pressure"
> issues that keep popping up, especially if they're still happening on
> a comparatively simple test that is just creating files.  We have a
> specific test for this[1] that is currently being run against the fuse
> client but not the kernel client[2].  This is a good time to try and
> push that forward so I've kicked off an experimental run here:
> http://pulpito.ceph.com/jspray-2016-08-10_16:14:52-
> kcephfs:recovery-master-testing-basic-mira/
>
> In the meantime, although there are reports of similar issues with
> newer kernels, it would be very useful to confirm if the same issue is
> still occurring with more recent kernels.  Issues with cache trimming
> have occurred due to various (separate) bugs, so it's possible that
> while some people are still seeing cache trimming issues with recent
> kernels, the specific case you're hitting might be fixed.
>
> Test 2.3: restarting the MDS doesn't actually give you a completely
> empty cache (everything in the journal gets replayed to pre-populate
> the cache on MDS startup).  However, the results are still valid
> because you're using a different random order in the non-caching test
> case, and the number of inodes in your journal is probably much
> smaller than the overall cache size so it's only a little bit
> populated.  We don't currently have a "drop cache" command built into
> the MDS but it would be pretty easy to add one for use in testing
> (basically just call mds->mdcache->trim(0)).
>
> As one would imagine, the non-caching case is latency-dominated when
> the working set is larger than the cache, where each client is waiting
> for one open to finish before proceeding to the next.  The MDS is
> probably capable of handling many more operations per second, but it
> would need more parallel IO operations from the clients.  When a
> single client is doing opens one by one, you're potentially seeing a
> full network+disk latency for each one (though in practice the OSD
> read cache will be helping a lot here).  This non-caching case would
> be the main argument for giving the metadata pool low latency (SSD)
> storage.
>
> Test 2.5: The observation that the CPU bottleneck makes using fast
> storage for the metadata pool less useful (in sequential/cached cases)
> is valid, although it could still be useful to isolate the metadata
> OSDs (probably SSDs since not so much capacity is needed) to avoid
> competing with data operations.  For random access in the non-caching
> cases (2.3, 2.4) I think you would probably see an improvement from
> SSDs.
>
> Thanks again to the team from ebay for sharing all this.
>
> John
>
>
>
> 1. https://github.com/ceph/ceph-qa-suite/blob/master/tasks/
> cephfs/test_client_limits.py#L96
> 2. http://tracker.ceph.com/issues/9466
>
>
> >
> > Xiaoxi
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

Re: [ceph-users] CephFS Samba VFS RHEL packages

[Brett Niver]

So Blair,

As far as you know right now, you don't need anything from the CephFS team,
correct?
Thanks,
Brett


On Fri, Jul 22, 2016 at 2:18 AM, Yan, Zheng  wrote:

> On Fri, Jul 22, 2016 at 11:15 AM, Blair Bethwaite
>  wrote:
> > Thanks Zheng,
> >
> > On 22 July 2016 at 12:12, Yan, Zheng  wrote:
> >> We actively back-port fixes to RHEL 7.x kernel.  When RHCS2.0 release,
> >> the RHEL kernel should contain fixes up to 3.7 upstream kernel.
> >
> > You meant 4.7 right?
>
> I mean 4.7. sorry
>
> >
> > --
> > Cheers,
> > ~Blairo
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] failing to respond to cache pressure

[Brett Niver]

Hi Oliver,

Our corresponding RHCS downstream release of CephFS will be labeled "Tech
Preview" which means its unsupported, but we believe that it's stable
enough for experimentation.  When we do release Cephfs as "production
ready" that means we've done even more exhaustive testing and that this is
a supported downstream product.

Since our downstream follows our upstream releases, we try to use the same
terminology both up and downstream so that folks can better understand how
to assess individual releases.

I hope that helps,
Brett

On Tuesday, May 17, 2016, Oliver Dzombic <i...@ip-interactive.de> wrote:

> Hi Brett,
>
> aside from the question if what Brian experience has anything to do with
> code stability:
>
> since this is new for me, that there is a difference between "stable"
> and "production ready" i would be happy if you could tell me how the
> table looks like.
>
> One of the team was joking something like:
>
> "stable" > "really stable" > "rock stable" > "pre-production ready" >
> "production ready on your own risk" > "production ready, but can still
> go boom" > "production ready, but still kinda wonky" > "EOL"
>
>
> I did not check the past major releases in deep, but i think i never saw
> something "bigger" but "stable" as the word to point out, that the code
> is ready for usage, where you dont need to expect something too much
> evil to happen.. alias "production ready".
>
> So any clearification if this code is now stable, or not, and if yes,
> what's the exact difference between "stable" and "production ready", is
> very welcome.
>
> Thank you !
>
> --
> Mit freundlichen Gruessen / Best regards
>
> Oliver Dzombic
> IP-Interactive
>
> mailto:i...@ip-interactive.de <javascript:;>
>
> Anschrift:
>
> IP Interactive UG ( haftungsbeschraenkt )
> Zum Sonnenberg 1-3
> 63571 Gelnhausen
>
> HRB 93402 beim Amtsgericht Hanau
> Geschäftsführung: Oliver Dzombic
>
> Steuer Nr.: 35 236 3622 1
> UST ID: DE274086107
>
>
> Am 16.05.2016 um 15:59 schrieb Brett Niver:
> > The terminology we're using to describe CephFS in Jewel is "stable" as
> > opposed to production ready.
> >
> > Thanks,
> > Brett
> >
> > On Monday, May 16, 2016, John Spray <jsp...@redhat.com <javascript:;>
> > <mailto:jsp...@redhat.com <javascript:;>>> wrote:
> >
> > On Mon, May 16, 2016 at 5:42 AM, Andrus, Brian Contractor
> > <bdand...@nps.edu <javascript:;>> wrote:
> > > So this ‘production ready’ CephFS for jewel seems a little not
> quite….
> > >
> > >
> > >
> > > Currently I have a single system mounting CephFS and merely
> > scp-ing data to
> > > it.
> > >
> > > The CephFS mount has 168 TB used, 345 TB / 514 TB avail.
> > >
> > >
> > >
> > > Every so often, I get a HEALTH_WARN message of mds0: Client
> failing to
> > > respond to cache pressure
> >
> > What client, what version?
> >
> > > Even if I stop the scp, it will not go away until I umount/remount
> the
> > > filesystem.
> > >
> > >
> > >
> > > For testing, I had the cephfs mounted on about 50 systems and when
> > updated
> > > started on the, I got all kinds of issues with it all.
> >
> > All kinds of issues...?  Need more specific bug reports than that to
> > fix things.
> >
> > John
> >
> > > I figured having updated run on a few systems would be a good ‘see
> > what
> > > happens’ if there is a fair amount of access to it.
> > >
> > >
> > >
> > > So, should I not be even considering using CephFS as a large
> > storage mount
> > > for a compute cluster? Is there a sweet spot for what CephFS would
> > be good
> > > for?
> > >
> > >
> > >
> > >
> > >
> > > Brian Andrus
> > >
> > > ITACS/Research Computing
> > >
> > > Naval Postgraduate School
> > >
> > > Monterey, California
> > >
> > > voice: 831-656-6238
> > >
> > >
> > >
> > >
> > >
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com <javascript:;>
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com <javascript:;>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com <javascript:;>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com <javascript:;>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] failing to respond to cache pressure

[Brett Niver]

The terminology we're using to describe CephFS in Jewel is "stable" as
opposed to production ready.

Thanks,
Brett

On Monday, May 16, 2016, John Spray  wrote:

> On Mon, May 16, 2016 at 5:42 AM, Andrus, Brian Contractor
> > wrote:
> > So this ‘production ready’ CephFS for jewel seems a little not quite….
> >
> >
> >
> > Currently I have a single system mounting CephFS and merely scp-ing data
> to
> > it.
> >
> > The CephFS mount has 168 TB used, 345 TB / 514 TB avail.
> >
> >
> >
> > Every so often, I get a HEALTH_WARN message of mds0: Client failing to
> > respond to cache pressure
>
> What client, what version?
>
> > Even if I stop the scp, it will not go away until I umount/remount the
> > filesystem.
> >
> >
> >
> > For testing, I had the cephfs mounted on about 50 systems and when
> updated
> > started on the, I got all kinds of issues with it all.
>
> All kinds of issues...?  Need more specific bug reports than that to fix
> things.
>
> John
>
> > I figured having updated run on a few systems would be a good ‘see what
> > happens’ if there is a fair amount of access to it.
> >
> >
> >
> > So, should I not be even considering using CephFS as a large storage
> mount
> > for a compute cluster? Is there a sweet spot for what CephFS would be
> good
> > for?
> >
> >
> >
> >
> >
> > Brian Andrus
> >
> > ITACS/Research Computing
> >
> > Naval Postgraduate School
> >
> > Monterey, California
> >
> > voice: 831-656-6238
> >
> >
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com