subject:"\[ceph\-users\] OSD Crashes"

Re: [ceph-users] osd crashes with large object size (>10GB) in luminos Rados

2017-09-26 Thread Alexander Kushnirenko

Nick,

Thanks, I will look into the latest bareos version.  They did mention
libradosstriper on github.

There is another question.  On jewel I have 25GB size objects.  Once I
upgrade to luminous those objects will be "out of bounds".
1. Will OSD start and Will I be able to read them?
2. Will they chop themselves into little pieces automatically or do I need
to get -- put_back them?

Thank you,
Alexander



On Tue, Sep 26, 2017 at 4:29 PM, Nick Fisk <n...@fisk.me.uk> wrote:

> Bareos needs to be re-written to use libradosstriper or it should
> internally shard the data across multiple objects. Objects shouldn’t be
> stored as large as that and performance will also suffer.
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Alexander Kushnirenko
> *Sent:* 26 September 2017 13:50
> *To:* ceph-users@lists.ceph.com
> *Subject:* [ceph-users] osd crashes with large object size (>10GB) in
> luminos Rados
>
>
>
> Hello,
>
>
>
> We successfully use rados to store backup volumes in jewel version of
> CEPH. Typical volume size is 25-50GB.  Backup software (bareos) use Rados
> objects as backup volumes and it works fine.  Recently we tried luminous
> for the same purpose.
>
>
>
> In luminous developers reduced osd_max_object_size from 100G to 128M.  As
> I understood for the performance reasons.  But it broke down interaction
> with bareos backup software.  You can reverse osd_max_object_size to 100G,
> but then the OSD start to crash once you start to put objects of about 4GB
> in size (4,294,951,051).
>
>
>
> Any suggestion how to approach this problem?
>
>
>
> Alexander.
>
>
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]: 
> /build/ceph-12.2.0/src/os/bluestore/BlueStore.cc:
> In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> ObjectStore::Transaction*)' thread 7f04ac2f9700 time 2017-09-26
> 15:12:58.230268
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]: 
> /build/ceph-12.2.0/src/os/bluestore/BlueStore.cc:
> 9282: FAILED assert(0 == "unexpected error")
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]: 2017-09-26 15:12:58.229837
> 7f04ac2f9700 -1 bluestore(/var/lib/ceph/osd/ceph-0) _txc_add_transaction
> error (7) Argument list too long not handled on operation 10 (op 1,
> counting from 0)
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]: 2017-09-26 15:12:58.229869
> 7f04ac2f9700 -1 bluestore(/var/lib/ceph/osd/ceph-0) unexpected error code
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x563c7b5f83a2]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  2: 
> (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> ObjectStore::Transaction*)+0x15fa) [0x563c7b4ac2ba]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  3: 
> (BlueStore::queue_transactions(ObjectStore::Sequencer*,
> std::vector<ObjectStore::Transaction, std::allocator
> >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x536)
> [0x563c7b4ad916]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  4: (PrimaryLogPG::queue_transacti
> ons(std::vector<ObjectStore::Transaction, 
> std::allocator
> >&, boost::intrusive_ptr)+0x66) [0x563c7b1d17f6]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  5: 
> (ReplicatedBackend::submit_transaction(hobject_t
> const&, object_stat_sum_t const&, eversion_t const&,
> std::unique_ptr<PGTransaction, std::default_delete >&&,
> eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t,
> std::allocator > const&, 
> boost::optional&,
> Context*, Context*, Context*, unsigned long, osd_reqid_t,
> boost::intrusive_ptr)+0xcbf) [0x563c7b30436f]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  6: 
> (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
> PrimaryLogPG::OpContext*)+0x9fa) [0x563c7b16d68a]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  7: 
> (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x131d)
> [0x563c7b1b7a5d]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  8: (PrimaryLogPG::do_op(boost::in
> trusive_ptr&)+0x2ece) [0x563c7b1bb26e]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  9: 
> (PrimaryLogPG::do_request(boost::intrusive_ptr&,
> ThreadPool::TPHandle&)+0xea6) [0x563c7b175446]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  10: 
> (OSD::dequeue_op(boost::intrusive_ptr,
> boost::intrusive_ptr, ThreadPool::TPHandle&)+0x3ab)
> [0x563c7aff919b]
>
> Sep 26 15:12:58 ceph02 ceph-osd[1417]:  11: (PGQueueable::RunVis::operator
> ()(boost::intrusive_ptr const&)+0x5a) [0

[ceph-users] osd crashes with large object size (>10GB) in luminos Rados

2017-09-26 Thread Alexander Kushnirenko

Hello,

We successfully use rados to store backup volumes in jewel version of CEPH.
Typical volume size is 25-50GB.  Backup software (bareos) use Rados objects
as backup volumes and it works fine.  Recently we tried luminous for the
same purpose.

In luminous developers reduced osd_max_object_size from 100G to 128M.  As I
understood for the performance reasons.  But it broke down interaction with
bareos backup software.  You can reverse osd_max_object_size to 100G, but
then the OSD start to crash once you start to put objects of about 4GB in
size (4,294,951,051).

Any suggestion how to approach this problem?

Alexander.

Sep 26 15:12:58 ceph02 ceph-osd[1417]:
/build/ceph-12.2.0/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)' thread 7f04ac2f9700 time 2017-09-26
15:12:58.230268
Sep 26 15:12:58 ceph02 ceph-osd[1417]:
/build/ceph-12.2.0/src/os/bluestore/BlueStore.cc: 9282: FAILED assert(0 ==
"unexpected error")
Sep 26 15:12:58 ceph02 ceph-osd[1417]: 2017-09-26 15:12:58.229837
7f04ac2f9700 -1 bluestore(/var/lib/ceph/osd/ceph-0) _txc_add_transaction
error (7) Argument list too long not handled on operation 10 (op 1,
counting from 0)
Sep 26 15:12:58 ceph02 ceph-osd[1417]: 2017-09-26 15:12:58.229869
7f04ac2f9700 -1 bluestore(/var/lib/ceph/osd/ceph-0) unexpected error code
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  ceph version 12.2.0
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x102) [0x563c7b5f83a2]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  2:
(BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)+0x15fa) [0x563c7b4ac2ba]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  3:
(BlueStore::queue_transactions(ObjectStore::Sequencer*,
std::vector&,
boost::intrusive_ptr, ThreadPool::TPHandle*)+0x536)
[0x563c7b4ad916]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  4:
(PrimaryLogPG::queue_transactions(std::vector&,
boost::intrusive_ptr)+0x66) [0x563c7b1d17f6]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  5:
(ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t
const&, eversion_t const&, std::unique_ptr&&, eversion_t const&, eversion_t
const&, std::vector
const&, boost::optional&, Context*, Context*,
Context*, unsigned long, osd_reqid_t,
boost::intrusive_ptr)+0xcbf) [0x563c7b30436f]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  6:
(PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
PrimaryLogPG::OpContext*)+0x9fa) [0x563c7b16d68a]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  7:
(PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x131d)
[0x563c7b1b7a5d]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  8:
(PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x2ece)
[0x563c7b1bb26e]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  9:
(PrimaryLogPG::do_request(boost::intrusive_ptr&,
ThreadPool::TPHandle&)+0xea6) [0x563c7b175446]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  10:
(OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr,
ThreadPool::TPHandle&)+0x3ab) [0x563c7aff919b]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  11:
(PGQueueable::RunVis::operator()(boost::intrusive_ptr
const&)+0x5a) [0x563c7b29154a]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  12:
(OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x103d) [0x563c7b01fd9d]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  13:
(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x8ef)
[0x563c7b5fd20f]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  14:
(ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x563c7b600510]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  15: (()+0x7494) [0x7f04c56e2494]
Sep 26 15:12:58 ceph02 ceph-osd[1417]:  16: (clone()+0x3f) [0x7f04c4769aff]
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD crashes on EC recovery

2016-08-10 Thread Brian Felton

Roeland,

We're seeing the same problems in our cluster.  I can't offer you a
solution that gets the OSD back, but I can tell you what I did to work
around it.

We're running 5 0.94.6 clusters with 9 nodes / 648 HDD OSDs with a k=7, m=2
erasure coded .rgw.buckets pool.  During the backfilling after a recent
disk replacement, we had four OSDs that got in a very similar state.

2016-08-09 07:40:12.475699 7f025b06b700 -1 osd/ECBackend.cc: In function
'void ECBackend::handle_recovery_push(PushOp&, RecoveryMessages*)' thread
7f025b06b700 time 2016-08-09 07:40:12.472819
osd/ECBackend.cc: 281: FAILED assert(op.attrset.count(string("_")))

 ceph version 0.94.6-2 (f870be457b16e4ff56ced74ed3a3c9a4c781f281)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xba997b]
 2: (ECBackend::handle_recovery_push(PushOp&, RecoveryMessages*)+0xd7f)
[0xa239ff]
 3: (ECBackend::handle_message(std::tr1::shared_ptr)+0x1de)
[0xa2600e]
 4: (ReplicatedPG::do_request(std::tr1::shared_ptr&,
ThreadPool::TPHandle&)+0x167) [0x8305e7]
 5: (OSD::dequeue_op(boost::intrusive_ptr,
std::tr1::shared_ptr, ThreadPool::TPHandle&)+0x3bd) [0x6a157d]
 6: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x338) [0x6a1aa8]
 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f)
[0xb994cf]
 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb9b5f0]
 9: (()+0x8184) [0x7f0284e35184]
 10: (clone()+0x6d) [0x7f028324c37d]

To allow the cluster to recover, we ended up reweighting the OSDs that got
into this state to 0 (ceph osd crush reweight  0).  This will of
course kick off a long round of backfilling, but it eventually recovers.
We've never found a solution that gets the OSD healthy again that doesn't
involve nuking the underlying disk and starting over.  We've had 10 OSDs
get in this state across 2 clusters in the last few months.  The
failure/crash message is always the same.  If someone does know of a way to
recover the OSD, that would be great.

I hope this helps.

Brian Felton

On Wed, Aug 10, 2016 at 10:17 AM, Roeland Mertens <
roeland.mert...@genomicsplc.com> wrote:

> Hi,
>
> we run a Ceph 10.2.1 cluster across 35 nodes with a total of 595 OSDs, we
> have a mixture of normally replicated volumes and EC volumes using the
> following erasure-code-profile:
>
> # ceph osd erasure-code-profile get rsk8m5
> jerasure-per-chunk-alignment=false
> k=8
> m=5
> plugin=jerasure
> ruleset-failure-domain=host
> ruleset-root=default
> technique=reed_sol_van
> w=8
>
> Now we had a disk failure and on swap out we seem to have encountered a
> bug where during recovery OSDs crash when trying to fix certain pgs that
> may have been corrupted.
>
> For example:
>-3> 2016-08-10 12:38:21.302938 7f893e2d7700  5 -- op tracker -- seq:
> 3434, time: 2016-08-10 12:38:21.302938, event: queued_for_pg, op:
> MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
> -2> 2016-08-10 12:38:21.302981 7f89bef50700  1 --
> 10.93.105.11:6831/2674119 --> 10.93.105.22:6802/357033 --
> osd_map(47662..47663 src has 32224..47663) v3 -- ?+0 0x559c1057f3c0 con
> 0x559c0664a700
> -1> 2016-08-10 12:38:21.302996 7f89bef50700  5 -- op tracker -- seq:
> 3434, time: 2016-08-10 12:38:21.302996, event: reached_pg, op:
> MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
>  0> 2016-08-10 12:38:21.306193 7f89bef50700 -1 osd/ECBackend.cc: In
> function 'virtual void 
> OnRecoveryReadComplete::finish(std::pair ECBackend::read_result_t&>&)' thread 7f89bef50700 time 2016-08-10
> 12:38:21.303012
> osd/ECBackend.cc: 203: FAILED assert(res.errors.empty())
>
> then the ceph-osd daemon goes splat. I've attached an extract of a logfile
> showing a bit more.
>
> Anyone have any ideas? I'm stuck now with a pg that's stuck as
> down+remapped+peering. ceph pg query tells me that peering is blocked to
> the loss of an osd, though restarting it just results in another crash of
> the ceph-osd daemon. We tried to force a rebuild by using
> ceph-objectstore-tool to delete the pg segment on some of the OSDs that are
> crashing but that didn't help one iota.
>
> Any help would be greatly appreciated,
>
> regards,
>
> Roeland
>
> --
> This email is sent on behalf of Genomics plc, a public limited company
> registered in England and Wales with registered number 8839972, VAT
> registered number 189 2635 65 and registered office at King Charles House,
> Park End Street, Oxford, OX1 1JD, United Kingdom.
> The contents of this e-mail and any attachments are confidential to the
> intended recipient. If you are not the intended recipient please do not use
> or publish its contents, contact Genomics plc immediately at
> i...@genomicsplc.com  then delete. You may not
> copy, forward, use or disclose the contents of this email to anybody else
> if you are not the intended recipient. Emails are not secure and may
> contain viruses.
>
>

[ceph-users] OSD crashes on EC recovery

2016-08-10 Thread Roeland Mertens


Hi,

we run a Ceph 10.2.1 cluster across 35 nodes with a total of 595 OSDs, 
we have a mixture of normally replicated volumes and EC volumes using 
the following erasure-code-profile:


# ceph osd erasure-code-profile get rsk8m5
jerasure-per-chunk-alignment=false
k=8
m=5
plugin=jerasure
ruleset-failure-domain=host
ruleset-root=default
technique=reed_sol_van
w=8

Now we had a disk failure and on swap out we seem to have encountered a 
bug where during recovery OSDs crash when trying to fix certain pgs that 
may have been corrupted.


For example:
   -3> 2016-08-10 12:38:21.302938 7f893e2d7700  5 -- op tracker -- seq: 
3434, time: 2016-08-10 12:38:21.302938, event: queued_for_pg, op: 
MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
-2> 2016-08-10 12:38:21.302981 7f89bef50700  1 -- 
10.93.105.11:6831/2674119 --> 10.93.105.22:6802/357033 -- 
osd_map(47662..47663 src has 32224..47663) v3 -- ?+0 0x559c1057f3c0 con 
0x559c0664a700
-1> 2016-08-10 12:38:21.302996 7f89bef50700  5 -- op tracker -- 
seq: 3434, time: 2016-08-10 12:38:21.302996, event: reached_pg, op: 
MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
 0> 2016-08-10 12:38:21.306193 7f89bef50700 -1 osd/ECBackend.cc: In 
function 'virtual void 
OnRecoveryReadComplete::finish(std::pair&)' thread 7f89bef50700 time 2016-08-10 
12:38:21.303012

osd/ECBackend.cc: 203: FAILED assert(res.errors.empty())

then the ceph-osd daemon goes splat. I've attached an extract of a 
logfile showing a bit more.


Anyone have any ideas? I'm stuck now with a pg that's stuck as 
down+remapped+peering. ceph pg query tells me that peering is blocked to 
the loss of an osd, though restarting it just results in another crash 
of the ceph-osd daemon. We tried to force a rebuild by using 
ceph-objectstore-tool to delete the pg segment on some of the OSDs that 
are crashing but that didn't help one iota.


Any help would be greatly appreciated,

regards,

Roeland

--
This email is sent on behalf of Genomics plc, a public limited company 
registered in England and Wales with registered number 8839972, VAT 
registered number 189 2635 65 and registered office at King Charles House, 
Park End Street, Oxford, OX1 1JD, United Kingdom.
The contents of this e-mail and any attachments are confidential to the 
intended recipient. If you are not the intended recipient please do not use 
or publish its contents, contact Genomics plc immediately at 
i...@genomicsplc.com  then delete. You may not copy, 
forward, use or disclose the contents of this email to anybody else if you 
are not the intended recipient. Emails are not secure and may contain 
viruses.
-4> 2016-08-10 12:38:21.302910 7f893e2d7700  1 -- 10.93.105.11:6831/2674119 
<== osd.290 10.93.105.22:6802/357033 42  MOSDECSubOpReadReply(63.1a18s0 
47661 ECSubReadReply(tid=1, attrs_read=0)) v1  170+0+0 (1521384358 0 0) 
0x559bf
b611400 con 0x559c0664a700
-3> 2016-08-10 12:38:21.302938 7f893e2d7700  5 -- op tracker -- seq: 3434, 
time: 2016-08-10 12:38:21.302938, event: queued_for_pg, op: 
MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
-2> 2016-08-10 12:38:21.302981 7f89bef50700  1 -- 10.93.105.11:6831/2674119 
--> 10.93.105.22:6802/357033 -- osd_map(47662..47663 src has 32224..47663) v3 
-- ?+0 0x559c1057f3c0 con 0x559c0664a700
-1> 2016-08-10 12:38:21.302996 7f89bef50700  5 -- op tracker -- seq: 3434, 
time: 2016-08-10 12:38:21.302996, event: reached_pg, op: 
MOSDECSubOpReadReply(63.1a18s0 47661 ECSubReadReply(tid=1, attrs_read=0))
 0> 2016-08-10 12:38:21.306193 7f89bef50700 -1 osd/ECBackend.cc: In 
function 'virtual void 
OnRecoveryReadComplete::finish(std::pair&)' thread 7f89bef50700 time 2016-08-10 
12:38:21.303012
osd/ECBackend.cc: 203: FAILED assert(res.errors.empty())

 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) 
[0x559be1135e2b]
 2: (OnRecoveryReadComplete::finish(std::pair&)+0x192) [0x559be0cf6122]
 3: (GenContext&>::complete(std::pair&)+0x9) [0x559be0ce3b89]
 4: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x63) 
[0x559be0cda003]
 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, 
RecoveryMessages*)+0xf68) [0x559be0cdafd8]
 6: (ECBackend::handle_message(std::shared_ptr)+0x186) 
[0x559be0ce2236]
 7: (ReplicatedPG::do_request(std::shared_ptr&, 
ThreadPool::TPHandle&)+0xed) [0x559be0c1c30d]
 8: (OSD::dequeue_op(boost::intrusive_ptr, std::shared_ptr, 
ThreadPool::TPHandle&)+0x3f5) [0x559be0adb285]
 9: (PGQueueable::RunVis::operator()(std::shared_ptr&)+0x5d) 
[0x559be0adb4ad]
 10: (OSD::ShardedOpWQ::_process(unsigned int,

Re: [ceph-users] OSD Crashes

2016-05-02 Thread Varada Kari

You can run chown from /var/lib/ceph with -H --dereference,
something like

chown ceph:ceph -RH --dereference /var/lib/ceph/

need to change the ownership of the partition to ceph:ceph, ceph-disk
does that in a fresh install. Journal is a symlink in osd, so we need to
traverse the symlink and change.
The above worked for me.

Varada

On Friday 29 April 2016 10:00 PM, Garg, Pankaj wrote:
> I think the issue is possibly coming from my Journal drives after upgrade to 
> Infernalis. I have 2 SSDs, which have 6 partitions each for a total of 12 
> Journals / server.
>
> When I create OSDS, I pass the partition names as Journals 
> For e.g.  ceph-deploy osd prepare x86Ceph7:/dev/sdd:/dev/sdb1
>
> This works, but since the ownership on the journals is not ceph:ceph, 
> everything fails, until I run chown ceph:ceph /dev/sda4.
> This change doesn’t persist after reboots.
>
> An idea how to fix this. 
>
> Thanks
> Pankaj
>
>
>
> -Original Message-
> From: Somnath Roy [mailto:somnath@sandisk.com] 
> Sent: Friday, April 29, 2016 9:03 AM
> To: Garg, Pankaj; Samuel Just
> Cc: ceph-users@lists.ceph.com
> Subject: RE: [ceph-users] OSD Crashes
>
> Check system log and search for the corresponding drive. It should have the 
> information what is failing..
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Garg, Pankaj
> Sent: Friday, April 29, 2016 8:59 AM
> To: Samuel Just
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] OSD Crashes
>
> I can see that. I guess what would that be symptomatic of? How is it doing 
> that on 6 different systems and on multiple OSDs?
>
> -Original Message-
> From: Samuel Just [mailto:sj...@redhat.com]
> Sent: Friday, April 29, 2016 8:57 AM
> To: Garg, Pankaj
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] OSD Crashes
>
> Your fs is throwing an EIO on open.
> -Sam
>
> On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj 
> <pankaj.g...@caviumnetworks.com> wrote:
>> Hi,
>>
>> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 
>> nodes, each with 12 HDD Drives and 2SSD Drives. All these were 
>> initially running Hammer, and then were successfully updated to Infernalis 
>> (9.2.0).
>>
>> I recently deleted all my OSDs and swapped my drives with new ones on 
>> the
>> x86 Systems, and the ARM servers were swapped with different ones 
>> (keeping drives same).
>>
>> I again provisioned the OSDs, keeping the same cluster and Ceph 
>> versions as before. But now, every time I try to run RADOS bench, my 
>> OSDs start crashing (on both ARM and x86 servers).
>>
>> I’m not sure why this is happening on all 6 systems. On the x86, it’s 
>> the same Ceph bits as before, and the only thing different is the new drives.
>>
>> It’s the same stack (pasted below) on all the OSDs too.
>>
>> Can anyone provide any clues?
>>
>>
>>
>> Thanks
>>
>> Pankaj
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
>> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236 
>> 
>> osd_repop(client.2794263.0:37721 284.6d4 
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26) v1  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400 
>> con 0x5634c5168420
>>
>>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
>> osd_repop(client.2794263.0:37721 284.6d4 
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
>> osd_repop(client.2794263.0:37721 284.6d4 
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
>> osd_repop(client.2794263.0:37721 284.6d4 
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 0.00, event: dispatched, op:
>> osd_repop(client.2794263.0:37721 284.6d4 
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj

I think the issue is possibly coming from my Journal drives after upgrade to 
Infernalis. I have 2 SSDs, which have 6 partitions each for a total of 12 
Journals / server.

When I create OSDS, I pass the partition names as Journals 
For e.g.  ceph-deploy osd prepare x86Ceph7:/dev/sdd:/dev/sdb1

This works, but since the ownership on the journals is not ceph:ceph, 
everything fails, until I run chown ceph:ceph /dev/sda4.
This change doesn’t persist after reboots.

An idea how to fix this. 

Thanks
Pankaj



-Original Message-
From: Somnath Roy [mailto:somnath@sandisk.com] 
Sent: Friday, April 29, 2016 9:03 AM
To: Garg, Pankaj; Samuel Just
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] OSD Crashes

Check system log and search for the corresponding drive. It should have the 
information what is failing..

Thanks & Regards
Somnath

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Garg, 
Pankaj
Sent: Friday, April 29, 2016 8:59 AM
To: Samuel Just
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD Crashes

I can see that. I guess what would that be symptomatic of? How is it doing that 
on 6 different systems and on multiple OSDs?

-Original Message-
From: Samuel Just [mailto:sj...@redhat.com]
Sent: Friday, April 29, 2016 8:57 AM
To: Garg, Pankaj
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD Crashes

Your fs is throwing an EIO on open.
-Sam

On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj <pankaj.g...@caviumnetworks.com> 
wrote:
> Hi,
>
> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 
> nodes, each with 12 HDD Drives and 2SSD Drives. All these were 
> initially running Hammer, and then were successfully updated to Infernalis 
> (9.2.0).
>
> I recently deleted all my OSDs and swapped my drives with new ones on 
> the
> x86 Systems, and the ARM servers were swapped with different ones 
> (keeping drives same).
>
> I again provisioned the OSDs, keeping the same cluster and Ceph 
> versions as before. But now, every time I try to run RADOS bench, my 
> OSDs start crashing (on both ARM and x86 servers).
>
> I’m not sure why this is happening on all 6 systems. On the x86, it’s 
> the same Ceph bits as before, and the only thing different is the new drives.
>
> It’s the same stack (pasted below) on all the OSDs too.
>
> Can anyone provide any clues?
>
>
>
> Thanks
>
> Pankaj
>
>
>
>
>
>
>
>
>
>
>
>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236 
> 
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26) v1  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400 
> con 0x5634c5168420
>
>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 0.00, event: dispatched, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
> -9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
> -8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284 
> tick_without_osd_lock
>
> -7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306 
>  osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2 
> 47+0+0
> (846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
>
> -6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 -- 
> osd_ping(ping_reply e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0
> 0x5634c7ba8000 con 0x5634c58dd760
>
>

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Samuel Just

You could strace the process to see precisely what ceph-osd is doing
to provoke the EIO.
-Sam

On Fri, Apr 29, 2016 at 9:03 AM, Somnath Roy <somnath@sandisk.com> wrote:
> Check system log and search for the corresponding drive. It should have the 
> information what is failing..
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Garg, Pankaj
> Sent: Friday, April 29, 2016 8:59 AM
> To: Samuel Just
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] OSD Crashes
>
> I can see that. I guess what would that be symptomatic of? How is it doing 
> that on 6 different systems and on multiple OSDs?
>
> -Original Message-
> From: Samuel Just [mailto:sj...@redhat.com]
> Sent: Friday, April 29, 2016 8:57 AM
> To: Garg, Pankaj
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] OSD Crashes
>
> Your fs is throwing an EIO on open.
> -Sam
>
> On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj 
> <pankaj.g...@caviumnetworks.com> wrote:
>> Hi,
>>
>> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64
>> nodes, each with 12 HDD Drives and 2SSD Drives. All these were
>> initially running Hammer, and then were successfully updated to Infernalis 
>> (9.2.0).
>>
>> I recently deleted all my OSDs and swapped my drives with new ones on
>> the
>> x86 Systems, and the ARM servers were swapped with different ones
>> (keeping drives same).
>>
>> I again provisioned the OSDs, keeping the same cluster and Ceph
>> versions as before. But now, every time I try to run RADOS bench, my
>> OSDs start crashing (on both ARM and x86 servers).
>>
>> I’m not sure why this is happening on all 6 systems. On the x86, it’s
>> the same Ceph bits as before, and the only thing different is the new drives.
>>
>> It’s the same stack (pasted below) on all the OSDs too.
>>
>> Can anyone provide any clues?
>>
>>
>>
>> Thanks
>>
>> Pankaj
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
>> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236
>> 
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26) v1  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400
>> con 0x5634c5168420
>>
>>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 0.00, event: dispatched, op:
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>> -9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq:
>> 29404, time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op:
>> osd_repop(client.2794263.0:37721 284.6d4
>> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
>> 12284'26)
>>
>> -8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284
>> tick_without_osd_lock
>>
>> -7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 --
>> 192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306
>>  osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2 
>> 47+0+0
>> (846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
>>
>> -6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 --
>> 192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 --
>> osd_ping(ping_reply e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0
>> 0x5634c7ba8000 con 0x5634c58dd760
>>
>> -5> 2016-04-28 08:09:45.974300 7f1f0981d700  1 --
>

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Somnath Roy

Check system log and search for the corresponding drive. It should have the 
information what is failing..

Thanks & Regards
Somnath

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Garg, 
Pankaj
Sent: Friday, April 29, 2016 8:59 AM
To: Samuel Just
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD Crashes

I can see that. I guess what would that be symptomatic of? How is it doing that 
on 6 different systems and on multiple OSDs?

-Original Message-
From: Samuel Just [mailto:sj...@redhat.com]
Sent: Friday, April 29, 2016 8:57 AM
To: Garg, Pankaj
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD Crashes

Your fs is throwing an EIO on open.
-Sam

On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj <pankaj.g...@caviumnetworks.com> 
wrote:
> Hi,
>
> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64
> nodes, each with 12 HDD Drives and 2SSD Drives. All these were
> initially running Hammer, and then were successfully updated to Infernalis 
> (9.2.0).
>
> I recently deleted all my OSDs and swapped my drives with new ones on
> the
> x86 Systems, and the ARM servers were swapped with different ones
> (keeping drives same).
>
> I again provisioned the OSDs, keeping the same cluster and Ceph
> versions as before. But now, every time I try to run RADOS bench, my
> OSDs start crashing (on both ARM and x86 servers).
>
> I’m not sure why this is happening on all 6 systems. On the x86, it’s
> the same Ceph bits as before, and the only thing different is the new drives.
>
> It’s the same stack (pasted below) on all the OSDs too.
>
> Can anyone provide any clues?
>
>
>
> Thanks
>
> Pankaj
>
>
>
>
>
>
>
>
>
>
>
>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236
> 
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26) v1  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400
> con 0x5634c5168420
>
>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 0.00, event: dispatched, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
> -9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v
> 12284'26)
>
> -8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284
> tick_without_osd_lock
>
> -7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306
>  osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2 
> 47+0+0
> (846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
>
> -6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 --
> osd_ping(ping_reply e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0
> 0x5634c7ba8000 con 0x5634c58dd760
>
> -5> 2016-04-28 08:09:45.974300 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306 
> osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  47+0+0
> (846632602 0 0) 0x5634c8129400 con 0x5634c58dcf20
>
> -4> 2016-04-28 08:09:45.974337 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 --> 192.168.240.115:0/26572 --
> osd_ping(ping_reply
> e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0 0x5634c617d600 con
> 0x5634c58dcf20
>
> -3> 2016-04-28 08:09:46.174079 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102) write couldn't open
> 287.6f9_head/287/ae33fef9/benchmark_data_ceph7_17591_object39895/head:
> (117) Structure needs cl

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj

I can see that. I guess what would that be symptomatic of? How is it doing that 
on 6 different systems and on multiple OSDs?

-Original Message-
From: Samuel Just [mailto:sj...@redhat.com] 
Sent: Friday, April 29, 2016 8:57 AM
To: Garg, Pankaj
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD Crashes

Your fs is throwing an EIO on open.
-Sam

On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj <pankaj.g...@caviumnetworks.com> 
wrote:
> Hi,
>
> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 
> nodes, each with 12 HDD Drives and 2SSD Drives. All these were 
> initially running Hammer, and then were successfully updated to Infernalis 
> (9.2.0).
>
> I recently deleted all my OSDs and swapped my drives with new ones on 
> the
> x86 Systems, and the ARM servers were swapped with different ones 
> (keeping drives same).
>
> I again provisioned the OSDs, keeping the same cluster and Ceph 
> versions as before. But now, every time I try to run RADOS bench, my 
> OSDs start crashing (on both ARM and x86 servers).
>
> I’m not sure why this is happening on all 6 systems. On the x86, it’s 
> the same Ceph bits as before, and the only thing different is the new drives.
>
> It’s the same stack (pasted below) on all the OSDs too.
>
> Can anyone provide any clues?
>
>
>
> Thanks
>
> Pankaj
>
>
>
>
>
>
>
>
>
>
>
>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236 
> 
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26) v1  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400 
> con 0x5634c5168420
>
>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26)
>
>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26)
>
>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26)
>
>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 0.00, event: dispatched, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26)
>
> -9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op:
> osd_repop(client.2794263.0:37721 284.6d4 
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 
> 12284'26)
>
> -8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284 
> tick_without_osd_lock
>
> -7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306 
>  osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  
> 47+0+0
> (846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
>
> -6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 -- 
> osd_ping(ping_reply e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0
> 0x5634c7ba8000 con 0x5634c58dd760
>
> -5> 2016-04-28 08:09:45.974300 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306  
> osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  47+0+0
> (846632602 0 0) 0x5634c8129400 con 0x5634c58dcf20
>
> -4> 2016-04-28 08:09:45.974337 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 --> 192.168.240.115:0/26572 -- 
> osd_ping(ping_reply
> e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0 0x5634c617d600 con
> 0x5634c58dcf20
>
> -3> 2016-04-28 08:09:46.174079 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102) write couldn't open
> 287.6f9_head/287/ae33fef9/benchmark_data_ceph7_17591_object39895/head: 
> (117) Structure needs cleaning
>
> -2> 2016-04-28 08:09:46.174103 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102)  error (117) Structure needs 
> cleaning not handled on operation 0x5634c885df9e (16590.1.0, or op 0, 
> counting from
> 0)
>
> -1> 2016-04-28 08:09:46.174109 7f1f11f92700  0
> filestore(/var/lib/ceph/o

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Samuel Just

Your fs is throwing an EIO on open.
-Sam

On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj
 wrote:
> Hi,
>
> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 nodes,
> each with 12 HDD Drives and 2SSD Drives. All these were initially running
> Hammer, and then were successfully updated to Infernalis (9.2.0).
>
> I recently deleted all my OSDs and swapped my drives with new ones on the
> x86 Systems, and the ARM servers were swapped with different ones (keeping
> drives same).
>
> I again provisioned the OSDs, keeping the same cluster and Ceph versions as
> before. But now, every time I try to run RADOS bench, my OSDs start crashing
> (on both ARM and x86 servers).
>
> I’m not sure why this is happening on all 6 systems. On the x86, it’s the
> same Ceph bits as before, and the only thing different is the new drives.
>
> It’s the same stack (pasted below) on all the OSDs too.
>
> Can anyone provide any clues?
>
>
>
> Thanks
>
> Pankaj
>
>
>
>
>
>
>
>
>
>
>
>   -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 --
> 192.168.240.117:6820/14377 <== osd.93 192.168.240.116:6811/47080 1236 
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26) v1
>  981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400 con 0x5634c5168420
>
>-13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423882, event: header_read, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
>
>-12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423884, event: throttled, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
>
>-11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.423942, event: all_read, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
>
>-10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 0.00, event: dispatched, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
>
> -9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq:
> 29404, time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op:
> osd_repop(client.2794263.0:37721 284.6d4
> 284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
>
> -8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284
> tick_without_osd_lock
>
> -7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306 
> osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  47+0+0
> (846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
>
> -6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 --
> 192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 --
> osd_ping(ping_reply e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0
> 0x5634c7ba8000 con 0x5634c58dd760
>
> -5> 2016-04-28 08:09:45.974300 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306 
> osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  47+0+0
> (846632602 0 0) 0x5634c8129400 con 0x5634c58dcf20
>
> -4> 2016-04-28 08:09:45.974337 7f1f0981d700  1 --
> 10.18.240.117:6821/14377 --> 192.168.240.115:0/26572 -- osd_ping(ping_reply
> e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0 0x5634c617d600 con
> 0x5634c58dcf20
>
> -3> 2016-04-28 08:09:46.174079 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102) write couldn't open
> 287.6f9_head/287/ae33fef9/benchmark_data_ceph7_17591_object39895/head: (117)
> Structure needs cleaning
>
> -2> 2016-04-28 08:09:46.174103 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102)  error (117) Structure needs cleaning
> not handled on operation 0x5634c885df9e (16590.1.0, or op 0, counting from
> 0)
>
> -1> 2016-04-28 08:09:46.174109 7f1f11f92700  0
> filestore(/var/lib/ceph/osd/ceph-102) unexpected error code
>
>  0> 2016-04-28 08:09:46.178707 7f1f11791700 -1 os/FileStore.cc: In
> function 'int FileStore::lfn_open(coll_t, const ghobject_t&, bool, FDRef*,
> Index*)' thread 7f1f11791700 time 2016-04-28 08:09:46.173250
>
> os/FileStore.cc: 335: FAILED assert(!m_filestore_fail_eio || r != -5)
>
>
>
> ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x8b) [0x5634c02ec7eb]
>
> 2: (FileStore::lfn_open(coll_t, ghobject_t const&, bool,
> std::shared_ptr*, Index*)+0x1191) [0x5634bffb2d01]
>
> 3: (FileStore::_write(coll_t, ghobject_t const&, unsigned long, unsigned
> long, ceph::buffer::list const&, unsigned int)+0xf0) [0x5634bffbb7b0]
>
> 4:

[ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj

Hi,
I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 nodes, each 
with 12 HDD Drives and 2SSD Drives. All these were initially running Hammer, 
and then were successfully updated to Infernalis (9.2.0).
I recently deleted all my OSDs and swapped my drives with new ones on the x86 
Systems, and the ARM servers were swapped with different ones (keeping drives 
same).
I again provisioned the OSDs, keeping the same cluster and Ceph versions as 
before. But now, every time I try to run RADOS bench, my OSDs start crashing 
(on both ARM and x86 servers).
I'm not sure why this is happening on all 6 systems. On the x86, it's the same 
Ceph bits as before, and the only thing different is the new drives.
It's the same stack (pasted below) on all the OSDs too.
Can anyone provide any clues?

Thanks
Pankaj





  -14> 2016-04-28 08:09:45.423950 7f1ef05b1700  1 -- 192.168.240.117:6820/14377 
<== osd.93 192.168.240.116:6811/47080 1236  
osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26) v1 
 981+0+4759 (3923326827 0 3705383247) 0x5634cbabc400 con 0x5634c5168420
   -13> 2016-04-28 08:09:45.423981 7f1ef05b1700  5 -- op tracker -- seq: 29404, 
time: 2016-04-28 08:09:45.423882, event: header_read, op: 
osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
   -12> 2016-04-28 08:09:45.423991 7f1ef05b1700  5 -- op tracker -- seq: 29404, 
time: 2016-04-28 08:09:45.423884, event: throttled, op: 
osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
   -11> 2016-04-28 08:09:45.423996 7f1ef05b1700  5 -- op tracker -- seq: 29404, 
time: 2016-04-28 08:09:45.423942, event: all_read, op: 
osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
   -10> 2016-04-28 08:09:45.424001 7f1ef05b1700  5 -- op tracker -- seq: 29404, 
time: 0.00, event: dispatched, op: osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
-9> 2016-04-28 08:09:45.424014 7f1ef05b1700  5 -- op tracker -- seq: 29404, 
time: 2016-04-28 08:09:45.424014, event: queued_for_pg, op: 
osd_repop(client.2794263.0:37721 284.6d4 
284/afa8fed4/benchmark_data_x86Ceph1_147212_object37720/head v 12284'26)
-8> 2016-04-28 08:09:45.561827 7f1f15799700  5 osd.102 12284 
tick_without_osd_lock
-7> 2016-04-28 08:09:45.973944 7f1f0801a700  1 -- 
192.168.240.117:6821/14377 <== osd.73 192.168.240.115:0/26572 1306  
osd_ping(ping e12284 stamp 2016-04-28 08:09:45.971751) v2  47+0+0 
(846632602 0 0) 0x5634c8305c00 con 0x5634c58dd760
-6> 2016-04-28 08:09:45.973995 7f1f0801a700  1 -- 
192.168.240.117:6821/14377 --> 192.168.240.115:0/26572 -- osd_ping(ping_reply 
e12284 stamp 2016-04-28 08:09:45.971751) v2 -- ?+0 0x5634c7ba8000 con 
0x5634c58dd760
-5> 2016-04-28 08:09:45.974300 7f1f0981d700  1 -- 10.18.240.117:6821/14377 
<== osd.73 192.168.240.115:0/26572 1306  osd_ping(ping e12284 stamp 
2016-04-28 08:09:45.971751) v2  47+0+0 (846632602 0 0) 0x5634c8129400 con 
0x5634c58dcf20
-4> 2016-04-28 08:09:45.974337 7f1f0981d700  1 -- 10.18.240.117:6821/14377 
--> 192.168.240.115:0/26572 -- osd_ping(ping_reply e12284 stamp 2016-04-28 
08:09:45.971751) v2 -- ?+0 0x5634c617d600 con 0x5634c58dcf20
-3> 2016-04-28 08:09:46.174079 7f1f11f92700  0 
filestore(/var/lib/ceph/osd/ceph-102) write couldn't open 
287.6f9_head/287/ae33fef9/benchmark_data_ceph7_17591_object39895/head: (117) 
Structure needs cleaning
-2> 2016-04-28 08:09:46.174103 7f1f11f92700  0 
filestore(/var/lib/ceph/osd/ceph-102)  error (117) Structure needs cleaning not 
handled on operation 0x5634c885df9e (16590.1.0, or op 0, counting from 0)
-1> 2016-04-28 08:09:46.174109 7f1f11f92700  0 
filestore(/var/lib/ceph/osd/ceph-102) unexpected error code
 0> 2016-04-28 08:09:46.178707 7f1f11791700 -1 os/FileStore.cc: In function 
'int FileStore::lfn_open(coll_t, const ghobject_t&, bool, FDRef*, Index*)' 
thread 7f1f11791700 time 2016-04-28 08:09:46.173250
os/FileStore.cc: 335: FAILED assert(!m_filestore_fail_eio || r != -5)

ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) 
[0x5634c02ec7eb]
2: (FileStore::lfn_open(coll_t, ghobject_t const&, bool, 
std::shared_ptr*, Index*)+0x1191) [0x5634bffb2d01]
3: (FileStore::_write(coll_t, ghobject_t const&, unsigned long, unsigned long, 
ceph::buffer::list const&, unsigned int)+0xf0) [0x5634bffbb7b0]
4: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, 
ThreadPool::TPHandle*)+0x2901) [0x5634bffc6f51]
5: (FileStore::_do_transactions(std::list >&, unsigned long, 
ThreadPool::TPHandle*)+0x64) [0x5634bffcc404]
6:

[ceph-users] OSD crashes when starting

2015-08-07 Thread Gerd Jakobovitsch


Dear all,

I got to an unrecoverable crash at one specific OSD, every time I try to 
restart it. It happened first at firefly 0.80.8, I updated to 0.80.10, 
but it continued to happen.


Due to this failure, I have several PGs down+peering, that won't recover 
even marking the OSD out.


Could someone help me? Is it possible to edit/rebuild the leveldb-based 
log that seems to be causing the problem?


Here is what the logfile informs me:

[(12:54:45) root@spcsnp2 ~]# service ceph start osd.31
=== osd.31 ===
create-or-move updated item name 'osd.31' weight 2.73 at location 
{host=spcsnp2,root=default} to crush map

Starting Ceph osd.31 on spcsnp2...
starting osd.31 at :/0 osd_data /var/lib/ceph/osd/ceph-31 
/var/lib/ceph/osd/ceph-31/journal
2015-08-07 12:55:12.916880 7fd614c8f780  0 ceph version 0.80.10 
(ea6c958c38df1216bf95c927f143d8b13c4a9e70), process ceph-osd, pid 23260
[(12:55:12) root@spcsnp2 ~]# 2015-08-07 12:55:12.928614 7fd614c8f780  0 
filestore(/var/lib/ceph/osd/ceph-31) mount detected xfs (libxfs)
2015-08-07 12:55:12.928622 7fd614c8f780  1 
filestore(/var/lib/ceph/osd/ceph-31)  disabling 'filestore replica 
fadvise' due to known issues with fadvise(DONTNEED) on xfs
2015-08-07 12:55:12.931410 7fd614c8f780  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-31) detect_features: 
FIEMAP ioctl is supported and appears to work
2015-08-07 12:55:12.931419 7fd614c8f780  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-31) detect_features: 
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-08-07 12:55:12.939290 7fd614c8f780  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-31) detect_features: 
syscall(SYS_syncfs, fd) fully supported
2015-08-07 12:55:12.939326 7fd614c8f780  0 
xfsfilestorebackend(/var/lib/ceph/osd/ceph-31) detect_feature: extsize 
is disabled by conf

2015-08-07 12:55:45.587019 7fd614c8f780 -1 *** Caught signal (Aborted) **
 in thread 7fd614c8f780

 ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
 1: /usr/bin/ceph-osd() [0xab7562]
 2: (()+0xf030) [0x7fd6141ce030]
 3: (gsignal()+0x35) [0x7fd612d41475]
 4: (abort()+0x180) [0x7fd612d446f0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fd61359689d]
 6: (()+0x63996) [0x7fd613594996]
 7: (()+0x639c3) [0x7fd6135949c3]
 8: (()+0x63bee) [0x7fd613594bee]
 9: (tc_new()+0x48e) [0x7fd614414aee]
 10: (std::string::_Rep::_S_create(unsigned long, unsigned long, 
std::allocatorchar const)+0x59) [0x7fd6135f0999]
 11: (std::string::_Rep::_M_clone(std::allocatorchar const, unsigned 
long)+0x28) [0x7fd6135f1708]

 12: (std::string::reserve(unsigned long)+0x30) [0x7fd6135f17f0]
 13: (std::string::append(char const*, unsigned long)+0xb5) 
[0x7fd6135f1ab5]
 14: (leveldb::log::Reader::ReadRecord(leveldb::Slice*, 
std::string*)+0x2a2) [0x7fd614670fa2]
 15: (leveldb::DBImpl::RecoverLogFile(unsigned long, 
leveldb::VersionEdit*, unsigned long*)+0x180) [0x7fd614669360]
 16: (leveldb::DBImpl::Recover(leveldb::VersionEdit*)+0x5c2) 
[0x7fd61466bdf2]
 17: (leveldb::DB::Open(leveldb::Options const, std::string const, 
leveldb::DB**)+0xff) [0x7fd61466c11f]

 18: (LevelDBStore::do_open(std::ostream, bool)+0xd8) [0xa123a8]
 19: (FileStore::mount()+0x18e0) [0x9b7080]
 20: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x78f52a]
 21: (main()+0x2234) [0x7331c4]
 22: (__libc_start_main()+0xfd) [0x7fd612d2dead]
 23: /usr/bin/ceph-osd() [0x736e99]
 NOTE: a copy of the executable, or `objdump -rdS executable` is 
needed to interpret this.


--- begin dump of recent events ---
   -56 2015-08-07 12:55:12.915675 7fd614c8f780  5 asok(0x1a20230) 
register_command perfcounters_dump hook 0x1a10010
   -55 2015-08-07 12:55:12.915697 7fd614c8f780  5 asok(0x1a20230) 
register_command 1 hook 0x1a10010
   -54 2015-08-07 12:55:12.915700 7fd614c8f780  5 asok(0x1a20230) 
register_command perf dump hook 0x1a10010
   -53 2015-08-07 12:55:12.915704 7fd614c8f780  5 asok(0x1a20230) 
register_command perfcounters_schema hook 0x1a10010
   -52 2015-08-07 12:55:12.915706 7fd614c8f780  5 asok(0x1a20230) 
register_command 2 hook 0x1a10010
   -51 2015-08-07 12:55:12.915709 7fd614c8f780  5 asok(0x1a20230) 
register_command perf schema hook 0x1a10010
   -50 2015-08-07 12:55:12.915711 7fd614c8f780  5 asok(0x1a20230) 
register_command config show hook 0x1a10010
   -49 2015-08-07 12:55:12.915714 7fd614c8f780  5 asok(0x1a20230) 
register_command config set hook 0x1a10010
   -48 2015-08-07 12:55:12.915716 7fd614c8f780  5 asok(0x1a20230) 
register_command config get hook 0x1a10010
   -47 2015-08-07 12:55:12.915718 7fd614c8f780  5 asok(0x1a20230) 
register_command log flush hook 0x1a10010
   -46 2015-08-07 12:55:12.915721 7fd614c8f780  5 asok(0x1a20230) 
register_command log dump hook 0x1a10010
   -45 2015-08-07 12:55:12.915723 7fd614c8f780  5 asok(0x1a20230) 
register_command log reopen hook 0x1a10010
   -44 2015-08-07 12:55:12.916880 7fd614c8f780  0 ceph version 0.80.10 
(ea6c958c38df1216bf95c927f143d8b13c4a9e70), process ceph-osd, pid 23260
   -43 2015-08-07

Re: [ceph-users] OSD crashes

2015-07-22 Thread Alex Gorbachev

We have been error free for almost 3 weeks now.  The following settings on
all OSD nodes were changed:

vm.swappiness=1
vm.min_free_kbytes=262144

My discussion on XFS list is here:
http://www.spinics.net/lists/xfs/msg33645.html

Thanks,
Alex

On Fri, Jul 3, 2015 at 6:27 AM, Jan Schermer j...@schermer.cz wrote:

 What’s the value of /proc/sys/vm/min_free_kbytes on your system? Increase
 it to 256M (better do it if there’s lots of free memory) and see if it
 helps.
 It can also be set too high, hard to find any formula how to set it
 correctly...

 Jan


 On 03 Jul 2015, at 10:16, Alex Gorbachev a...@iss-integration.com wrote:

 Hello, we are experiencing severe OSD timeouts, OSDs are not taken out and
 we see the following in syslog on Ubuntu 14.04.2 with Firefly 0.80.9.

 Thank you for any advice.

 Alex


 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261899] BUG: unable to
 handle kernel paging request at 0019001c
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261923] IP:
 [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261941] PGD 1035954067 PUD 0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261955] Oops:  [#1] SMP
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261969] Modules linked in:
 xfs libcrc32c ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal
 intel_powerclamp co
 retemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
 aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core
 lpc_ich joy
 dev mei_me mei ioatdma wmi 8021q ipmi_si garp 8250_fintek mrp
 ipmi_msghandler stp llc bonding mac_hid lp parport mlx4_en vxlan
 ip6_udp_tunnel udp_tunnel hid_
 generic usbhid hid igb ahci mpt2sas mlx4_core i2c_algo_bit libahci dca
 raid_class ptp scsi_transport_sas pps_core arcmsr
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262182] CPU: 10 PID: 8711
 Comm: ceph-osd Not tainted 4.1.0-040100-generic #201506220235
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262197] Hardware name:
 Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262215] task:
 8800721f1420 ti: 880fbad54000 task.ti: 880fbad54000
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262229] RIP:
 0010:[8118e476]  [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262248] RSP:
 0018:880fbad571a8  EFLAGS: 00010246
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262258] RAX:
 880004000158 RBX: 000e RCX: 
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262303] RDX:
 880004000158 RSI: 880fbad571c0 RDI: 0019
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262347] RBP:
 880fbad57208 R08: 00c0 R09: 00ff
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262391] R10:
  R11: 0220 R12: 00b6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262435] R13:
 880fbad57268 R14: 000a R15: 880fbad572d8
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262479] FS:
  7f98cb0e0700() GS:88103f48() knlGS:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262524] CS:  0010 DS: 
 ES:  CR0: 80050033
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262551] CR2:
 0019001c CR3: 001034f0e000 CR4: 000407e0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262596] Stack:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262618]  880fbad571f8
 880cf6076b30 880bdde05da8 00e6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262669]  0100
 880cf6076b28 00b5 880fbad57258
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262721]  880fbad57258
 880fbad572d8  880cf6076b28
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262772] Call Trace:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262801]
  [8119b482] pagevec_lookup_entries+0x22/0x30
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262831]
  [8119bd84] truncate_inode_pages_range+0xf4/0x700
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262862]
  [8119c415] truncate_inode_pages+0x15/0x20
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262891]
  [8119c53f] truncate_inode_pages_final+0x5f/0xa0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262949]
  [c0431c2c] xfs_fs_evict_inode+0x3c/0xe0 [xfs]
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262981]
  [81220558] evict+0xb8/0x190
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263009]
  [81220671] dispose_list+0x41/0x50
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263037]
  [8122176f] prune_icache_sb+0x4f/0x60
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263067]
  [81208ab5] super_cache_scan+0x155/0x1a0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263096]
  [8119d26f] do_shrink_slab+0x13f/0x2c0
 Jul  3 03:42:06

Re: [ceph-users] OSD crashes

2015-07-03 Thread Alex Gorbachev

Thanks Jan.  /proc/sys/vm/min_free_kbytes was set to 32M, I set it to 256M
with system having 64 GB RAM.  Also my swappiness was set to 0, no problems
in lab tests, but I wonder if we hit some limit on 24/7 OSD operation.

I will update after some days of running with these parameter.  Best
regards, Alex

On Fri, Jul 3, 2015 at 6:27 AM, Jan Schermer j...@schermer.cz wrote:

 What’s the value of /proc/sys/vm/min_free_kbytes on your system? Increase
 it to 256M (better do it if there’s lots of free memory) and see if it
 helps.
 It can also be set too high, hard to find any formula how to set it
 correctly...

 Jan


 On 03 Jul 2015, at 10:16, Alex Gorbachev a...@iss-integration.com wrote:

 Hello, we are experiencing severe OSD timeouts, OSDs are not taken out and
 we see the following in syslog on Ubuntu 14.04.2 with Firefly 0.80.9.

 Thank you for any advice.

 Alex


 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261899] BUG: unable to
 handle kernel paging request at 0019001c
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261923] IP:
 [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261941] PGD 1035954067 PUD 0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261955] Oops:  [#1] SMP
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261969] Modules linked in:
 xfs libcrc32c ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal
 intel_powerclamp co
 retemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
 aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core
 lpc_ich joy
 dev mei_me mei ioatdma wmi 8021q ipmi_si garp 8250_fintek mrp
 ipmi_msghandler stp llc bonding mac_hid lp parport mlx4_en vxlan
 ip6_udp_tunnel udp_tunnel hid_
 generic usbhid hid igb ahci mpt2sas mlx4_core i2c_algo_bit libahci dca
 raid_class ptp scsi_transport_sas pps_core arcmsr
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262182] CPU: 10 PID: 8711
 Comm: ceph-osd Not tainted 4.1.0-040100-generic #201506220235
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262197] Hardware name:
 Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262215] task:
 8800721f1420 ti: 880fbad54000 task.ti: 880fbad54000
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262229] RIP:
 0010:[8118e476]  [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262248] RSP:
 0018:880fbad571a8  EFLAGS: 00010246
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262258] RAX:
 880004000158 RBX: 000e RCX: 
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262303] RDX:
 880004000158 RSI: 880fbad571c0 RDI: 0019
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262347] RBP:
 880fbad57208 R08: 00c0 R09: 00ff
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262391] R10:
  R11: 0220 R12: 00b6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262435] R13:
 880fbad57268 R14: 000a R15: 880fbad572d8
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262479] FS:
  7f98cb0e0700() GS:88103f48() knlGS:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262524] CS:  0010 DS: 
 ES:  CR0: 80050033
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262551] CR2:
 0019001c CR3: 001034f0e000 CR4: 000407e0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262596] Stack:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262618]  880fbad571f8
 880cf6076b30 880bdde05da8 00e6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262669]  0100
 880cf6076b28 00b5 880fbad57258
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262721]  880fbad57258
 880fbad572d8  880cf6076b28
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262772] Call Trace:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262801]
  [8119b482] pagevec_lookup_entries+0x22/0x30
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262831]
  [8119bd84] truncate_inode_pages_range+0xf4/0x700
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262862]
  [8119c415] truncate_inode_pages+0x15/0x20
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262891]
  [8119c53f] truncate_inode_pages_final+0x5f/0xa0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262949]
  [c0431c2c] xfs_fs_evict_inode+0x3c/0xe0 [xfs]
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262981]
  [81220558] evict+0xb8/0x190
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263009]
  [81220671] dispose_list+0x41/0x50
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263037]
  [8122176f] prune_icache_sb+0x4f/0x60
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263067]
  [81208ab5] super_cache_scan+0x155/0x1a0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263096]

Re: [ceph-users] OSD crashes

2015-07-03 Thread Jan Schermer

What’s the value of /proc/sys/vm/min_free_kbytes on your system? Increase it to 
256M (better do it if there’s lots of free memory) and see if it helps.
It can also be set too high, hard to find any formula how to set it correctly...

Jan


 On 03 Jul 2015, at 10:16, Alex Gorbachev a...@iss-integration.com wrote:
 
 Hello, we are experiencing severe OSD timeouts, OSDs are not taken out and we 
 see the following in syslog on Ubuntu 14.04.2 with Firefly 0.80.9.
 
 Thank you for any advice.
 
 Alex
 
 
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261899] BUG: unable to handle 
 kernel paging request at 0019001c
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261923] IP: 
 [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261941] PGD 1035954067 PUD 0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261955] Oops:  [#1] SMP
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.261969] Modules linked in: xfs 
 libcrc32c ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp 
 co
 retemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel 
 aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core 
 lpc_ich joy
 dev mei_me mei ioatdma wmi 8021q ipmi_si garp 8250_fintek mrp ipmi_msghandler 
 stp llc bonding mac_hid lp parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel 
 hid_
 generic usbhid hid igb ahci mpt2sas mlx4_core i2c_algo_bit libahci dca 
 raid_class ptp scsi_transport_sas pps_core arcmsr
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262182] CPU: 10 PID: 8711 Comm: 
 ceph-osd Not tainted 4.1.0-040100-generic #201506220235
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262197] Hardware name: 
 Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262215] task: 8800721f1420 
 ti: 880fbad54000 task.ti: 880fbad54000
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262229] RIP: 
 0010:[8118e476]  [8118e476] find_get_entries+0x66/0x160
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262248] RSP: 
 0018:880fbad571a8  EFLAGS: 00010246
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262258] RAX: 880004000158 
 RBX: 000e RCX: 
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262303] RDX: 880004000158 
 RSI: 880fbad571c0 RDI: 0019
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262347] RBP: 880fbad57208 
 R08: 00c0 R09: 00ff
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262391] R10:  
 R11: 0220 R12: 00b6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262435] R13: 880fbad57268 
 R14: 000a R15: 880fbad572d8
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262479] FS:  
 7f98cb0e0700() GS:88103f48() knlGS:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262524] CS:  0010 DS:  ES: 
  CR0: 80050033
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262551] CR2: 0019001c 
 CR3: 001034f0e000 CR4: 000407e0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262596] Stack:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262618]  880fbad571f8 
 880cf6076b30 880bdde05da8 00e6
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262669]  0100 
 880cf6076b28 00b5 880fbad57258
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262721]  880fbad57258 
 880fbad572d8  880cf6076b28
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262772] Call Trace:
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262801]  [8119b482] 
 pagevec_lookup_entries+0x22/0x30
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262831]  [8119bd84] 
 truncate_inode_pages_range+0xf4/0x700
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262862]  [8119c415] 
 truncate_inode_pages+0x15/0x20
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262891]  [8119c53f] 
 truncate_inode_pages_final+0x5f/0xa0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262949]  [c0431c2c] 
 xfs_fs_evict_inode+0x3c/0xe0 [xfs]
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.262981]  [81220558] 
 evict+0xb8/0x190
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263009]  [81220671] 
 dispose_list+0x41/0x50
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263037]  [8122176f] 
 prune_icache_sb+0x4f/0x60
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263067]  [81208ab5] 
 super_cache_scan+0x155/0x1a0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263096]  [8119d26f] 
 do_shrink_slab+0x13f/0x2c0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263126]  [811a22b0] ? 
 shrink_lruvec+0x330/0x370
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263157]  [811b4189] ? 
 isolate_migratepages_block+0x299/0x5c0
 Jul  3 03:42:06 roc-4r-sca020 kernel: [554036.263188]  [8119d558]

Re: [ceph-users] osd crashes with large object size (>10GB) in luminos Rados

[ceph-users] osd crashes with large object size (>10GB) in luminos Rados

Re: [ceph-users] OSD crashes on EC recovery

[ceph-users] OSD crashes on EC recovery

Re: [ceph-users] OSD Crashes

Re: [ceph-users] OSD Crashes

Re: [ceph-users] OSD Crashes

Re: [ceph-users] OSD Crashes

Re: [ceph-users] OSD Crashes

Re: [ceph-users] OSD Crashes

[ceph-users] OSD Crashes

[ceph-users] OSD crashes when starting

Re: [ceph-users] OSD crashes

Re: [ceph-users] OSD crashes

Re: [ceph-users] OSD crashes

15 matches

Site Navigation

Mail list logo

Footer information