[Cluster-devel] [PATCHv3 dlm/next 20/20] fs: dlm: check for invalid namelen
This patch adds an additional check inside the dlm locking from user space functionality that the namelen isn't above the maximum allowed dlm resource name length. If the namelen is above the maximum allowed we have a invalid state and out of buffer access can occur. Cut off the namelen attribute to maximum size is not an option because we might run into name conflicts and the user should be get aware of that. Signed-off-by: Alexander Aring --- fs/dlm/user.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/dlm/user.c b/fs/dlm/user.c index e5cefa90b1ce..9f2f743eeb31 100644 --- a/fs/dlm/user.c +++ b/fs/dlm/user.c @@ -241,6 +241,9 @@ static int device_user_lock(struct dlm_user_proc *proc, uint32_t lkid; int error = -ENOMEM; + if (params->namelen > DLM_RESNAME_MAXLEN) + return -EINVAL; + ls = dlm_find_lockspace_local(proc->lockspace); if (!ls) return -ENOENT; -- 2.26.2
[Cluster-devel] [PATCHv3 dlm/next 19/20] fs: dlm: remove obsolete code and comment
This patch removes obsolete macros and a comment. The macro was there for temporary copy a dlm message on a stack buffer, nowadays we operate directly on receive buffer which was given to the socket receive API. Signed-off-by: Alexander Aring --- fs/dlm/dlm_internal.h | 6 -- 1 file changed, 6 deletions(-) diff --git a/fs/dlm/dlm_internal.h b/fs/dlm/dlm_internal.h index da2297705713..a735f2816ef7 100644 --- a/fs/dlm/dlm_internal.h +++ b/fs/dlm/dlm_internal.h @@ -41,12 +41,6 @@ #include #include "config.h" -/* Size of the temp buffer midcomms allocates on the stack. - We try to make this large enough so most messages fit. - FIXME: should sctp make this unnecessary? */ - -#define DLM_INBUF_LEN 148 - struct dlm_ls; struct dlm_lkb; struct dlm_rsb; -- 2.26.2
[Cluster-devel] [PATCHv3 dlm/next 16/20] fs: dlm: add per node receive flush
This patch will add a functionality to flush pending dlm messages from
the receiving workqueue. Upcoming patches will use it to make sure that
nothing can be received e.g. after a node gets removed from the nodes
hash. Receiving messages will occur into a node lookup which might
create an node for the hash again.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 9 +
fs/dlm/lowcomms.h | 1 +
2 files changed, 10 insertions(+)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 9a3899ad1765..0e4cbabc680f 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1607,6 +1607,15 @@ static void clean_one_writequeue(struct connection *con)
spin_unlock_bh(>writequeue_lock);
}
+void dlm_lowcomms_rx_flush(int nodeid)
+{
+ struct connection *con;
+
+ con = nodeid2con(nodeid, 0);
+ if (con)
+ flush_work(>rwork);
+}
+
/* Called from recovery when it knows that a node has
left the cluster */
int dlm_lowcomms_close(int nodeid)
diff --git a/fs/dlm/lowcomms.h b/fs/dlm/lowcomms.h
index 8286531f9a9e..c4c789a68cf6 100644
--- a/fs/dlm/lowcomms.h
+++ b/fs/dlm/lowcomms.h
@@ -39,6 +39,7 @@ int dlm_lowcomms_addr(int nodeid, struct sockaddr_storage
*addr, int len);
void dlm_lowcomms_put_buffer(void *mh);
void dlm_lowcomms_get_buffer(void *mh);
void dlm_lowcomms_resend_buffer(void *mh);
+void dlm_lowcomms_rx_flush(int nodeid);
#endif /* __LOWCOMMS_DOT_H__ */
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 17/20] fs: dlm: add reliable connection if reconnect
This patch introduce to make a tcp lowcomms connection reliable even if
reconnects occurs. This is done by an application layer re-transmission
handling and sequence numbers in dlm protocols. There are three new dlm
commands:
DLM_OPTS:
This will encapsulate an existing dlm message (and rcom message if they
don't have an own application side re-transmission handling). As optional
handling additional tlv's (type length fields) can be appended. This can
be for example a sequence number field. However because in DLM_OPTS the
lockspace field is unused and a sequence number is a mandatory field it
isn't made as a tlv and we put the sequence number inside the lockspace
id. The possibility to add optional options are still there for future
purposes.
DLM_ACK:
Just a dlm header to acknowledge the receive of a DLM_OPTS message to
it's sender.
DLM_FIN:
A new DLM message to synchronize pending message to the other dlm end if
the node want to disconnects. Each side waits until it receives this
message and disconnects. It's important that this message has nothing to
do with the application logic because it might run in a timeout if
acknowledge messages are dropped.
To explain the basic functionality take a look into the
dlm_midcomms_receive_buffer() function. This function will take care
that dlm messages are delivered according to their sequence numbers and
request re-transmission via sending acknowledge messages. However there
exists three cases:
1. sequence number is the one which is expected. That means everything
is working fine. Additional there is always a check if the next
message was already queued for future, this will occur when there was
some messages drops before.
2. A sequence number is in the future, in this case we queue it for might
future delivery, see case 1.
3. A sequence number is in the past, in this case we drop this message
because it was already delivered.
To send acknowledge we always send the sequence number which is
expected, if the other node sends multiple acknowledge for the same
sequence numbers it will trigger a re-transmission. In case no acknowledge
is send back, a timer with a timeout handling is running and will trigger
a re-tranmission as well. Sending multiple ack's with the same sequence or
messages with the same sequence should not have any effects that breaks
dlm. Only messages in the far future can break dlm, that's why important
that the closing connection is right synchronized with DLM_FIN which
also resets the sequence numbers.
As RCOM_STATUS and RCOM_NAMES messages are the first messages which are
exchanged and they have they own re-transmission handling, there exists
logic that these messages must be first. If these messages arrives we
store the dlm version field. This handling is on DLM 3.1 and after this
patch 3.2 the same. A backwards compatibility handling has been added
which seems to work on tests without tcpkill, however it's not recommended
to use DLM 3.1 and 3.2 at the same time, because DLM 3.2 tries to fix long
term bugs in the DLM protocol.
Signed-off-by: Alexander Aring
---
fs/dlm/dlm_internal.h | 30 +-
fs/dlm/lowcomms.h |7 +-
fs/dlm/midcomms.c | 1076 +++--
3 files changed, 1060 insertions(+), 53 deletions(-)
diff --git a/fs/dlm/dlm_internal.h b/fs/dlm/dlm_internal.h
index 917de7367a32..da2297705713 100644
--- a/fs/dlm/dlm_internal.h
+++ b/fs/dlm/dlm_internal.h
@@ -368,18 +368,26 @@ static inline int rsb_flag(struct dlm_rsb *r, enum
rsb_flags flag)
/* dlm_header is first element of all structs sent between nodes */
#define DLM_HEADER_MAJOR 0x0003
-#define DLM_HEADER_MINOR 0x0001
+#define DLM_HEADER_MINOR 0x0002
+
+#define DLM_VERSION_3_10x00030001
+#define DLM_VERSION_3_20x00030002
#define DLM_HEADER_SLOTS 0x0001
#define DLM_MSG1
#define DLM_RCOM 2
+#define DLM_OPTS 3
+#define DLM_ACK4
+#define DLM_FIN5
struct dlm_header {
uint32_th_version;
union {
/* for DLM_MSG and DLM_RCOM */
uint32_th_lockspace;
+ /* for DLM_ACK and DLM_OPTS */
+ uint32_th_seq;
} u;
uint32_th_nodeid; /* nodeid of sender */
uint16_th_length;
@@ -387,7 +395,6 @@ struct dlm_header {
uint8_t h_pad;
};
-
#define DLM_MSG_REQUEST1
#define DLM_MSG_CONVERT2
#define DLM_MSG_UNLOCK 3
@@ -455,10 +462,29 @@ struct dlm_rcom {
charrc_buf[];
};
+struct dlm_opt_header {
+ uint16_tt_type;
+ uint16_tt_length;
+ uint32_to_pad;
+ /* need to be 8 byte aligned */
+ chart_value[];
+};
+
+/* encapsulation header */
[Cluster-devel] [PATCHv3 dlm/next 18/20] fs: dlm: don't allow half transmitted messages
This patch will clean a dirty page buffer if a reconnect occurs. If a page
buffer was half transmitted we cannot start inside the middle of a dlm
message if a node connects again. I observed invalid length receptions
errors and was guessing that this behaviour occurs, after this patch I
never saw an invalid message length again. This patch might drops more
messages for dlm version 3.1 but 3.1 can't deal with half messages as
well, for 3.2 it might trigger more re-transmissions but will not leave dlm
in a broken state.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 84 ++-
1 file changed, 54 insertions(+), 30 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 0e4cbabc680f..44a34becb780 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -112,6 +112,7 @@ struct writequeue_entry {
int len;
int end;
int users;
+ bool dirty;
struct connection *con;
struct list_head msgs;
struct kref ref;
@@ -647,6 +648,36 @@ static void make_sockaddr(struct sockaddr_storage *saddr,
uint16_t port,
memset((char *)saddr + *addr_len, 0, sizeof(struct sockaddr_storage) -
*addr_len);
}
+static void dlm_page_release(struct kref *kref)
+{
+ struct writequeue_entry *e = container_of(kref, struct writequeue_entry,
+ ref);
+
+ __free_page(e->page);
+ kfree(e);
+}
+
+static void dlm_msg_release(struct kref *kref)
+{
+ struct dlm_msg *msg = container_of(kref, struct dlm_msg, ref);
+
+ kref_put(>entry->ref, dlm_page_release);
+ kfree(msg);
+}
+
+static void free_entry(struct writequeue_entry *e)
+{
+ struct dlm_msg *msg, *tmp;
+
+ list_for_each_entry_safe(msg, tmp, >msgs, list) {
+ list_del(>list);
+ kref_put(>ref, dlm_msg_release);
+ }
+
+ list_del(>list);
+ kref_put(>ref, dlm_page_release);
+}
+
static void dlm_close_sock(struct socket **sock)
{
if (*sock) {
@@ -661,6 +692,7 @@ static void close_connection(struct connection *con, bool
and_other,
bool tx, bool rx)
{
bool closing = test_and_set_bit(CF_CLOSING, >flags);
+ struct writequeue_entry *e;
if (tx && !closing && cancel_work_sync(>swork)) {
log_print("canceled swork for node %d", con->nodeid);
@@ -679,6 +711,26 @@ static void close_connection(struct connection *con, bool
and_other,
close_connection(con->othercon, false, true, true);
}
+ /* if we send a writequeue entry only a half way, we drop the
+* whole entry because reconnection and that we not start of the
+* middle of a msg which will confuse the other end.
+*
+* we can always drop messages because retransmits, but what we
+* cannot allow is to transmit half messages which may be processed
+* at the other side.
+*
+* our policy is to start on a clean state when disconnects, we don't
+* know what's send/received on transport layer in this case.
+*/
+ spin_lock_bh(>writequeue_lock);
+ if (!list_empty(>writequeue)) {
+ e = list_first_entry(>writequeue, struct writequeue_entry,
+list);
+ if (e->dirty)
+ free_entry(e);
+ }
+ spin_unlock_bh(>writequeue_lock);
+
con->rx_leftover = 0;
con->retries = 0;
clear_bit(CF_CONNECTED, >flags);
@@ -958,36 +1010,6 @@ static int accept_from_sock(struct listen_connection *con)
return result;
}
-static void dlm_page_release(struct kref *kref)
-{
- struct writequeue_entry *e = container_of(kref, struct writequeue_entry,
- ref);
-
- __free_page(e->page);
- kfree(e);
-}
-
-static void dlm_msg_release(struct kref *kref)
-{
- struct dlm_msg *msg = container_of(kref, struct dlm_msg, ref);
-
- kref_put(>entry->ref, dlm_page_release);
- kfree(msg);
-}
-
-static void free_entry(struct writequeue_entry *e)
-{
- struct dlm_msg *msg, *tmp;
-
- list_for_each_entry_safe(msg, tmp, >msgs, list) {
- list_del(>list);
- kref_put(>ref, dlm_msg_release);
- }
-
- list_del(>list);
- kref_put(>ref, dlm_page_release);
-}
-
/*
* writequeue_entry_complete - try to delete and free write queue entry
* @e: write queue entry to try to delete
@@ -999,6 +1021,8 @@ static void writequeue_entry_complete(struct
writequeue_entry *e, int completed)
{
e->offset += completed;
e->len -= completed;
+ /* signal that page was half way transmitted */
+ e->dirty = true;
if (e->len == 0 && e->users == 0)
free_entry(e);
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 11/20] fs: dlm: make new buffer handling softirq ready
This patch makes the writequeue and message handling ready to be called
from a softirq by using spinlock handling to stop software interrupts
on local cpu while they are hold. The coming midcomms re-transmit
handling will introduce a timer which is using this functionality when
the timer expires.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 28 ++--
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 438badc2d870..a84223b549ed 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1393,7 +1393,7 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
{
struct writequeue_entry *e;
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
if (!list_empty(>writequeue)) {
e = list_last_entry(>writequeue, struct writequeue_entry,
list);
if (DLM_WQ_REMAIN_BYTES(e) >= len) {
@@ -1405,12 +1405,12 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
e->end += len;
e->users++;
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
return e;
}
}
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
e = new_writequeue_entry(con, allocation);
if (!e)
@@ -1420,12 +1420,12 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
*ppc = page_address(e->page);
e->end += len;
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
if (cb)
cb(*ppc, priv);
list_add_tail(>list, >writequeue);
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
return e;
};
@@ -1474,7 +1474,7 @@ void dlm_lowcomms_commit_buffer(void *mh)
struct connection *con = e->con;
int users;
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
list_add(>list, >msgs);
kref_get(>ref);
@@ -1483,13 +1483,13 @@ void dlm_lowcomms_commit_buffer(void *mh)
goto out;
e->len = DLM_WQ_LENGTH_BYTES(e);
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
queue_work(send_workqueue, >swork);
return;
out:
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
return;
}
@@ -1520,7 +1520,7 @@ static void send_to_sock(struct connection *con)
if (con->sock == NULL)
goto out_connect;
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
for (;;) {
if (list_empty(>writequeue))
break;
@@ -1529,7 +1529,7 @@ static void send_to_sock(struct connection *con)
len = e->len;
offset = e->offset;
BUG_ON(len == 0 && e->users == 0);
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
ret = 0;
if (len) {
@@ -1557,10 +1557,10 @@ static void send_to_sock(struct connection *con)
count = 0;
}
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
writequeue_entry_complete(e, ret);
}
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
out:
mutex_unlock(>sock_mutex);
return;
@@ -1583,11 +1583,11 @@ static void clean_one_writequeue(struct connection *con)
{
struct writequeue_entry *e, *safe;
- spin_lock(>writequeue_lock);
+ spin_lock_bh(>writequeue_lock);
list_for_each_entry_safe(e, safe, >writequeue, list) {
free_entry(e);
}
- spin_unlock(>writequeue_lock);
+ spin_unlock_bh(>writequeue_lock);
}
/* Called from recovery when it knows that a node has
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 14/20] fs: dlm: remove unaligned memory access handling
This patch removes unaligned memory access handling for receiving
midcomms messages. This handling will not fix the unaligned memory
access in general. All messages should be length aligned to 8 bytes,
there exists cases where this isn't the case. It's part of the sending
handling to not send such messages. As the sending handling itself, with
the internal allocator of page buffers, can occur in unaligned memory
access of dlm message fields we just ignore that problem for now as it
seems this code is used by architecture which can handle it.
This patch adds a comment to take care about that problem in a major
bump of dlm protocol.
Signed-off-by: Alexander Aring
---
fs/dlm/midcomms.c | 31 +++
1 file changed, 15 insertions(+), 16 deletions(-)
diff --git a/fs/dlm/midcomms.c b/fs/dlm/midcomms.c
index e058e017c77d..45dd3d7d857f 100644
--- a/fs/dlm/midcomms.c
+++ b/fs/dlm/midcomms.c
@@ -22,8 +22,6 @@
* into packets and sends them to the comms layer.
*/
-#include
-
#include "dlm_internal.h"
#include "lowcomms.h"
#include "config.h"
@@ -93,13 +91,22 @@ int dlm_process_incoming_buffer(int nodeid, unsigned char
*buf, int len)
while (len >= sizeof(struct dlm_header)) {
hd = (struct dlm_header *)ptr;
- /* no message should be more than this otherwise we
-* cannot deliver this message to upper layers
+ /* no message should be more than DEFAULT_BUFFER_SIZE or
+* less than dlm_header size.
+*
+* Some messages does not have a 8 byte length boundary yet
+* which can occur in a unaligned memory access of some dlm
+* messages. However this problem need to be fixed at the
+* sending side, for now it seems nobody run into architecture
+* related issues yet but it slows down some processing.
+* Fixing this issue should be scheduled in future by doing
+* the next major version bump.
*/
- msglen = get_unaligned_le16(>h_length);
- if (msglen > DEFAULT_BUFFER_SIZE) {
- log_print("received invalid length header: %u, will
abort message parsing",
- msglen);
+ msglen = le16_to_cpu(hd->h_length);
+ if (msglen > DEFAULT_BUFFER_SIZE ||
+ msglen < sizeof(struct dlm_header)) {
+ log_print("received invalid length header: %u from node
%d, will abort message parsing",
+ msglen, nodeid);
return -EBADMSG;
}
@@ -132,14 +139,6 @@ int dlm_process_incoming_buffer(int nodeid, unsigned char
*buf, int len)
goto skip;
}
- /* for aligned memory access, we just copy current message
-* to begin of the buffer which contains already parsed buffer
-* data and should provide align access for upper layers
-* because the start address of the buffer has a aligned
-* address. This memmove can be removed when the upperlayer
-* is capable of unaligned memory access.
-*/
- memmove(buf, ptr, msglen);
dlm_receive_buffer((union dlm_packet *)buf, nodeid);
skip:
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 15/20] fs: dlm: add union in dlm header for lockspace id
This patch adds union inside the lockspace id to handle it also for
another use case for a different dlm command.
Signed-off-by: Alexander Aring
---
fs/dlm/dlm_internal.h | 5 -
fs/dlm/lock.c | 8
fs/dlm/midcomms.c | 1 -
fs/dlm/rcom.c | 4 ++--
fs/dlm/util.c | 6 --
5 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/fs/dlm/dlm_internal.h b/fs/dlm/dlm_internal.h
index 04fe9f525ac7..917de7367a32 100644
--- a/fs/dlm/dlm_internal.h
+++ b/fs/dlm/dlm_internal.h
@@ -377,7 +377,10 @@ static inline int rsb_flag(struct dlm_rsb *r, enum
rsb_flags flag)
struct dlm_header {
uint32_th_version;
- uint32_th_lockspace;
+ union {
+ /* for DLM_MSG and DLM_RCOM */
+ uint32_th_lockspace;
+ } u;
uint32_th_nodeid; /* nodeid of sender */
uint16_th_length;
uint8_t h_cmd; /* DLM_MSG, DLM_RCOM */
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index b3fd823009f4..daa5747c8556 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -3544,7 +3544,7 @@ static int _create_message(struct dlm_ls *ls, int mb_len,
ms = (struct dlm_message *) mb;
ms->m_header.h_version = (DLM_HEADER_MAJOR | DLM_HEADER_MINOR);
- ms->m_header.h_lockspace = ls->ls_global_id;
+ ms->m_header.u.h_lockspace = ls->ls_global_id;
ms->m_header.h_nodeid = dlm_our_nodeid();
ms->m_header.h_length = mb_len;
ms->m_header.h_cmd = DLM_MSG;
@@ -5038,16 +5038,16 @@ void dlm_receive_buffer(union dlm_packet *p, int nodeid)
if (hd->h_nodeid != nodeid) {
log_print("invalid h_nodeid %d from %d lockspace %x",
- hd->h_nodeid, nodeid, hd->h_lockspace);
+ hd->h_nodeid, nodeid, hd->u.h_lockspace);
return;
}
- ls = dlm_find_lockspace_global(hd->h_lockspace);
+ ls = dlm_find_lockspace_global(hd->u.h_lockspace);
if (!ls) {
if (dlm_config.ci_log_debug) {
printk_ratelimited(KERN_DEBUG "dlm: invalid lockspace "
"%u from %d cmd %d type %d\n",
- hd->h_lockspace, nodeid, hd->h_cmd, type);
+ hd->u.h_lockspace, nodeid, hd->h_cmd, type);
}
if (hd->h_cmd == DLM_RCOM && type == DLM_RCOM_STATUS)
diff --git a/fs/dlm/midcomms.c b/fs/dlm/midcomms.c
index 45dd3d7d857f..71a1f0c910b3 100644
--- a/fs/dlm/midcomms.c
+++ b/fs/dlm/midcomms.c
@@ -149,4 +149,3 @@ int dlm_process_incoming_buffer(int nodeid, unsigned char
*buf, int len)
return ret;
}
-
diff --git a/fs/dlm/rcom.c b/fs/dlm/rcom.c
index 7a7d4a8e4706..06f7a5f1d99d 100644
--- a/fs/dlm/rcom.c
+++ b/fs/dlm/rcom.c
@@ -49,7 +49,7 @@ static int create_rcom(struct dlm_ls *ls, int to_nodeid, int
type, int len,
rc = (struct dlm_rcom *) mb;
rc->rc_header.h_version = (DLM_HEADER_MAJOR | DLM_HEADER_MINOR);
- rc->rc_header.h_lockspace = ls->ls_global_id;
+ rc->rc_header.u.h_lockspace = ls->ls_global_id;
rc->rc_header.h_nodeid = dlm_our_nodeid();
rc->rc_header.h_length = mb_len;
rc->rc_header.h_cmd = DLM_RCOM;
@@ -476,7 +476,7 @@ int dlm_send_ls_not_ready(int nodeid, struct dlm_rcom
*rc_in)
rc = (struct dlm_rcom *) mb;
rc->rc_header.h_version = (DLM_HEADER_MAJOR | DLM_HEADER_MINOR);
- rc->rc_header.h_lockspace = rc_in->rc_header.h_lockspace;
+ rc->rc_header.u.h_lockspace = rc_in->rc_header.u.h_lockspace;
rc->rc_header.h_nodeid = dlm_our_nodeid();
rc->rc_header.h_length = mb_len;
rc->rc_header.h_cmd = DLM_RCOM;
diff --git a/fs/dlm/util.c b/fs/dlm/util.c
index 74a8c5bfe9b5..58acbcc2081a 100644
--- a/fs/dlm/util.c
+++ b/fs/dlm/util.c
@@ -23,7 +23,8 @@
void header_out(struct dlm_header *hd)
{
hd->h_version = cpu_to_le32(hd->h_version);
- hd->h_lockspace = cpu_to_le32(hd->h_lockspace);
+ /* does it for others u32 in union as well */
+ hd->u.h_lockspace = cpu_to_le32(hd->u.h_lockspace);
hd->h_nodeid= cpu_to_le32(hd->h_nodeid);
hd->h_length= cpu_to_le16(hd->h_length);
}
@@ -31,7 +32,8 @@ void header_out(struct dlm_header *hd)
void header_in(struct dlm_header *hd)
{
hd->h_version = le32_to_cpu(hd->h_version);
- hd->h_lockspace = le32_to_cpu(hd->h_lockspace);
+ /* does it for others u32 in union as well */
+ hd->u.h_lockspace = le32_to_cpu(hd->u.h_lockspace);
hd->h_nodeid= le32_to_cpu(hd->h_nodeid);
hd->h_length= le16_to_cpu(hd->h_length);
}
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 13/20] fs: dlm: move out some hash functionality
This patch moves out some lowcomms hash functionality into lowcomms
header to provide them to other layers like midcomms as well.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 9 -
fs/dlm/lowcomms.h | 10 ++
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index f021dfecff91..9a3899ad1765 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -59,7 +59,6 @@
#include "config.h"
#define NEEDED_RMEM (4*1024*1024)
-#define CONN_HASH_SIZE 32
/* Number of messages to send before rescheduling */
#define MAX_SEND_MSG_COUNT 25
@@ -165,14 +164,6 @@ static void sctp_connect_to_sock(struct connection *con);
static void tcp_connect_to_sock(struct connection *con);
static void dlm_tcp_shutdown(struct connection *con);
-/* This is deliberately very simple because most clusters have simple
- sequential nodeids, so we should be able to go straight to a connection
- struct in the array */
-static inline int nodeid_hash(int nodeid)
-{
- return nodeid & (CONN_HASH_SIZE-1);
-}
-
static struct connection *__find_con(int nodeid)
{
int r, idx;
diff --git a/fs/dlm/lowcomms.h b/fs/dlm/lowcomms.h
index 30924b6f03e1..8286531f9a9e 100644
--- a/fs/dlm/lowcomms.h
+++ b/fs/dlm/lowcomms.h
@@ -13,6 +13,16 @@
#define __LOWCOMMS_DOT_H__
#define LOWCOMMS_MAX_TX_BUFFER_LEN 4096
+#define CONN_HASH_SIZE 32
+
+/* This is deliberately very simple because most clusters have simple
+ * sequential nodeids, so we should be able to go straight to a connection
+ * struct in the array
+ */
+static inline int nodeid_hash(int nodeid)
+{
+ return nodeid & (CONN_HASH_SIZE-1);
+}
/* switch to check if dlm is running */
extern int dlm_allow_conn;
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 10/20] fs: dlm: make buffer handling per msg
This patch makes the void pointer handle for lowcomms functionality per
message and not per page allocation entry. A refcount handling for the
handle was added to keep the message alive until the user doesn't need
it anymore.
There exists now a per message callback which will be called when
allocating a new buffer. This callback will be guaranteed to be called
according the order of the sending buffer, which can be used that the
caller increments a sequence number.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 100 +-
fs/dlm/lowcomms.h | 5 ++-
fs/dlm/midcomms.c | 8 +++-
3 files changed, 101 insertions(+), 12 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 8c826e95493c..438badc2d870 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -114,6 +114,17 @@ struct writequeue_entry {
int end;
int users;
struct connection *con;
+ struct list_head msgs;
+ struct kref ref;
+};
+
+struct dlm_msg {
+ struct writequeue_entry *entry;
+ void *ppc;
+ int len;
+
+ struct list_head list;
+ struct kref ref;
};
struct dlm_node_addr {
@@ -956,12 +967,36 @@ static int accept_from_sock(struct listen_connection *con)
return result;
}
-static void free_entry(struct writequeue_entry *e)
+static void dlm_page_release(struct kref *kref)
{
+ struct writequeue_entry *e = container_of(kref, struct writequeue_entry,
+ ref);
+
__free_page(e->page);
kfree(e);
}
+static void dlm_msg_release(struct kref *kref)
+{
+ struct dlm_msg *msg = container_of(kref, struct dlm_msg, ref);
+
+ kref_put(>entry->ref, dlm_page_release);
+ kfree(msg);
+}
+
+static void free_entry(struct writequeue_entry *e)
+{
+ struct dlm_msg *msg, *tmp;
+
+ list_for_each_entry_safe(msg, tmp, >msgs, list) {
+ list_del(>list);
+ kref_put(>ref, dlm_msg_release);
+ }
+
+ list_del(>list);
+ kref_put(>ref, dlm_page_release);
+}
+
/*
* writequeue_entry_complete - try to delete and free write queue entry
* @e: write queue entry to try to delete
@@ -974,10 +1009,8 @@ static void writequeue_entry_complete(struct
writequeue_entry *e, int completed)
e->offset += completed;
e->len -= completed;
- if (e->len == 0 && e->users == 0) {
- list_del(>list);
+ if (e->len == 0 && e->users == 0)
free_entry(e);
- }
}
/*
@@ -1347,12 +1380,16 @@ static struct writequeue_entry
*new_writequeue_entry(struct connection *con,
entry->con = con;
entry->users = 1;
+ kref_init(>ref);
+ INIT_LIST_HEAD(>msgs);
return entry;
}
static struct writequeue_entry *new_wq_entry(struct connection *con, int len,
-gfp_t allocation, char **ppc)
+gfp_t allocation, char **ppc,
+void (*cb)(void *buf, void *priv),
+void *priv)
{
struct writequeue_entry *e;
@@ -1360,7 +1397,12 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
if (!list_empty(>writequeue)) {
e = list_last_entry(>writequeue, struct writequeue_entry,
list);
if (DLM_WQ_REMAIN_BYTES(e) >= len) {
+ kref_get(>ref);
+
*ppc = page_address(e->page) + e->end;
+ if (cb)
+ cb(*ppc, priv);
+
e->end += len;
e->users++;
spin_unlock(>writequeue_lock);
@@ -1374,19 +1416,26 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
if (!e)
return NULL;
+ kref_get(>ref);
*ppc = page_address(e->page);
e->end += len;
spin_lock(>writequeue_lock);
+ if (cb)
+ cb(*ppc, priv);
+
list_add_tail(>list, >writequeue);
spin_unlock(>writequeue_lock);
return e;
};
-void *dlm_lowcomms_get_buffer(int nodeid, int len, gfp_t allocation, char
**ppc)
+void *dlm_lowcomms_new_buffer(int nodeid, int len, gfp_t allocation, char
**ppc,
+ void (*cb)(void *buf, void *priv), void *priv)
{
+ struct writequeue_entry *e;
struct connection *con;
+ struct dlm_msg *msg;
if (len > DEFAULT_BUFFER_SIZE ||
len < sizeof(struct dlm_header)) {
@@ -1399,16 +1448,36 @@ void *dlm_lowcomms_get_buffer(int nodeid, int len,
gfp_t allocation, char **ppc)
if (!con)
return NULL;
- return new_wq_entry(con, len, allocation, ppc);
+ msg = kzalloc(sizeof(*msg), allocation);
+ if (!msg)
+ return NULL;
+
+
[Cluster-devel] [PATCHv3 dlm/next 12/20] fs: dlm: add functionality to re-transmit a message
This patch introduces a irqsafe retransmit functionality for a lowcomms
message handle. It's just allocates a new buffer and transmit it again.
To avoid another connection look some refactor was done to make a new
buffer allocation with a preexisting connection pointer.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 54 +++
fs/dlm/lowcomms.h | 1 +
2 files changed, 41 insertions(+), 14 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index a84223b549ed..f021dfecff91 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1430,24 +1430,14 @@ static struct writequeue_entry *new_wq_entry(struct
connection *con, int len,
return e;
};
-void *dlm_lowcomms_new_buffer(int nodeid, int len, gfp_t allocation, char
**ppc,
- void (*cb)(void *buf, void *priv), void *priv)
+static void *dlm_lowcomms_new_buffer_con(struct connection *con, int len,
+gfp_t allocation, char **ppc,
+void (*cb)(void *buf, void *priv),
+void *priv)
{
struct writequeue_entry *e;
- struct connection *con;
struct dlm_msg *msg;
- if (len > DEFAULT_BUFFER_SIZE ||
- len < sizeof(struct dlm_header)) {
- BUILD_BUG_ON(PAGE_SIZE < DEFAULT_BUFFER_SIZE);
- log_print("failed to allocate a buffer of size %d", len);
- return NULL;
- }
-
- con = nodeid2con(nodeid, allocation);
- if (!con)
- return NULL;
-
msg = kzalloc(sizeof(*msg), allocation);
if (!msg)
return NULL;
@@ -1467,6 +1457,25 @@ void *dlm_lowcomms_new_buffer(int nodeid, int len, gfp_t
allocation, char **ppc,
return msg;
}
+void *dlm_lowcomms_new_buffer(int nodeid, int len, gfp_t allocation, char
**ppc,
+ void (*cb)(void *buf, void *priv), void *priv)
+{
+ struct connection *con;
+
+ if (len > DEFAULT_BUFFER_SIZE ||
+ len < sizeof(struct dlm_header)) {
+ BUILD_BUG_ON(PAGE_SIZE < DEFAULT_BUFFER_SIZE);
+ log_print("failed to allocate a buffer of size %d", len);
+ return NULL;
+ }
+
+ con = nodeid2con(nodeid, allocation);
+ if (!con)
+ return NULL;
+
+ return dlm_lowcomms_new_buffer_con(con, len, GFP_ATOMIC, ppc, cb, priv);
+}
+
void dlm_lowcomms_commit_buffer(void *mh)
{
struct dlm_msg *msg = mh;
@@ -1507,6 +1516,23 @@ void dlm_lowcomms_get_buffer(void *mh)
kref_get(>ref);
}
+/* irqsafe */
+void dlm_lowcomms_resend_buffer(void *mh)
+{
+ struct dlm_msg *msg = mh;
+ void *mh_new;
+ char *ppc;
+
+ mh_new = dlm_lowcomms_new_buffer_con(msg->entry->con, msg->len,
GFP_ATOMIC,
+, NULL, NULL);
+ if (!mh_new)
+ return;
+
+ memcpy(ppc, msg->ppc, msg->len);
+ dlm_lowcomms_commit_buffer(mh_new);
+ dlm_lowcomms_put_buffer(mh_new);
+}
+
/* Send a message */
static void send_to_sock(struct connection *con)
{
diff --git a/fs/dlm/lowcomms.h b/fs/dlm/lowcomms.h
index 6417c5fca215..30924b6f03e1 100644
--- a/fs/dlm/lowcomms.h
+++ b/fs/dlm/lowcomms.h
@@ -28,6 +28,7 @@ int dlm_lowcomms_connect_node(int nodeid);
int dlm_lowcomms_addr(int nodeid, struct sockaddr_storage *addr, int len);
void dlm_lowcomms_put_buffer(void *mh);
void dlm_lowcomms_get_buffer(void *mh);
+void dlm_lowcomms_resend_buffer(void *mh);
#endif /* __LOWCOMMS_DOT_H__ */
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 09/20] fs: dlm: add more midcomms hooks
This patch prepares hooks to redirect to the midcomms layer which will
be used by the midcomms re-transmit handling.
There exists the new concept of stateless buffers allocation and
commits. This can be used to bypass the midcomms re-transmit handling. It
is used by RCOM_STATUS and RCOM_NAMES messages, because they have their
own ping-like re-transmit handling. As well these two messages will be
used to determine the DLM version per node, because these two messages
are per observation the first messages which are exchanged.
The midcomms_remove_member() hook should be called when there is nothing
to send to the other node and the other node is still capable to
transmit dlm messages to the other node which called
midcomms_remove_member(). I experienced that the dlm protocol has a lack
of support for synchronize this event on protocol level. The result was
that there was still something to transmit but the other node was already
gone. This hook can be used to provide such synchronization. Although I
am not totally sure about the placement of this hook, I did not observed
issues yet when providing such synchronization on protocol layer.
Signed-off-by: Alexander Aring
---
fs/dlm/config.c| 3 ++-
fs/dlm/lock.c | 6 ++---
fs/dlm/lockspace.c | 5 +++--
fs/dlm/member.c| 16 ++
fs/dlm/member.h| 1 +
fs/dlm/midcomms.c | 44 +
fs/dlm/midcomms.h | 10 +
fs/dlm/rcom.c | 55 +++---
fs/dlm/recoverd.c | 3 +++
9 files changed, 115 insertions(+), 28 deletions(-)
diff --git a/fs/dlm/config.c b/fs/dlm/config.c
index ab26cf135710..ba8b1f104df3 100644
--- a/fs/dlm/config.c
+++ b/fs/dlm/config.c
@@ -20,6 +20,7 @@
#include
#include "config.h"
+#include "midcomms.h"
#include "lowcomms.h"
/*
@@ -532,7 +533,7 @@ static void drop_comm(struct config_group *g, struct
config_item *i)
struct dlm_comm *cm = config_item_to_comm(i);
if (local_comm == cm)
local_comm = NULL;
- dlm_lowcomms_close(cm->nodeid);
+ dlm_midcomms_close(cm->nodeid);
while (cm->addr_count--)
kfree(cm->addr[cm->addr_count]);
config_item_put(i);
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index b93df39d0915..b3fd823009f4 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -59,7 +59,7 @@
#include "dlm_internal.h"
#include
#include "memory.h"
-#include "lowcomms.h"
+#include "midcomms.h"
#include "requestqueue.h"
#include "util.h"
#include "dir.h"
@@ -3537,7 +3537,7 @@ static int _create_message(struct dlm_ls *ls, int mb_len,
pass into lowcomms_commit and a message buffer (mb) that we
write our data into */
- mh = dlm_lowcomms_get_buffer(to_nodeid, mb_len, GFP_NOFS, );
+ mh = dlm_midcomms_get_buffer(to_nodeid, mb_len, GFP_NOFS, );
if (!mh)
return -ENOBUFS;
@@ -3589,7 +3589,7 @@ static int create_message(struct dlm_rsb *r, struct
dlm_lkb *lkb,
static int send_message(struct dlm_mhandle *mh, struct dlm_message *ms)
{
dlm_message_out(ms);
- dlm_lowcomms_commit_buffer(mh);
+ dlm_midcomms_commit_buffer(mh);
return 0;
}
diff --git a/fs/dlm/lockspace.c b/fs/dlm/lockspace.c
index 561dcad08ad6..c9e0f5ac9f9a 100644
--- a/fs/dlm/lockspace.c
+++ b/fs/dlm/lockspace.c
@@ -16,6 +16,7 @@
#include "member.h"
#include "recoverd.h"
#include "dir.h"
+#include "midcomms.h"
#include "lowcomms.h"
#include "config.h"
#include "memory.h"
@@ -390,7 +391,7 @@ static int threads_start(void)
}
/* Thread for sending/receiving messages for all lockspace's */
- error = dlm_lowcomms_start();
+ error = dlm_midcomms_start();
if (error) {
log_print("cannot start dlm lowcomms %d", error);
goto scand_fail;
@@ -407,7 +408,7 @@ static int threads_start(void)
static void threads_stop(void)
{
dlm_scand_stop();
- dlm_lowcomms_stop();
+ dlm_midcomms_stop();
}
static int new_lockspace(const char *name, const char *cluster,
diff --git a/fs/dlm/member.c b/fs/dlm/member.c
index ceef3f2074ff..8291566766f3 100644
--- a/fs/dlm/member.c
+++ b/fs/dlm/member.c
@@ -15,6 +15,7 @@
#include "recover.h"
#include "rcom.h"
#include "config.h"
+#include "midcomms.h"
#include "lowcomms.h"
int dlm_slots_version(struct dlm_header *h)
@@ -521,6 +522,20 @@ static struct dlm_config_node *find_config_node(struct
dlm_recover *rv,
return NULL;
}
+void dlm_flush_removed_members(struct dlm_ls *ls, struct dlm_recover *rv)
+{
+ const struct dlm_config_node *node;
+ const struct dlm_member *memb;
+
+ list_for_each_entry(memb, >ls_nodes, list) {
+ node = find_config_node(rv, memb->nodeid);
+ if (node && !node->new)
+ continue;
+
+ midcomms_remove_member(memb->nodeid);
+ }
+}
+
int dlm_recover_members(struct dlm_ls *ls, struct
[Cluster-devel] [PATCHv3 dlm/next 08/20] fs: dlm: simplify writequeue handling
This patch cleans up the current dlm sending allocator handling by using
some named macros, list functionality and removes some goto statements.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 83 ---
1 file changed, 43 insertions(+), 40 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 02ab939ad8d3..8c826e95493c 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -102,6 +102,9 @@ struct listen_connection {
struct work_struct rwork;
};
+#define DLM_WQ_REMAIN_BYTES(e) (PAGE_SIZE - e->end)
+#define DLM_WQ_LENGTH_BYTES(e) (e->end - e->offset)
+
/* An entry waiting to be sent */
struct writequeue_entry {
struct list_head list;
@@ -1332,7 +1335,7 @@ static struct writequeue_entry
*new_writequeue_entry(struct connection *con,
{
struct writequeue_entry *entry;
- entry = kmalloc(sizeof(struct writequeue_entry), allocation);
+ entry = kzalloc(sizeof(*entry), allocation);
if (!entry)
return NULL;
@@ -1342,20 +1345,48 @@ static struct writequeue_entry
*new_writequeue_entry(struct connection *con,
return NULL;
}
- entry->offset = 0;
- entry->len = 0;
- entry->end = 0;
- entry->users = 0;
entry->con = con;
+ entry->users = 1;
return entry;
}
+static struct writequeue_entry *new_wq_entry(struct connection *con, int len,
+gfp_t allocation, char **ppc)
+{
+ struct writequeue_entry *e;
+
+ spin_lock(>writequeue_lock);
+ if (!list_empty(>writequeue)) {
+ e = list_last_entry(>writequeue, struct writequeue_entry,
list);
+ if (DLM_WQ_REMAIN_BYTES(e) >= len) {
+ *ppc = page_address(e->page) + e->end;
+ e->end += len;
+ e->users++;
+ spin_unlock(>writequeue_lock);
+
+ return e;
+ }
+ }
+ spin_unlock(>writequeue_lock);
+
+ e = new_writequeue_entry(con, allocation);
+ if (!e)
+ return NULL;
+
+ *ppc = page_address(e->page);
+ e->end += len;
+
+ spin_lock(>writequeue_lock);
+ list_add_tail(>list, >writequeue);
+ spin_unlock(>writequeue_lock);
+
+ return e;
+};
+
void *dlm_lowcomms_get_buffer(int nodeid, int len, gfp_t allocation, char
**ppc)
{
struct connection *con;
- struct writequeue_entry *e;
- int offset = 0;
if (len > DEFAULT_BUFFER_SIZE ||
len < sizeof(struct dlm_header)) {
@@ -1368,35 +1399,7 @@ void *dlm_lowcomms_get_buffer(int nodeid, int len, gfp_t
allocation, char **ppc)
if (!con)
return NULL;
- spin_lock(>writequeue_lock);
- e = list_entry(con->writequeue.prev, struct writequeue_entry, list);
- if ((>list == >writequeue) ||
- (PAGE_SIZE - e->end < len)) {
- e = NULL;
- } else {
- offset = e->end;
- e->end += len;
- e->users++;
- }
- spin_unlock(>writequeue_lock);
-
- if (e) {
- got_one:
- *ppc = page_address(e->page) + offset;
- return e;
- }
-
- e = new_writequeue_entry(con, allocation);
- if (e) {
- spin_lock(>writequeue_lock);
- offset = e->end;
- e->end += len;
- e->users++;
- list_add_tail(>list, >writequeue);
- spin_unlock(>writequeue_lock);
- goto got_one;
- }
- return NULL;
+ return new_wq_entry(con, len, allocation, ppc);
}
void dlm_lowcomms_commit_buffer(void *mh)
@@ -1409,7 +1412,8 @@ void dlm_lowcomms_commit_buffer(void *mh)
users = --e->users;
if (users)
goto out;
- e->len = e->end - e->offset;
+
+ e->len = DLM_WQ_LENGTH_BYTES(e);
spin_unlock(>writequeue_lock);
queue_work(send_workqueue, >swork);
@@ -1435,11 +1439,10 @@ static void send_to_sock(struct connection *con)
spin_lock(>writequeue_lock);
for (;;) {
- e = list_entry(con->writequeue.next, struct writequeue_entry,
- list);
- if ((struct list_head *) e == >writequeue)
+ if (list_empty(>writequeue))
break;
+ e = list_first_entry(>writequeue, struct writequeue_entry,
list);
len = e->len;
offset = e->offset;
BUG_ON(len == 0 && e->users == 0);
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 07/20] fs: dlm: use GFP_ZERO for page buffer
This patch uses GFP_ZERO for allocate a page for the internal dlm
sending buffer allocator instead of calling memset zero after every
allocation. An already allocated space will never be reused again.
Signed-off-by: Alexander Aring
---
fs/dlm/lock.c | 2 --
fs/dlm/lowcomms.c | 2 +-
fs/dlm/rcom.c | 2 --
3 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index 002123efc6b0..b93df39d0915 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -3541,8 +3541,6 @@ static int _create_message(struct dlm_ls *ls, int mb_len,
if (!mh)
return -ENOBUFS;
- memset(mb, 0, mb_len);
-
ms = (struct dlm_message *) mb;
ms->m_header.h_version = (DLM_HEADER_MAJOR | DLM_HEADER_MINOR);
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 2b468fbb2b43..02ab939ad8d3 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1336,7 +1336,7 @@ static struct writequeue_entry
*new_writequeue_entry(struct connection *con,
if (!entry)
return NULL;
- entry->page = alloc_page(allocation);
+ entry->page = alloc_page(allocation | __GFP_ZERO);
if (!entry->page) {
kfree(entry);
return NULL;
diff --git a/fs/dlm/rcom.c b/fs/dlm/rcom.c
index 73ddee5159d7..f5b1bd65728d 100644
--- a/fs/dlm/rcom.c
+++ b/fs/dlm/rcom.c
@@ -41,7 +41,6 @@ static int create_rcom(struct dlm_ls *ls, int to_nodeid, int
type, int len,
to_nodeid, type, len);
return -ENOBUFS;
}
- memset(mb, 0, mb_len);
rc = (struct dlm_rcom *) mb;
@@ -462,7 +461,6 @@ int dlm_send_ls_not_ready(int nodeid, struct dlm_rcom
*rc_in)
mh = dlm_lowcomms_get_buffer(nodeid, mb_len, GFP_NOFS, );
if (!mh)
return -ENOBUFS;
- memset(mb, 0, mb_len);
rc = (struct dlm_rcom *) mb;
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 05/20] fs: dlm: change allocation limits
While running tcpkill I experienced invalid header length values while
receiving to check that a node doesn't try to send a invalid dlm message
we also check on applications minimum allocation limit. Also use
DEFAULT_BUFFER_SIZE as maximum allocation limit. The define
LOWCOMMS_MAX_TX_BUFFER_LEN is to calculate maximum buffer limits on
application layer, future midcomms layer will subtract their needs from
this define.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index d25b9132c593..2b468fbb2b43 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1357,8 +1357,9 @@ void *dlm_lowcomms_get_buffer(int nodeid, int len, gfp_t
allocation, char **ppc)
struct writequeue_entry *e;
int offset = 0;
- if (len > LOWCOMMS_MAX_TX_BUFFER_LEN) {
- BUILD_BUG_ON(PAGE_SIZE < LOWCOMMS_MAX_TX_BUFFER_LEN);
+ if (len > DEFAULT_BUFFER_SIZE ||
+ len < sizeof(struct dlm_header)) {
+ BUILD_BUG_ON(PAGE_SIZE < DEFAULT_BUFFER_SIZE);
log_print("failed to allocate a buffer of size %d", len);
return NULL;
}
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 04/20] fs: dlm: add check if dlm is currently running
This patch adds checks for dlm config attributes regarding to protocol
parameters as it makes only sense to change them when dlm is not running.
It also adds a check for valid protocol specifiers and return invalid
argument if they are not supported.
Signed-off-by: Alexander Aring
---
fs/dlm/config.c | 34 --
fs/dlm/lowcomms.c | 2 +-
fs/dlm/lowcomms.h | 3 +++
3 files changed, 36 insertions(+), 3 deletions(-)
diff --git a/fs/dlm/config.c b/fs/dlm/config.c
index 73e6643903af..ab26cf135710 100644
--- a/fs/dlm/config.c
+++ b/fs/dlm/config.c
@@ -164,6 +164,36 @@ static ssize_t cluster_##name##_show(struct config_item
*item, char *buf) \
} \
CONFIGFS_ATTR(cluster_, name);
+static int dlm_check_protocol_and_dlm_running(unsigned int x)
+{
+ switch (x) {
+ case 0:
+ /* TCP */
+ break;
+ case 1:
+ /* SCTP */
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (dlm_allow_conn)
+ return -EBUSY;
+
+ return 0;
+}
+
+static int dlm_check_zero_and_dlm_running(unsigned int x)
+{
+ if (!x)
+ return -EINVAL;
+
+ if (dlm_allow_conn)
+ return -EBUSY;
+
+ return 0;
+}
+
static int dlm_check_zero(unsigned int x)
{
if (!x)
@@ -180,7 +210,7 @@ static int dlm_check_buffer_size(unsigned int x)
return 0;
}
-CLUSTER_ATTR(tcp_port, dlm_check_zero);
+CLUSTER_ATTR(tcp_port, dlm_check_zero_and_dlm_running);
CLUSTER_ATTR(buffer_size, dlm_check_buffer_size);
CLUSTER_ATTR(rsbtbl_size, dlm_check_zero);
CLUSTER_ATTR(recover_timer, dlm_check_zero);
@@ -188,7 +218,7 @@ CLUSTER_ATTR(toss_secs, dlm_check_zero);
CLUSTER_ATTR(scan_secs, dlm_check_zero);
CLUSTER_ATTR(log_debug, NULL);
CLUSTER_ATTR(log_info, NULL);
-CLUSTER_ATTR(protocol, NULL);
+CLUSTER_ATTR(protocol, dlm_check_protocol_and_dlm_running);
CLUSTER_ATTR(mark, NULL);
CLUSTER_ATTR(timewarn_cs, dlm_check_zero);
CLUSTER_ATTR(waitwarn_us, NULL);
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index d772e1d4461d..d25b9132c593 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -134,7 +134,7 @@ static DEFINE_SPINLOCK(dlm_node_addrs_spin);
static struct listen_connection listen_con;
static struct sockaddr_storage *dlm_local_addr[DLM_MAX_ADDR_COUNT];
static int dlm_local_count;
-static int dlm_allow_conn;
+int dlm_allow_conn;
/* Work queues */
static struct workqueue_struct *recv_workqueue;
diff --git a/fs/dlm/lowcomms.h b/fs/dlm/lowcomms.h
index 0918f9376489..f74888ed43b4 100644
--- a/fs/dlm/lowcomms.h
+++ b/fs/dlm/lowcomms.h
@@ -14,6 +14,9 @@
#define LOWCOMMS_MAX_TX_BUFFER_LEN 4096
+/* switch to check if dlm is running */
+extern int dlm_allow_conn;
+
int dlm_lowcomms_start(void);
void dlm_lowcomms_stop(void);
void dlm_lowcomms_exit(void);
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 06/20] fs: dlm: public header in out utility
This patch allows to use header_out() and header_in() outside of dlm
util functionality.
Signed-off-by: Alexander Aring
---
fs/dlm/util.c | 4 ++--
fs/dlm/util.h | 2 ++
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/fs/dlm/util.c b/fs/dlm/util.c
index cfd0d00b19ae..74a8c5bfe9b5 100644
--- a/fs/dlm/util.c
+++ b/fs/dlm/util.c
@@ -20,7 +20,7 @@
#define DLM_ERRNO_ETIMEDOUT 110
#define DLM_ERRNO_EINPROGRESS 115
-static void header_out(struct dlm_header *hd)
+void header_out(struct dlm_header *hd)
{
hd->h_version = cpu_to_le32(hd->h_version);
hd->h_lockspace = cpu_to_le32(hd->h_lockspace);
@@ -28,7 +28,7 @@ static void header_out(struct dlm_header *hd)
hd->h_length= cpu_to_le16(hd->h_length);
}
-static void header_in(struct dlm_header *hd)
+void header_in(struct dlm_header *hd)
{
hd->h_version = le32_to_cpu(hd->h_version);
hd->h_lockspace = le32_to_cpu(hd->h_lockspace);
diff --git a/fs/dlm/util.h b/fs/dlm/util.h
index cc719ca9397e..d46f23c7a6a0 100644
--- a/fs/dlm/util.h
+++ b/fs/dlm/util.h
@@ -15,6 +15,8 @@ void dlm_message_out(struct dlm_message *ms);
void dlm_message_in(struct dlm_message *ms);
void dlm_rcom_out(struct dlm_rcom *rc);
void dlm_rcom_in(struct dlm_rcom *rc);
+void header_out(struct dlm_header *hd);
+void header_in(struct dlm_header *hd);
#endif
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 03/20] fs: dlm: add errno handling to check callback
This allows to return individual errno values for the config attribute
check callback instead of returning invalid argument only.
Signed-off-by: Alexander Aring
---
fs/dlm/config.c | 23 ---
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/fs/dlm/config.c b/fs/dlm/config.c
index 49c5f9407098..73e6643903af 100644
--- a/fs/dlm/config.c
+++ b/fs/dlm/config.c
@@ -125,7 +125,7 @@ static ssize_t cluster_cluster_name_store(struct
config_item *item,
CONFIGFS_ATTR(cluster_, cluster_name);
static ssize_t cluster_set(struct dlm_cluster *cl, unsigned int *cl_field,
- int *info_field, bool (*check_cb)(unsigned int x),
+ int *info_field, int (*check_cb)(unsigned int x),
const char *buf, size_t len)
{
unsigned int x;
@@ -137,8 +137,11 @@ static ssize_t cluster_set(struct dlm_cluster *cl,
unsigned int *cl_field,
if (rc)
return rc;
- if (check_cb && check_cb(x))
- return -EINVAL;
+ if (check_cb) {
+ rc = check_cb(x);
+ if (rc)
+ return rc;
+ }
*cl_field = x;
*info_field = x;
@@ -161,14 +164,20 @@ static ssize_t cluster_##name##_show(struct config_item
*item, char *buf) \
} \
CONFIGFS_ATTR(cluster_, name);
-static bool dlm_check_zero(unsigned int x)
+static int dlm_check_zero(unsigned int x)
{
- return !x;
+ if (!x)
+ return -EINVAL;
+
+ return 0;
}
-static bool dlm_check_buffer_size(unsigned int x)
+static int dlm_check_buffer_size(unsigned int x)
{
- return (x < DEFAULT_BUFFER_SIZE);
+ if (x < DEFAULT_BUFFER_SIZE)
+ return -EINVAL;
+
+ return 0;
}
CLUSTER_ATTR(tcp_port, dlm_check_zero);
--
2.26.2
[Cluster-devel] [PATCHv3 dlm/next 00/20] fs: dlm: introduce dlm re-transmission layer
Hi, this is the final patch-series to make dlm reliable when re-connection occurs. You can easily generate a couple of re-connections by running: tcpkill -9 -i $IFACE port 21064 on your own to test these patches. At some time dlm will detect message drops and will re-transmit messages if necessary. It introduces a new dlm protocol behaviour and increases the dlm protocol version. I tested it with SCTP as well and tried to be backwards compatible with dlm protocol version 3.1. However I don't recommend at all to mix these versions in a setup since dlm version 3.2 fixes long-term issues. - Alex changes since v3: - make dlm messages to 8 byte boundary size (more pads), because there exists uint64_t fields and we should prepared for future 8 byte fields. This will make it directly aligned to 4 and 2 as well. - change unaligned memory access handling. I will not fix it yet. It seems nobody is using dlm on an architecture which cannot handle unaligned memory access at all (panics). However I added a note that this is a known problem. There is a slightly performance improvement (depends on many things e.g. if another message gets allocated after a (len % 8) != 0 message length got allocated). However I saw that such cases are rarely (for now some user space messages only) occur. The receiving side is not the problem here, the sending side is it and we run in a unaligned memory access in dlm messages fields there as well. However, fixing sending side will fix the receiving side and more length checks can be applied then to drop invalid message lengths. - be sure to remove node from hash at first at close call I am a little bit worried about the midcomms/lowcomms close call and the timer is running at exactly this time and maybe begins to re-transmit messages. I thought about to stop/start the timer but now I ended up to remove the node from the hash at first and be sure that no readers are left when calling lowcomms close. I think this should be fine because we "should" not receive any dlm messages from this node while close is running. - add patch "fs: dlm: add per node receive flush" As I was worried about that the lowcomms close call flushes the receive work on a socket close and we already removed the node from the hash, I added a functionality to flush the receive work right before we remove the node. With this functionality we male sure we don't receive any messages after we removed the node from the hash. - add patch "fs: dlm: remove obsolete code and comment" - add patch "fs: dlm: check for invalid namelen" changes since v2: - add patch "fs: dlm: set connected bit after accept" - add patch "fs: dlm: set subclass for othercon sock_mutex" - change title "fs: dlm: public utils header utils" to "fs: dlm: public header in out utility" - squash "fs: dlm: add check for minimum allocation length" into "fs: dlm: remove unaligned memory access handling" - make the midcomms timeout a little bit longer, because I saw sometimes it's not enough (I hope that was the reason) - midcomms: fix version mismatch handling - remove DLM_ACK in invalid sequence handling - add additional length check in dlm_opts_check_msglen() - use optlen to skip DLM_OPTS header - add DLM_MSGLEN_IS_NOT_ALIGNED to check if msglen is proper aligned before parsing - change dlm_midcomms_close() to close first then cut queues, because lowcomms close will may flush some messages which need to be dropped afterwards if seq doesn't fit. - remove newline in "fs: dlm: add more midcomms hooks" - may more changes which I don't have on track. - change defines handling for calculating max application buffer size vs max allocation size - run aspell on my commit msgs Alexander Aring (20): fs: dlm: set connected bit after accept fs: dlm: set subclass for othercon sock_mutex fs: dlm: add errno handling to check callback fs: dlm: add check if dlm is currently running fs: dlm: change allocation limits fs: dlm: public header in out utility fs: dlm: use GFP_ZERO for page buffer fs: dlm: simplify writequeue handling fs: dlm: add more midcomms hooks fs: dlm: make buffer handling per msg fs: dlm: make new buffer handling softirq ready fs: dlm: add functionality to re-transmit a message fs: dlm: move out some hash functionality fs: dlm: remove unaligned memory access handling fs: dlm: add union in dlm header for lockspace id fs: dlm: add per node receive flush fs: dlm: add reliable connection if reconnect fs: dlm: don't allow half transmitted messages fs: dlm: remove obsolete code and comment fs: dlm: check for invalid namelen fs/dlm/config.c | 60 ++- fs/dlm/dlm_internal.h | 41 +- fs/dlm/lock.c | 16 +- fs/dlm/lockspace.c|5 +- fs/dlm/lowcomms.c | 288 --- fs/dlm/lowcomms.h | 27 +- fs/dlm/member.c | 16 + fs/dlm/member.h |1 +
[Cluster-devel] [PATCHv3 dlm/next 01/20] fs: dlm: set connected bit after accept
This patch sets the CF_CONNECTED bit when dlm accepts a connection from another node. If we don't set this bit, next time if the connection socket gets writable it will assume an event that the connection is successfully connected. However that is only the case when the connection did a connect. Signed-off-by: Alexander Aring --- fs/dlm/lowcomms.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 372c34ff8594..2fd1e4c13663 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -930,6 +930,7 @@ static int accept_from_sock(struct listen_connection *con) addcon = newcon; } + set_bit(CF_CONNECTED, >flags); mutex_unlock(>sock_mutex); /* -- 2.26.2
[Cluster-devel] [PATCHv3 dlm/next 02/20] fs: dlm: set subclass for othercon sock_mutex
This patch sets the lockdep subclass for the othercon socket mutex. In
various places the connection socket mutex is held while locking the
othercon socket mutex. This patch will remove lockdep warnings when such
case occurs.
Signed-off-by: Alexander Aring
---
fs/dlm/lowcomms.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 2fd1e4c13663..d772e1d4461d 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -911,13 +911,14 @@ static int accept_from_sock(struct listen_connection *con)
goto accept_err;
}
+ lockdep_set_subclass(>sock_mutex, 1);
newcon->othercon = othercon;
} else {
/* close other sock con if we have something new */
close_connection(othercon, false, true, false);
}
- mutex_lock_nested(>sock_mutex, 1);
+ mutex_lock(>sock_mutex);
add_sock(newsock, othercon);
addcon = othercon;
mutex_unlock(>sock_mutex);
--
2.26.2
Re: [Cluster-devel] [PATCH] fs: amend SLAB_RECLAIM_ACCOUNT on gfs2_quotad_cachep
- Original Message -
> From: Zhaoyang Huang
>
> As gfs2_quotad_cachep has registered the shrinker, amending
> SLAB_RECLAIM_ACCOUNT when create gfs2_quotad_cachep, which
> make the slab acount to be presiced.
>
> Signed-off-by: Zhaoyang Huang
> ---
> fs/gfs2/main.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/gfs2/main.c b/fs/gfs2/main.c
> index 136484e..db39de9 100644
> --- a/fs/gfs2/main.c
> +++ b/fs/gfs2/main.c
> @@ -136,7 +136,7 @@ static int __init init_gfs2_fs(void)
>
> gfs2_quotad_cachep = kmem_cache_create("gfs2_quotad",
> sizeof(struct gfs2_quota_data),
> -0, 0, NULL);
> +0, SLAB_RECLAIM_ACCOUNT, NULL);
> if (!gfs2_quotad_cachep)
> goto fail_cachep6;
>
> --
> 1.9.1
>
>
Hi,
Thanks for the patch.
We should also do this for gfs2_glock_cachep. Can you add that to your patch?
Regards,
Bob Peterson
Re: [Cluster-devel] [GFS2 PATCH] gfs2: make recovery workqueue operate on a gfs2 mount point, not journal
Hi, - Original Message - > Hi, > > On 22/12/2020 20:38, Bob Peterson wrote: > > Hi, > > > > Before this patch, journal recovery was done by a workqueue function that > > operated on a per-journal basis. The problem is, these could run > > simultaneously > > which meant that they could all use the same bio, sd_log_bio, to do their > > writing to all the various journals. These operations overwrote one another > > eventually causing memory corruption. > > Why not just add more bios so that this issue goes away? It would make > more sense than preventing recovery from running in parallel. In general > recovery should be spread amoung nodes anyway, so the case of having > multiple recoveries running on the same node in parallel should be > fairly rare too, > > Steve. As I understand it, if we allocate a bio from the same bio_set (as bio_alloc does) we need to submit the previous bio before getting the next one, which means recovery processes cannot work in parallel, even if they use different bio pointers. We can, of course, allocate several bio_sets, one for each journal, but I remember Jeff Moyer telling me it would use 1MB per bio_set of memory, which seems high. (I've not verified that.) I'm testing up to 60 mounts times 5 cluster nodes (5 journals) which would add up to 300MB of memory. That's not horrible but I remember we decided not to allocate separate per-mount rb_trees for glock indexing because of the memory needed, and that seems much less by comparison. We could also introduce new locking (and multiple bio pointers) to prevent the bio from being used by multiple recoveries at the same time. I actually tried that on an earlier attempt and immediately ran into deadlock issues, probably because our journal writes also use the same bio. This way is pretty simple and there are fewer recovery processes to worry about when analyzing vmcores. Bob
[Cluster-devel] [PATCH] fs: amend SLAB_RECLAIM_ACCOUNT on gfs2_quotad_cachep
From: Zhaoyang Huang
As gfs2_quotad_cachep has registered the shrinker, amending
SLAB_RECLAIM_ACCOUNT when create gfs2_quotad_cachep, which
make the slab acount to be presiced.
Signed-off-by: Zhaoyang Huang
---
fs/gfs2/main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/gfs2/main.c b/fs/gfs2/main.c
index 136484e..db39de9 100644
--- a/fs/gfs2/main.c
+++ b/fs/gfs2/main.c
@@ -136,7 +136,7 @@ static int __init init_gfs2_fs(void)
gfs2_quotad_cachep = kmem_cache_create("gfs2_quotad",
sizeof(struct gfs2_quota_data),
- 0, 0, NULL);
+ 0, SLAB_RECLAIM_ACCOUNT, NULL);
if (!gfs2_quotad_cachep)
goto fail_cachep6;
--
1.9.1
[Cluster-devel] [PATCH -next] dlm: use DEFINE_MUTEX (and mutex_init() had been too late)
Signed-off-by: Zheng Yongjun
---
fs/dlm/lockspace.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fs/dlm/lockspace.c b/fs/dlm/lockspace.c
index 624617c12250..2b3c32f2d29d 100644
--- a/fs/dlm/lockspace.c
+++ b/fs/dlm/lockspace.c
@@ -26,7 +26,7 @@
#include "ast.h"
static int ls_count;
-static struct mutexls_lock;
+static DEFINE_MUTEX(ls_lock);
static struct list_headlslist;
static spinlock_t lslist_lock;
static struct task_struct *scand_task;
@@ -231,7 +231,6 @@ static const struct kset_uevent_ops dlm_uevent_ops = {
int __init dlm_lockspace_init(void)
{
ls_count = 0;
- mutex_init(_lock);
INIT_LIST_HEAD();
spin_lock_init(_lock);
--
2.22.0
[Cluster-devel] [PATCH -next] dlm: debug_fs: use DEFINE_MUTEX (and mutex_init() had been too late)
Signed-off-by: Zheng Yongjun
---
fs/dlm/debug_fs.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fs/dlm/debug_fs.c b/fs/dlm/debug_fs.c
index d6bbccb0ed15..7a6fa8ac6f50 100644
--- a/fs/dlm/debug_fs.c
+++ b/fs/dlm/debug_fs.c
@@ -20,7 +20,7 @@
#define DLM_DEBUG_BUF_LEN 4096
static char debug_buf[DLM_DEBUG_BUF_LEN];
-static struct mutex debug_buf_lock;
+static DEFINE_MUTEX(debug_buf_lock);
static struct dentry *dlm_root;
@@ -794,7 +794,6 @@ void dlm_create_debug_file(struct dlm_ls *ls)
void __init dlm_register_debugfs(void)
{
- mutex_init(_buf_lock);
dlm_root = debugfs_create_dir("dlm", NULL);
}
--
2.22.0
[Cluster-devel] [PATCH v2 -next] dlm: debug_fs: use DEFINE_MUTEX() for mutex lock
mutex lock can be initialized automatically with DEFINE_MUTEX()
rather than explicitly calling mutex_init()
Signed-off-by: Zheng Yongjun
---
fs/dlm/debug_fs.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fs/dlm/debug_fs.c b/fs/dlm/debug_fs.c
index d6bbccb0ed15..7a6fa8ac6f50 100644
--- a/fs/dlm/debug_fs.c
+++ b/fs/dlm/debug_fs.c
@@ -20,7 +20,7 @@
#define DLM_DEBUG_BUF_LEN 4096
static char debug_buf[DLM_DEBUG_BUF_LEN];
-static struct mutex debug_buf_lock;
+static DEFINE_MUTEX(debug_buf_lock);
static struct dentry *dlm_root;
@@ -794,7 +794,6 @@ void dlm_create_debug_file(struct dlm_ls *ls)
void __init dlm_register_debugfs(void)
{
- mutex_init(_buf_lock);
dlm_root = debugfs_create_dir("dlm", NULL);
}
--
2.22.0
[Cluster-devel] [PATCH v2 -next] dlm: use DEFINE_MUTEX() for mutex lock
mutex lock can be initialized automatically with DEFINE_MUTEX()
rather than explicitly calling mutex_init().
Signed-off-by: Zheng Yongjun
---
fs/dlm/lockspace.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fs/dlm/lockspace.c b/fs/dlm/lockspace.c
index 624617c12250..2b3c32f2d29d 100644
--- a/fs/dlm/lockspace.c
+++ b/fs/dlm/lockspace.c
@@ -26,7 +26,7 @@
#include "ast.h"
static int ls_count;
-static struct mutexls_lock;
+static DEFINE_MUTEX(ls_lock);
static struct list_headlslist;
static spinlock_t lslist_lock;
static struct task_struct *scand_task;
@@ -231,7 +231,6 @@ static const struct kset_uevent_ops dlm_uevent_ops = {
int __init dlm_lockspace_init(void)
{
ls_count = 0;
- mutex_init(_lock);
INIT_LIST_HEAD();
spin_lock_init(_lock);
--
2.22.0
Re: [Cluster-devel] [GFS2 PATCH] gfs2: make recovery workqueue operate on a gfs2 mount point, not journal
Hi,
On 22/12/2020 20:38, Bob Peterson wrote:
Hi,
Before this patch, journal recovery was done by a workqueue function that
operated on a per-journal basis. The problem is, these could run simultaneously
which meant that they could all use the same bio, sd_log_bio, to do their
writing to all the various journals. These operations overwrote one another
eventually causing memory corruption.
Why not just add more bios so that this issue goes away? It would make
more sense than preventing recovery from running in parallel. In general
recovery should be spread amoung nodes anyway, so the case of having
multiple recoveries running on the same node in parallel should be
fairly rare too,
Steve.
This patch makes the recovery workqueue operate on a per-superblock basis,
which means a mount point using, for example journal0, could do recovery
for all journals that need recovery. This is done consecutively so the
sd_log_bio is only referenced by one recovery at a time, thus avoiding the
chaos.
Since the journal recovery requests can come in any order, and unpredictably,
the new work func loops until there are no more journals to be recovered.
Since multiple processes may request recovery of a journal, and since they
all now use the same sdp-based workqueue, it's okay for them to get an
error from queue_work: Queueing work while there's already work queued.
Signed-off-by: Bob Peterson
---
fs/gfs2/incore.h | 2 +-
fs/gfs2/ops_fstype.c | 2 +-
fs/gfs2/recovery.c | 32
3 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index 8e1ab8ed4abc..b393cbf9efeb 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -529,7 +529,6 @@ struct gfs2_jdesc {
struct list_head jd_list;
struct list_head extent_list;
unsigned int nr_extents;
- struct work_struct jd_work;
struct inode *jd_inode;
unsigned long jd_flags;
#define JDF_RECOVERY 1
@@ -746,6 +745,7 @@ struct gfs2_sbd {
struct completion sd_locking_init;
struct completion sd_wdack;
struct delayed_work sd_control_work;
+ struct work_struct sd_recovery_work;
/* Inode Stuff */
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 61fce59cb4d3..3d9a6d6d42cb 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -93,6 +93,7 @@ static struct gfs2_sbd *init_sbd(struct super_block *sb)
init_completion(>sd_locking_init);
init_completion(>sd_wdack);
spin_lock_init(>sd_statfs_spin);
+ INIT_WORK(>sd_recovery_work, gfs2_recover_func);
spin_lock_init(>sd_rindex_spin);
sdp->sd_rindex_tree.rb_node = NULL;
@@ -586,7 +587,6 @@ static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct
gfs2_holder *ji_gh)
INIT_LIST_HEAD(>extent_list);
INIT_LIST_HEAD(>jd_revoke_list);
- INIT_WORK(>jd_work, gfs2_recover_func);
jd->jd_inode = gfs2_lookupi(sdp->sd_jindex, , 1);
if (IS_ERR_OR_NULL(jd->jd_inode)) {
if (!jd->jd_inode)
diff --git a/fs/gfs2/recovery.c b/fs/gfs2/recovery.c
index c26c68ebd29d..cd3e66cdb560 100644
--- a/fs/gfs2/recovery.c
+++ b/fs/gfs2/recovery.c
@@ -399,9 +399,8 @@ static void recover_local_statfs(struct gfs2_jdesc *jd,
return;
}
-void gfs2_recover_func(struct work_struct *work)
+static void gfs2_recover_one(struct gfs2_jdesc *jd)
{
- struct gfs2_jdesc *jd = container_of(work, struct gfs2_jdesc, jd_work);
struct gfs2_inode *ip = GFS2_I(jd->jd_inode);
struct gfs2_sbd *sdp = GFS2_SB(jd->jd_inode);
struct gfs2_log_header_host head;
@@ -562,16 +561,41 @@ void gfs2_recover_func(struct work_struct *work)
wake_up_bit(>jd_flags, JDF_RECOVERY);
}
+void gfs2_recover_func(struct work_struct *work)
+{
+ struct gfs2_sbd *sdp = container_of(work, struct gfs2_sbd,
+ sd_recovery_work);
+ struct gfs2_jdesc *jd;
+ int count, recovered = 0;
+
+ do {
+ count = 0;
+ spin_lock(>sd_jindex_spin);
+ list_for_each_entry(jd, >sd_jindex_list, jd_list) {
+ if (test_bit(JDF_RECOVERY, >jd_flags)) {
+ spin_unlock(>sd_jindex_spin);
+ gfs2_recover_one(jd);
+ spin_lock(>sd_jindex_spin);
+ count++;
+ recovered++;
+ }
+ }
+ spin_unlock(>sd_jindex_spin);
+ } while (count);
+ if (recovered > 1)
+ fs_err(sdp, "Journals recovered: %d\n", recovered);
+}
+
int gfs2_recover_journal(struct gfs2_jdesc *jd, bool wait)
{
+ struct gfs2_sbd *sdp = GFS2_SB(jd->jd_inode);
int rv;
if (test_and_set_bit(JDF_RECOVERY, >jd_flags))
return -EBUSY;
