The branch, master has been updated
via 8bc77a0f86f pylibsmb: Multi-threaded use is now possible with SMB2
via a4e3092bd01 pylibsmb: Remove unused py_cli_state->is_smb1
via 47b773addc2 libsmb: Remove unused sync cli_smb2_list()
via 6baceb4de4f pylibsmb: Remove SMB2 special case for cli_list()
via 9dde2dc99b5 libsmb: Use async cli_smb2_list_send() in
cli_list_send()
via 8101c18362d libsmb: Prepare cli_list_send()/recv() for single-issue
subreqs
via 1f11b7b447b libsmb: Convert cli_list_recv() to single-recv
via d1269ef9796 libsmb: Make cli_smb2_list() asynchronous
from 1d12806d661 uptodateness.py: remove what appears to be debugging
lines
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 8bc77a0f86f32a7fc46cdcdce806e8f130069535
Author: Volker Lendecke <v...@samba.org>
Date: Wed Nov 18 15:00:07 2020 +0100
pylibsmb: Multi-threaded use is now possible with SMB2
No non-async callees are used anymore
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
Autobuild-User(master): Jeremy Allison <j...@samba.org>
Autobuild-Date(master): Thu Nov 19 04:12:11 UTC 2020 on sn-devel-184
commit a4e3092bd01c03ff795c0f875244bf4caaedcfae
Author: Volker Lendecke <v...@samba.org>
Date: Wed Nov 18 14:59:15 2020 +0100
pylibsmb: Remove unused py_cli_state->is_smb1
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit 47b773addc2dafad1198284c536ff788d87053bb
Author: Volker Lendecke <v...@samba.org>
Date: Tue Nov 17 12:31:20 2020 +0100
libsmb: Remove unused sync cli_smb2_list()
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit 6baceb4de4f2ce8c1dc3b564ecd213212e8b3171
Author: Volker Lendecke <v...@samba.org>
Date: Mon Nov 16 08:26:56 2020 +0100
pylibsmb: Remove SMB2 special case for cli_list()
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit 9dde2dc99b5aa8046fbf29ead877e34b235b8134
Author: Volker Lendecke <v...@samba.org>
Date: Mon Nov 16 08:26:09 2020 +0100
libsmb: Use async cli_smb2_list_send() in cli_list_send()
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit 8101c18362d9f32cd575c1cc407e56fcc9cbe44e
Author: Volker Lendecke <v...@samba.org>
Date: Wed Nov 18 14:13:22 2020 +0100
libsmb: Prepare cli_list_send()/recv() for single-issue subreqs
This prepares cli_list_recv() for the lowerlevel NT_STATUS_RETRY that
will come in once cli_list_send() uses cli_smb2_list_send() as well.
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit 1f11b7b447b38ab065640f30a2410fd3e81ae6fe
Author: Volker Lendecke <v...@samba.org>
Date: Mon Oct 26 09:21:17 2020 +0100
libsmb: Convert cli_list_recv() to single-recv
This converts the higher-level cli_list_recv() to the new
cli_smb2_list_recv() calling convention to just issue one entry per
recv() call in preparation of using the async cli_smb2_list_send() in
cli_list_send().
For SMB1 this will be a performance degradation, as we have to make
copies out of the arrays that cli_trans_recv() returns, but soon this
will become a performance improvement for the SMB2 directory
listing. And as hopefully most deployments these days are SMB2, I
think we can live with the SMB1 client directory listing
degradation. Also, we can also convert the lowerlevel SMB1 directory
listing routines in case someone actually sees problems from this
here.
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
commit d1269ef9796391a68b1e762a88d159824d78d3eb
Author: Volker Lendecke <v...@samba.org>
Date: Sat Nov 14 18:31:22 2020 +0100
libsmb: Make cli_smb2_list() asynchronous
Return directory entries as soon as possible via
cli_smb2_list_recv(). This returns just one entry per call to
cli_smb2_list_recv() right out of the buffer without assembling
potentially thousands of entries in a big array. You must call
cli_smb2_recv() until an error (except NT_STATUS_RETRY) happens. This
reduces our latency for smbclient's "dir" command significantly for
large directories. In the future I hope I can do the same thing also for
SMBC_readdir_ctx() to improve all users of our published libsmbclient.
Initial attempts of this routine issued fresh smb2_query_directory
requests asynchronously while the receivers of the entries did their
processing, for example showing them in smbclient's "dir"
command. However, this breaks because for example the "showacls"
smbclient option needs to do synchronous smb requests to do their job,
which we can't do while async requests are pending. Thus I came up
with a semi-synchronous approach to issue additional
smb2_query_directory requests from within cli_smb2_list_recv() and
return NT_STATUS_RETRY. This means that we will call back our caller
via the tevent_req_notify function when a fresh entry is available.
Signed-off-by: Volker Lendecke <v...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
-----------------------------------------------------------------------
Summary of changes:
source3/libsmb/cli_smb2_fnum.c | 390 +++++++++++++++++++++++------------------
source3/libsmb/cli_smb2_fnum.h | 16 +-
source3/libsmb/clilist.c | 195 +++++++++++++++++----
source3/libsmb/proto.h | 6 +-
source3/libsmb/pylibsmb.c | 83 ++++-----
5 files changed, 439 insertions(+), 251 deletions(-)
Changeset truncated at 500 lines:
diff --git a/source3/libsmb/cli_smb2_fnum.c b/source3/libsmb/cli_smb2_fnum.c
index 1313ee629c4..2dd76de967a 100644
--- a/source3/libsmb/cli_smb2_fnum.c
+++ b/source3/libsmb/cli_smb2_fnum.c
@@ -1279,212 +1279,262 @@ static bool windows_parent_dirname(TALLOC_CTX
*mem_ctx,
return true;
}
-/***************************************************************
- Wrapper that allows SMB2 to list a directory.
- Synchronous only.
-***************************************************************/
+struct cli_smb2_list_dir_data {
+ uint8_t *data;
+ uint32_t length;
+};
+
+struct cli_smb2_list_state {
+ struct tevent_context *ev;
+ struct cli_state *cli;
+ const char *mask;
+
+ uint16_t fnum;
-NTSTATUS cli_smb2_list(struct cli_state *cli,
- const char *pathname,
- uint32_t attribute,
- NTSTATUS (*fn)(struct file_info *finfo,
- const char *mask,
- void *private_data),
- void *private_data)
-{
NTSTATUS status;
- uint16_t fnum = 0xffff;
- char *parent_dir = NULL;
- const char *mask = NULL;
- struct smb2_hnd *ph = NULL;
- bool processed_file = false;
- TALLOC_CTX *frame = talloc_stackframe();
- TALLOC_CTX *subframe = NULL;
- bool mask_has_wild;
- uint32_t max_trans;
- uint32_t max_avail_len;
+ struct cli_smb2_list_dir_data *response;
+ uint32_t offset;
+};
+
+static void cli_smb2_list_opened(struct tevent_req *subreq);
+static void cli_smb2_list_done(struct tevent_req *subreq);
+static void cli_smb2_list_closed(struct tevent_req *subreq);
+
+struct tevent_req *cli_smb2_list_send(
+ TALLOC_CTX *mem_ctx,
+ struct tevent_context *ev,
+ struct cli_state *cli,
+ const char *pathname)
+{
+ struct tevent_req *req = NULL, *subreq = NULL;
+ struct cli_smb2_list_state *state = NULL;
+ char *parent = NULL;
bool ok;
- if (smbXcli_conn_has_async_calls(cli->conn)) {
- /*
- * Can't use sync call while an async call is in flight
- */
- status = NT_STATUS_INVALID_PARAMETER;
- goto fail;
+ req = tevent_req_create(mem_ctx, &state, struct cli_smb2_list_state);
+ if (req == NULL) {
+ return NULL;
}
+ state->ev = ev;
+ state->cli = cli;
+ state->status = NT_STATUS_OK;
if (smbXcli_conn_protocol(cli->conn) < PROTOCOL_SMB2_02) {
- status = NT_STATUS_INVALID_PARAMETER;
- goto fail;
+ tevent_req_nterror(req, NT_STATUS_INVALID_PARAMETER);
+ return tevent_req_post(req, ev);
}
- /* Get the directory name. */
- if (!windows_parent_dirname(frame,
- pathname,
- &parent_dir,
- &mask)) {
- status = NT_STATUS_NO_MEMORY;
- goto fail;
- }
-
- mask_has_wild = ms_has_wild(mask);
-
- status = cli_smb2_create_fnum(cli,
- parent_dir,
- 0, /* create_flags */
- SMB2_IMPERSONATION_IMPERSONATION,
- SEC_DIR_LIST|SEC_DIR_READ_ATTRIBUTE,/* desired_access */
- FILE_ATTRIBUTE_DIRECTORY, /* file attributes */
- FILE_SHARE_READ|FILE_SHARE_WRITE, /* share_access */
- FILE_OPEN, /* create_disposition */
- FILE_DIRECTORY_FILE, /* create_options */
- NULL,
- &fnum,
- NULL,
- NULL,
- NULL);
+ ok = windows_parent_dirname(state, pathname, &parent, &state->mask);
+ if (!ok) {
+ tevent_req_oom(req);
+ return tevent_req_post(req, ev);
+ }
- if (!NT_STATUS_IS_OK(status)) {
- goto fail;
+ subreq = cli_smb2_create_fnum_send(
+ state, /* mem_ctx */
+ ev, /* ev */
+ cli, /* cli */
+ parent, /* fname */
+ 0, /* create_flags */
+ SMB2_IMPERSONATION_IMPERSONATION, /* impersonation_level
*/
+ SEC_DIR_LIST|SEC_DIR_READ_ATTRIBUTE, /* desired_access */
+ FILE_ATTRIBUTE_DIRECTORY, /* file_attributes */
+ FILE_SHARE_READ|FILE_SHARE_WRITE, /* share_access */
+ FILE_OPEN, /* create_disposition */
+ FILE_DIRECTORY_FILE, /* create_options */
+ NULL); /* in_cblobs */
+ if (tevent_req_nomem(subreq, req)) {
+ return tevent_req_post(req, ev);
}
+ tevent_req_set_callback(subreq, cli_smb2_list_opened, req);
+ return req;
+}
- status = map_fnum_to_smb2_handle(cli,
- fnum,
- &ph);
- if (!NT_STATUS_IS_OK(status)) {
- goto fail;
+static void cli_smb2_list_opened(struct tevent_req *subreq)
+{
+ struct tevent_req *req = tevent_req_callback_data(
+ subreq, struct tevent_req);
+ struct cli_smb2_list_state *state = tevent_req_data(
+ req, struct cli_smb2_list_state);
+ NTSTATUS status;
+
+ status = cli_smb2_create_fnum_recv(
+ subreq, &state->fnum, NULL, NULL, NULL);
+ TALLOC_FREE(subreq);
+ if (tevent_req_nterror(req, status)) {
+ return;
}
/*
- * ideally, use the max transaction size, but don't send a request
- * bigger than we have credits available for
+ * Make our caller get back to us via cli_smb2_list_recv(),
+ * triggering the smb2_query_directory_send()
*/
- max_trans = smb2cli_conn_max_trans_size(cli->conn);
- ok = smb2cli_conn_req_possible(cli->conn, &max_avail_len);
- if (ok) {
- max_trans = MIN(max_trans, max_avail_len);
- }
-
- do {
- uint8_t *dir_data = NULL;
- uint32_t dir_data_length = 0;
- uint32_t next_offset = 0;
- subframe = talloc_stackframe();
-
- status = smb2cli_query_directory(cli->conn,
- cli->timeout,
- cli->smb2.session,
- cli->smb2.tcon,
- SMB2_FIND_ID_BOTH_DIRECTORY_INFO,
- 0, /* flags */
- 0, /* file_index */
- ph->fid_persistent,
- ph->fid_volatile,
- mask,
- max_trans,
- subframe,
- &dir_data,
- &dir_data_length);
+ tevent_req_defer_callback(req, state->ev);
+ tevent_req_notify_callback(req);
+}
- if (!NT_STATUS_IS_OK(status)) {
- if (NT_STATUS_EQUAL(status, STATUS_NO_MORE_FILES)) {
- break;
- }
- goto fail;
- }
+static void cli_smb2_list_done(struct tevent_req *subreq)
+{
+ struct tevent_req *req = tevent_req_callback_data(
+ subreq, struct tevent_req);
+ struct cli_smb2_list_state *state = tevent_req_data(
+ req, struct cli_smb2_list_state);
+ struct cli_smb2_list_dir_data *response = NULL;
- do {
- struct file_info *finfo = talloc_zero(subframe,
- struct file_info);
+ response = talloc(state, struct cli_smb2_list_dir_data);
+ if (tevent_req_nomem(response, req)) {
+ return;
+ }
- if (finfo == NULL) {
- status = NT_STATUS_NO_MEMORY;
- goto fail;
- }
+ state->status = smb2cli_query_directory_recv(
+ subreq, response, &response->data, &response->length);
+ TALLOC_FREE(subreq);
- status = parse_finfo_id_both_directory_info(dir_data,
- dir_data_length,
- finfo,
- &next_offset);
+ if (NT_STATUS_IS_OK(state->status)) {
+ state->response = response;
+ state->offset = 0;
- if (!NT_STATUS_IS_OK(status)) {
- goto fail;
- }
+ tevent_req_defer_callback(req, state->ev);
+ tevent_req_notify_callback(req);
+ return;
+ }
- /* Protect against server attack. */
- status = is_bad_finfo_name(cli, finfo);
- if (!NT_STATUS_IS_OK(status)) {
- smbXcli_conn_disconnect(cli->conn, status);
- goto fail;
- }
+ TALLOC_FREE(response);
- if (dir_check_ftype(finfo->attr, attribute)) {
- /*
- * Only process if attributes match.
- * SMB1 servers do the filtering, so
- * with SMB2 we need to emulate it in
- * the client.
- *
- *
https://bugzilla.samba.org/show_bug.cgi?id=10260
- */
- processed_file = true;
-
- status = fn(
- finfo,
- pathname,
- private_data);
-
- if (!NT_STATUS_IS_OK(status)) {
- break;
- }
- }
+ subreq = cli_smb2_close_fnum_send(
+ state, state->ev, state->cli, state->fnum);
+ if (tevent_req_nomem(subreq, req)) {
+ return;
+ }
+ tevent_req_set_callback(subreq, cli_smb2_list_closed, req);
+}
- TALLOC_FREE(finfo);
+static void cli_smb2_list_closed(struct tevent_req *subreq)
+{
+ NTSTATUS status = cli_smb2_close_fnum_recv(subreq);
+ tevent_req_simple_finish_ntstatus(subreq, status);
+}
- /* Move to next entry. */
- if (next_offset) {
- dir_data += next_offset;
- dir_data_length -= next_offset;
- }
- } while (next_offset != 0);
-
- TALLOC_FREE(subframe);
-
- if (!mask_has_wild) {
- /*
- * MacOSX 10 doesn't set STATUS_NO_MORE_FILES
- * when handed a non-wildcard path. Do it
- * for the server (with a non-wildcard path
- * there should only ever be one file returned.
- */
- status = STATUS_NO_MORE_FILES;
- break;
+/*
+ * Return the next finfo directory.
+ *
+ * This parses the blob returned from QUERY_DIRECTORY step by step. If
+ * the blob ends, this triggers a fresh QUERY_DIRECTORY and returns
+ * NT_STATUS_RETRY, which will then trigger the caller again when the
+ * QUERY_DIRECTORY has returned with another buffer. This way we
+ * guarantee that no asynchronous request is open after this call
+ * returns an entry, so that other synchronous requests can be issued
+ * on the same connection while the directoy listing proceeds.
+ */
+NTSTATUS cli_smb2_list_recv(
+ struct tevent_req *req,
+ TALLOC_CTX *mem_ctx,
+ struct file_info **pfinfo)
+{
+ struct cli_smb2_list_state *state = tevent_req_data(
+ req, struct cli_smb2_list_state);
+ struct cli_smb2_list_dir_data *response = NULL;
+ struct file_info *finfo = NULL;
+ NTSTATUS status;
+ uint32_t next_offset = 0;
+ bool in_progress;
+
+ in_progress = tevent_req_is_in_progress(req);
+
+ if (!in_progress) {
+ if (!tevent_req_is_nterror(req, &status)) {
+ status = NT_STATUS_NO_MORE_FILES;
+ }
+ goto fail;
+ }
+
+ response = state->response;
+ if (response == NULL) {
+ struct tevent_req *subreq = NULL;
+ struct cli_state *cli = state->cli;
+ struct smb2_hnd *ph = NULL;
+ uint32_t max_trans, max_avail_len;
+ bool ok;
+
+ if (!NT_STATUS_IS_OK(state->status)) {
+ status = state->status;
+ goto fail;
}
- } while (NT_STATUS_IS_OK(status));
+ status = map_fnum_to_smb2_handle(cli, state->fnum, &ph);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto fail;
+ }
- if (NT_STATUS_EQUAL(status, STATUS_NO_MORE_FILES)) {
- status = NT_STATUS_OK;
+ max_trans = smb2cli_conn_max_trans_size(cli->conn);
+ ok = smb2cli_conn_req_possible(cli->conn, &max_avail_len);
+ if (ok) {
+ max_trans = MIN(max_trans, max_avail_len);
+ }
+
+ subreq = smb2cli_query_directory_send(
+ state, /* mem_ctx */
+ state->ev, /* ev */
+ cli->conn, /* conn */
+ cli->timeout, /* timeout_msec */
+ cli->smb2.session, /* session */
+ cli->smb2.tcon, /* tcon */
+ SMB2_FIND_ID_BOTH_DIRECTORY_INFO, /* level */
+ 0, /* flags */
+ 0, /* file_index */
+ ph->fid_persistent, /* fid_persistent */
+ ph->fid_volatile, /* fid_volatile */
+ state->mask, /* mask */
+ max_trans); /* outbuf_len */
+ if (subreq == NULL) {
+ status = NT_STATUS_NO_MEMORY;
+ goto fail;
+ }
+ tevent_req_set_callback(subreq, cli_smb2_list_done, req);
+ return NT_STATUS_RETRY;
}
- if (NT_STATUS_IS_OK(status) && !processed_file) {
- /*
- * In SMB1 findfirst returns NT_STATUS_NO_SUCH_FILE
- * if no files match. Emulate this in the client.
- */
- status = NT_STATUS_NO_SUCH_FILE;
+ SMB_ASSERT(response->length > state->offset);
+
+ finfo = talloc_zero(mem_ctx, struct file_info);
+ if (finfo == NULL) {
+ status = NT_STATUS_NO_MEMORY;
+ goto fail;
}
- fail:
+ status = parse_finfo_id_both_directory_info(
+ response->data + state->offset,
+ response->length - state->offset,
+ finfo,
+ &next_offset);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto fail;
+ }
- if (fnum != 0xffff) {
- cli_smb2_close_fnum(cli, fnum);
+ status = is_bad_finfo_name(state->cli, finfo);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto fail;
}
- cli->raw_status = status;
+ /*
+ * parse_finfo_id_both_directory_info() checks for overflow,
+ * no need to check again here.
+ */
+ state->offset += next_offset;
- TALLOC_FREE(subframe);
- TALLOC_FREE(frame);
+ if (next_offset == 0) {
+ TALLOC_FREE(state->response);
+ }
+
+ tevent_req_defer_callback(req, state->ev);
+ tevent_req_notify_callback(req);
+
+ *pfinfo = finfo;
+ return NT_STATUS_OK;
+
+fail:
+ TALLOC_FREE(finfo);
+ tevent_req_received(req);
return status;
}
diff --git a/source3/libsmb/cli_smb2_fnum.h b/source3/libsmb/cli_smb2_fnum.h
index c28d55be9e9..91781b6a11b 100644
--- a/source3/libsmb/cli_smb2_fnum.h
+++ b/source3/libsmb/cli_smb2_fnum.h
@@ -93,13 +93,15 @@ struct tevent_req *cli_smb2_unlink_send(
const char *fname,
const struct smb2_create_blobs *in_cblobs);
NTSTATUS cli_smb2_unlink_recv(struct tevent_req *req);
-NTSTATUS cli_smb2_list(struct cli_state *cli,
- const char *pathname,
- uint32_t attribute,
- NTSTATUS (*fn)(struct file_info *finfo,
- const char *mask,
- void *private_data),
- void *private_data);
+struct tevent_req *cli_smb2_list_send(
+ TALLOC_CTX *mem_ctx,
+ struct tevent_context *ev,
+ struct cli_state *cli,
+ const char *pathname);
+NTSTATUS cli_smb2_list_recv(
+ struct tevent_req *req,
+ TALLOC_CTX *mem_ctx,
+ struct file_info **pfinfo);
NTSTATUS cli_smb2_qpathinfo_basic(struct cli_state *cli,
const char *name,
SMB_STRUCT_STAT *sbuf,
diff --git a/source3/libsmb/clilist.c b/source3/libsmb/clilist.c
index d0abd476d74..25040438d9e 100644
--- a/source3/libsmb/clilist.c
+++ b/source3/libsmb/clilist.c
@@ -970,9 +970,12 @@ NTSTATUS cli_list_trans(struct cli_state *cli, const char
*mask,
}
struct cli_list_state {
+ struct tevent_context *ev;
+ struct tevent_req *subreq;
NTSTATUS (*recv_fn)(struct tevent_req *req, TALLOC_CTX *mem_ctx,
struct file_info **finfo);
struct file_info *finfo;
+ size_t num_received;
};
static void cli_list_done(struct tevent_req *subreq);
@@ -984,26 +987,32 @@ struct tevent_req *cli_list_send(TALLOC_CTX *mem_ctx,
uint32_t attribute,
uint16_t info_level)
{
- struct tevent_req *req, *subreq;
+ struct tevent_req *req = NULL;
struct cli_list_state *state;
+ enum protocol_types proto = smbXcli_conn_protocol(cli->conn);
req = tevent_req_create(mem_ctx, &state, struct cli_list_state);
if (req == NULL) {
return NULL;
}
+ state->ev = ev;
- if (smbXcli_conn_protocol(cli->conn) <= PROTOCOL_LANMAN1) {
- subreq = cli_list_old_send(state, ev, cli, mask, attribute);
- state->recv_fn = cli_list_old_recv;
- } else {
--
Samba Shared Repository
