The branch, v4-17-test has been updated
via 947790f8d76 mdssvc: fix returning file modification date for older
Mac releases
via c7ee8854d20 mdssvc: fix date marshalling
via a6fba5581db mdssvc: prepare for returning timestamps with
sub-seconds granularity
via e7eb2286214 mdssvc: reduce pagesize to 50
via 31a6264f8c5 tests/mdssvc: match hits:total:value to be the actual
amount of entries in hits
via 57d8d00799c mdssvc: fix enforcement of "elasticsearch:max results"
via 506e9a2d725 mdssvc: add and use SL_PAGESIZE
via 19f10cb9d7d mdssvc: fix long running backend queries
via 9d97cd01641 mdssvc: set query state for continued queries to
SLQ_STATE_RUNNING
from cdce89e434e smbd: don't leak the fsp if close_file_smb() fails
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-17-test
- Log -----------------------------------------------------------------
commit 947790f8d76d3aab518116d8af1a55cecf2f67ae
Author: Ralph Boehme <s...@samba.org>
Date: Thu May 18 18:12:19 2023 +0200
mdssvc: fix returning file modification date for older Mac releases
Mac 10.10 uses kMDItemContentModificationDate instead of
kMDItemFSContentChangeDate.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
Autobuild-User(master): Jeremy Allison <j...@samba.org>
Autobuild-Date(master): Wed Jul 26 23:42:44 UTC 2023 on atb-devel-224
(cherry picked from commit c2e83ebe726b7bc42b329198214c784936f19888)
Autobuild-User(v4-17-test): Jule Anger <jan...@samba.org>
Autobuild-Date(v4-17-test): Mon Jul 31 09:11:27 UTC 2023 on sn-devel-184
commit c7ee8854d20d68a350a8a2bae2fd915d5211884d
Author: Ralph Boehme <s...@samba.org>
Date: Wed May 17 16:38:39 2023 +0200
mdssvc: fix date marshalling
Did this ever work? Possible just copied over from Netatalk and was always
broken... The Mac client expects the timevalue as seconds relative to
2001-01-01 00:00:00 UTC, packed as IEEE float.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 620ca1e68d02be45a94aa41217a141d211fceb1f)
commit a6fba5581dbe68c199cb1f043e11e8a36d66d992
Author: Ralph Boehme <s...@samba.org>
Date: Wed May 17 16:37:36 2023 +0200
mdssvc: prepare for returning timestamps with sub-seconds granularity
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 9dc66fecf7c1743d264c5c4f8978b77bab75ed86)
commit e7eb22862145cd25cd372a6fd438b1ed596b714e
Author: Ralph Boehme <s...@samba.org>
Date: Thu Apr 20 17:27:20 2023 +0200
mdssvc: reduce pagesize to 50
Lastest macOS queries additional file metadata per search result, which
causes
the mashalled paged result set including metadata to exceed the 64 KB result
fragment buffer.
Lacking fragementation support in mdssvc (it's supported by the protocol),
for
now just reduce the maximum number of results per search page.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 724a0518c901589fe1171d94648391832e056f4d)
commit 31a6264f8c597d3a5c264f8bf162a9d9aeffbd0a
Author: Ralph Boehme <s...@samba.org>
Date: Fri Apr 21 07:07:13 2023 +0200
tests/mdssvc: match hits:total:value to be the actual amount of entries in
hits
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 7f5e4edf64f7e4175f652bf8762d4edc110ad6b1)
commit 57d8d00799c882f4a2007443e6a3930500a3ffe6
Author: Ralph Boehme <s...@samba.org>
Date: Thu Apr 20 17:58:38 2023 +0200
mdssvc: fix enforcement of "elasticsearch:max results"
This wasn't enforced at all thus a query would return all available matches
without limit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit d8fa5c8e2a1794ea8dc663485315ebd9401b2628)
commit 506e9a2d725b519f8f69d79504333e9293eb3557
Author: Ralph Boehme <s...@samba.org>
Date: Thu Apr 20 17:24:30 2023 +0200
mdssvc: add and use SL_PAGESIZE
SL_PAGESIZE is the number of entries we want to process per paged search
result
set. This is different from MAX_SL_RESULTS which ought to be a default
maximum
value for total number of results returned for a search query.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 086c2602d074d4dc0d44f5534857e5f59a8690b2)
commit 19f10cb9d7d757a996eaa23a82c9e89ca9006bc5
Author: Ralph Boehme <s...@samba.org>
Date: Thu Mar 23 16:39:11 2023 +0100
mdssvc: fix long running backend queries
If a query is still running in the backend and we have no results yet,
returning
0 triggers a search termination by the client in latest macOS releases.
macOS
returns 0x23 in this case.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 925fefae20e52a3c89a56bdd0cd5b98cc951db5f)
commit 9d97cd01641949e34b34149b0a9141d0e8391b93
Author: Ralph Boehme <s...@samba.org>
Date: Wed Apr 19 14:38:45 2023 +0200
mdssvc: set query state for continued queries to SLQ_STATE_RUNNING
SLQ_STATE_RESULTS implies that there are already results attached to the slq
which is not the case. Instead the backend will start processing from where
it
left off when it hits the maximum result limit and had set the state to
SLQ_STATE_FULL.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <s...@samba.org>
Reviewed-by: Jeremy Allison <j...@samba.org>
(cherry picked from commit 4149ef97e5906604be1587622f390f121db183e2)
-----------------------------------------------------------------------
Summary of changes:
python/samba/tests/blackbox/mdsearch.py | 2 +-
python/samba/tests/dcerpc/mdssvc.py | 6 +++---
source3/rpc_server/mdssvc/marshalling.c | 29 +++++++++++++++++++++++------
source3/rpc_server/mdssvc/mdssvc.c | 21 +++++++++++++++++----
source3/rpc_server/mdssvc/mdssvc.h | 1 +
source3/rpc_server/mdssvc/mdssvc_es.c | 8 ++++----
6 files changed, 49 insertions(+), 18 deletions(-)
Changeset truncated at 500 lines:
diff --git a/python/samba/tests/blackbox/mdsearch.py
b/python/samba/tests/blackbox/mdsearch.py
index c8e75661f15..46af6f94624 100644
--- a/python/samba/tests/blackbox/mdsearch.py
+++ b/python/samba/tests/blackbox/mdsearch.py
@@ -100,7 +100,7 @@ class MdfindBlackboxTests(BlackboxTestCase):
config = os.environ["SMB_CONF_PATH"]
json_in = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
diff --git a/python/samba/tests/dcerpc/mdssvc.py
b/python/samba/tests/dcerpc/mdssvc.py
index 5002e5d26d6..7d09a555936 100644
--- a/python/samba/tests/dcerpc/mdssvc.py
+++ b/python/samba/tests/dcerpc/mdssvc.py
@@ -125,7 +125,7 @@ class MdssvcTests(RpcInterfaceTestCase):
def test_mdscli_search(self):
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
@@ -157,7 +157,7 @@ class MdssvcTests(RpcInterfaceTestCase):
r'kMDItemFSName=="x\\x"'
)
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(file.filename:x\\+x OR file.filename:x\\*x OR
file.filename:x=x OR file.filename:x'x OR file.filename:x\\?x OR
file.filename:x\\ x OR file.filename:x\\(x OR file.filename:x\\\"x OR
file.filename:x\\\\x) AND path.real.fulltext:\"%BASEPATH%\""
@@ -166,7 +166,7 @@ class MdssvcTests(RpcInterfaceTestCase):
}'''
fake_json_response = r'''{
"hits" : {
- "total" : {"value" : 2},
+ "total" : {"value" : 9},
"hits" : [
{"_source" : {"path" : {"real" : "%BASEPATH%/x+x"}}},
{"_source" : {"path" : {"real" : "%BASEPATH%/x*x"}}},
diff --git a/source3/rpc_server/mdssvc/marshalling.c
b/source3/rpc_server/mdssvc/marshalling.c
index 5f866d7fb6e..b3e16d9c18f 100644
--- a/source3/rpc_server/mdssvc/marshalling.c
+++ b/source3/rpc_server/mdssvc/marshalling.c
@@ -43,8 +43,8 @@
* RPC data marshalling and unmarshalling
******************************************************************************/
-/* Spotlight epoch is UNIX epoch minus SPOTLIGHT_TIME_DELTA */
-#define SPOTLIGHT_TIME_DELTA 280878921600ULL
+/* Spotlight epoch is 1.1.2001 00:00 UTC */
+#define SPOTLIGHT_TIME_DELTA 978307200 /* Diff from UNIX epoch to Spotlight
epoch */
#define SQ_TYPE_NULL 0x0000
#define SQ_TYPE_COMPLEX 0x0200
@@ -253,6 +253,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf,
ssize_t offset, size_t bufsi
{
uint64_t data;
uint64_t tag;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
tag = sl_pack_tag(SQ_TYPE_DATE, 2, 1);
offset = sl_push_uint64_val(buf, offset, bufsize, tag);
@@ -260,7 +264,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf,
ssize_t offset, size_t bufsi
return -1;
}
- data = (t.tv_sec + SPOTLIGHT_TIME_DELTA) << 24;
+ ieee_fp_union.d = (double)(t.tv_sec - SPOTLIGHT_TIME_DELTA);
+ ieee_fp_union.d += (double)t.tv_usec / 1000000;
+
+ data = ieee_fp_union.w;
offset = sl_push_uint64_val(buf, offset, bufsize, data);
if (offset == -1) {
return -1;
@@ -723,6 +730,11 @@ static int sl_unpack_date(DALLOC_CTX *query,
int i, result;
struct sl_tag tag;
uint64_t query_data64;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
+ double fraction;
sl_time_t t;
offset = sl_unpack_tag(buf, offset, bufsize, encoding, &tag);
@@ -735,9 +747,14 @@ static int sl_unpack_date(DALLOC_CTX *query,
if (offset == -1) {
return -1;
}
- query_data64 = query_data64 >> 24;
- t.tv_sec = query_data64 - SPOTLIGHT_TIME_DELTA;
- t.tv_usec = 0;
+ ieee_fp_union.w = query_data64;
+ fraction = ieee_fp_union.d - (uint64_t)ieee_fp_union.d;
+
+ t = (sl_time_t) {
+ .tv_sec = ieee_fp_union.d + SPOTLIGHT_TIME_DELTA,
+ .tv_usec = fraction * 1000000
+ };
+
result = dalloc_add_copy(query, &t, sl_time_t);
if (result != 0) {
return -1;
diff --git a/source3/rpc_server/mdssvc/mdssvc.c
b/source3/rpc_server/mdssvc/mdssvc.c
index 22cc7abd153..a7d5d7afc02 100644
--- a/source3/rpc_server/mdssvc/mdssvc.c
+++ b/source3/rpc_server/mdssvc/mdssvc.c
@@ -188,8 +188,10 @@ static bool add_filemeta(struct mds_ctx *mds_ctx,
if (result != 0) {
return false;
}
- } else if (strcmp(attribute, "kMDItemFSContentChangeDate") ==
0) {
- sl_time.tv_sec = sp->st_ex_mtime.tv_sec;
+ } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0
||
+ strcmp(attribute, "kMDItemContentModificationDate") ==
0)
+ {
+ sl_time = convert_timespec_to_timeval(sp->st_ex_mtime);
result = dalloc_add_copy(meta, &sl_time, sl_time_t);
if (result != 0) {
return false;
@@ -306,10 +308,21 @@ static bool create_result_handle(struct sl_query *slq)
static bool add_results(sl_array_t *array, struct sl_query *slq)
{
sl_filemeta_t *fm;
- uint64_t status = 0;
+ uint64_t status;
int result;
bool ok;
+ /*
+ * Taken from a network trace against a macOS SMB Spotlight server. If
+ * the first fetch-query-results has no results yet because the search
+ * is still running, macOS returns 0x23, otherwise 0x0.
+ */
+ if (slq->state >= SLQ_STATE_RESULTS ) {
+ status = 0;
+ } else {
+ status = 0x23;
+ }
+
/* FileMeta */
fm = dalloc_zero(array, sl_filemeta_t);
if (fm == NULL) {
@@ -1126,7 +1139,7 @@ static bool slrpc_fetch_query_results(struct mds_ctx
*mds_ctx,
goto error;
}
if (slq->state == SLQ_STATE_FULL) {
- slq->state = SLQ_STATE_RESULTS;
+ slq->state = SLQ_STATE_RUNNING;
slq->mds_ctx->backend->search_cont(slq);
}
break;
diff --git a/source3/rpc_server/mdssvc/mdssvc.h
b/source3/rpc_server/mdssvc/mdssvc.h
index 3b2ce250f1f..8aabf5b86e5 100644
--- a/source3/rpc_server/mdssvc/mdssvc.h
+++ b/source3/rpc_server/mdssvc/mdssvc.h
@@ -36,6 +36,7 @@
#define MAX_SL_FRAGMENT_SIZE 0xFFFFF
#define MAX_SL_RESULTS 100
+#define SL_PAGESIZE 50
#define MAX_SL_RUNTIME 30
#define MDS_TRACKER_ASYNC_TIMEOUT_MS 250
diff --git a/source3/rpc_server/mdssvc/mdssvc_es.c
b/source3/rpc_server/mdssvc/mdssvc_es.c
index dafb42610fa..8460b48b80a 100644
--- a/source3/rpc_server/mdssvc/mdssvc_es.c
+++ b/source3/rpc_server/mdssvc/mdssvc_es.c
@@ -398,7 +398,7 @@ static bool mds_es_search(struct sl_query *slq)
.ev = mds_es_ctx->mdssvc_es_ctx->mdssvc_ctx->ev_ctx,
.mds_es_ctx = mds_es_ctx,
.slq = slq,
- .size = MAX_SL_RESULTS,
+ .size = SL_PAGESIZE,
};
/* 0 would mean no limit */
@@ -502,7 +502,7 @@ static void mds_es_search_done(struct tevent_req *subreq)
goto trigger;
}
- if (slq->query_results->num_results >= MAX_SL_RESULTS) {
+ if (slq->query_results->num_results >= SL_PAGESIZE) {
slq->state = SLQ_STATE_FULL;
goto trigger;
}
@@ -693,7 +693,7 @@ static void mds_es_search_http_send_done(struct tevent_req
*subreq)
subreq = http_read_response_send(state,
state->ev,
state->s->mds_es_ctx->http_conn,
- MAX_SL_RESULTS * 8192);
+ SL_PAGESIZE * 8192);
if (tevent_req_nomem(subreq, req)) {
return;
}
@@ -800,7 +800,7 @@ static void mds_es_search_http_read_done(struct tevent_req
*subreq)
}
DBG_DEBUG("Hits: %zu\n", hits);
- for (i = 0; i < hits; i++) {
+ for (i = 0; i < hits && s->from + i < s->max; i++) {
const char *path = NULL;
match = json_array_get(matches, i);
--
Samba Shared Repository
