Nikita Rusetskii created SOLR-18667:
---------------------------------------

             Summary: Managed resources API decodes the path twice: terms with 
'+' or '%' can't be deleted
                 Key: SOLR-18667
                 URL: https://issues.apache.org/jira/browse/SOLR-18667
             Project: Solr
          Issue Type: Bug
          Components: Schema and Analysis
    Affects Versions: main(11.0), 10.1, 8.6.3, 9.6.1
            Reporter: Nikita Rusetskii


This is a follow-up to SOLR-6853. That issue fixed '/' in managed stopwords and 
synonyms (PR #1829, released in 9.4.1). Other special characters still don't 
work. Thomas Corthals reported this in SOLR-6853 after the merge.

*Cause*

{{RestManager.ManagedEndpoint.resolveResourceId}} calls {{URLDecoder.decode}} 
on {{SolrQueryRequest.getPath()}}. The servlet container already decoded this 
path, so Solr decodes it a second time. {{URLDecoder}} is made for HTML form 
data, so it also turns '\+' into a space.

This is old. 8.6.3 still used Restlet, and it also decodes twice. It gives the 
same results for '\+' and '%' as 9.6.1.

*Steps to reproduce*

{code}
bin/solr start --user-managed
bin/solr create -c techproducts -d sample_techproducts_configs
U=http://localhost:8983/solr/techproducts/schema/analysis/stopwords/test
curl -X PUT -H 'Content-type:application/json' --data-binary 
'{"class":"org.apache.solr.rest.schema.analysis.ManagedWordSetResource"}' "$U"
curl -X PUT -H 'Content-type:application/json' --data-binary '["a+b","a 
b","C++","100%"]' "$U"
curl -X DELETE "$U/a%2Bb"
curl "$U"
{code}

Expected: "a\+b" is deleted.
Actual: "a b" is deleted and "a\+b" stays. The list is 
\["100%","C\+\+","a\+b"\].

Two more cases:
* {{curl -X DELETE "$U/C%2B%2B"}} returns 404 "C   not found". Solr looks for 
"C" followed by two spaces.
* {{curl -X DELETE "$U/100%25"}} returns 500 "URLDecoder: Incomplete trailing 
escape (%) pattern" on 9.6.1. On 10.1.0 and main Jetty rejects it with 400 (see 
below). SOLR-13183 reports a 500 for {{/schema/%25}}. On 9.6.1 that request 
gives the same error, and the stack trace goes through {{resolveResourceId}}.

*Results*

I sent GET and DELETE for single terms. The term was encoded once (RFC 3986, 
like {{encodeURIComponent}}) or twice. 9.6.1 and 10.1.0 are the official Docker 
images (Jetty 10 and Jetty 12). main is a {{./gradlew devSlim}} build of commit 
63b6dcc (Jetty 12). 10.1.0 and main gave the same result for every request.

||Term||Encoded once, 9.6.1||Encoded once, 10.1.0 and main||Encoded twice, 
9.6.1||Encoded twice, 10.1.0 and main||
|C\+\+|404|404|works|400|
|a\+b|finds and deletes "a b"|finds and deletes "a b"|works|400|
|100%|500|400|works|400|
|AC/DC|works|400|works|400|
|a:b|404|works|works|400|
|C#, a;b, a?b, a\[b\], a@b, a b, über|works|works|works|400|

* On 9.6.1 encoding twice works for every term. This is the known workaround. 
Solarium has a {{useDoubleEncoding}} option for it.
* On 10.1.0 and main Jetty 12 rejects '%25' and '%2F' in the path with 400 
("Ambiguous URI path encoding", "Ambiguous URI path separator"). Solr doesn't 
see these requests. The workaround is gone there. A single term with '\+' or 
'%' can't be read or deleted. A GET of the whole list still shows it. An 
unencoded '/' still works (SOLR-6853).
* An unencoded '\+' in the path means a space on all these versions and on 
8.6.3: {{GET $U/a\+b}} returns "a b".
* A term with a backslash has the same problem as '%' on 10.1.0 and main: {{GET 
$U/a%5Cb}} returns 400. On 9.6.1 it works.
* A term "." can't be read or deleted on 9.6.1, 10.1.0 and main. Clients like 
{{encodeURIComponent}} don't encode '.'. {{GET $U/.}} returns the whole list, 
and {{DELETE $U/.}} deletes the whole list. I tried this only on a list that no 
field type uses. Encoded as "%2E" the request returns 400 on 9.6.1 and 10.1.0. 
The term ".." can't be read or deleted either.
* The LTR feature and model stores and the language-models store use the same 
{{ManagedEndpoint}} code. So I think names with '\+' or '%' are affected there 
too. I did not test them.

*Proposed fix*

I would like to hear an opinion before I start.

Solr should decode the path only once. The open question is what an unencoded 
'\+' means after that.

I would keep '\+' as a space, like today. {{ManagedEndpoint}} takes the part of 
the raw request URI that starts with "/schema" or "/config" and decodes it once 
with {{URLDecoder}}. Then "%2B" is '\+', and '\+' is still a space. Clients 
that send '\+' for a space keep working. Our own 
{{TestManagedSynonymGraphFilterFactory}} is one of them: with some seeds it 
puts the key "hansestadt hamburg" into the path with {{URLEncoder.encode}}. 
Clients that send "%20" for a space keep working too.

The raw URI is not normalized like the servlet path. The code uses it only when 
it gives the same path as the servlet path, apart from '\+'. In all other cases 
it uses the servlet path as now. This way the resource id is different from the 
path that authorization checks only in spaces that came from '\+'.

Clients that encode twice will break. On 10.x and main Jetty already rejects 
their requests with 400. On 9.x they work today, for example Solarium with 
{{useDoubleEncoding}}. I think the change fits main and 10.x, and not 9.x.

The other option is to remove the {{URLDecoder.decode}} call and use the 
servlet path as is. Then '\+' is '\+', as RFC 3986 says. It is simpler. But 
clients that send '\+' for a space will break, for example for multi-word 
synonyms. I think this option fits only main, with a note in the upgrade notes.

Both options leave some terms out. On Jetty 12 '%', an encoded '/' and a 
backslash can't be in the path. Allowing them in the Jetty URI compliance would 
change it for every path in Solr, so I don't want to do it here. A term "." or 
".." can't be in the path on all three versions. To support any term, the API 
could also take the term as a request parameter, for example {{DELETE 
$U?term=100%25}}. This is a new API. Maybe it fits better into a v2 API for 
managed resources, which doesn't exist yet.

The ref guide page "Managed Resources" doesn't say how to encode a term. I 
would add it with the fix.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to