This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch branch_9_0
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/branch_9_0 by this push:
new 6a5f62b5287 Upgrade notes and major changes cleanup and update (#851)
6a5f62b5287 is described below
commit 6a5f62b5287464398944491941eb237cccf35ebb
Author: Anshum Gupta <[email protected]>
AuthorDate: Wed May 11 15:36:39 2022 +0200
Upgrade notes and major changes cleanup and update (#851)
* Move 'other' section to bottom
* Minor fix to existing content
* Re-phrase 'local' vs 'internal' metrics (draft)
Co-authored-by: Jan Høydahl <[email protected]>
---
.../pages/major-changes-in-solr-9.adoc | 440 ++++++---------------
.../upgrade-notes/pages/solr-upgrade-notes.adoc | 13 +-
2 files changed, 122 insertions(+), 331 deletions(-)
diff --git
a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
index be1cb46afa2..b3430b78dd9 100644
---
a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
+++
b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
@@ -18,7 +18,7 @@
Solr 9.0 is a major new release of Solr.
-This page highlights the biggest changes, including new features you may want
to be aware of, and changes in default behavior and deprecated features that
have been removed.
+This page highlights the most important changes including new features and
changes in default behavior as well as previously deprecated features that have
now been removed.
== Solr 9 Upgrade Planning
@@ -30,15 +30,15 @@ A thorough review of the list in
xref:major-changes-in-earlier-8-x-versions[Majo
=== Upgrade Prerequisites
-*Solr 9 requires Java 11 as minimum Java version, and is also tested with Java
17.*
+*Solr 9 requires Java 11 as minimum Java version and is also tested with Java
17.*
-If using Solr's Java client (*SolrJ*) with `CloudSolrClient` to connect to
your SolrCloud cluster, make sure to upgrade SolrJ in all your client
applications to version 8.10 or higher (8.x), before upgrading your SolrCloud
cluster to version 9.0. Otherwise, SolrJ will not be able to connect to the
cluster once it has upgraded to Solr 9. Once you have upgraded all Solr
clusters that the client is connecting to, you can upgrade the SolrJ client to
9.x.
+If using `CloudSolrClient` to connect to your SolrCloud cluster, SolrJ must be
upgraded in all your client applications to version 8.10 or higher (8.x),
before upgrading your SolrCloud cluster to version 9.0. Otherwise, SolrJ will
not be able to connect to the cluster once it has upgraded to Solr 9.
-If you have an old collection that was initially created with a Solr version
prior to 5.0, that collection Solr may keep that collection's state in the
`/clusterstate.json` file at the root of Zookeeper. Solr 9 does no longer
support this file. You must upgrade such collections to the new per-collection
`state.json` format *before* upgrading to Solr 9. This is done by calling the
https://solr.apache.org/guide/8_11/cluster-node-management.html#migratestateformat[Collection
API MIGRATESTAT [...]
+If you have an old collection that was initially created with a Solr version
prior to 5.0, Solr may keep that collection's cluster state in the
`/clusterstate.json` file at the root of Zookeeper. Solr 9 no longer supports
this file. You must upgrade such collections to the new per-collection
`state.json` format *before* upgrading to Solr 9. This is done by calling the
https://solr.apache.org/guide/8_11/cluster-node-management.html#migratestateformat[Collection
API MIGRATESTATEFORMAT] act [...]
-If you are using Solr in standalone mode with the
xref:query-guide:query-elevation-component.adoc[] with it's elevation file in
the data directory, you'll have to move it to the
xref:configuration-guide:config-sets.adoc[Configset] folder instead.
+If you are using Solr in standalone mode with the
xref:query-guide:query-elevation-component.adoc[] with its elevation file in
the data directory, you'll have to move it to the
xref:configuration-guide:config-sets.adoc[configset] folder instead.
-Some metrics have changed names due to the removal of "master" and "slave"
terminology in 8.x. If you rely on metrics, alerts and monitors on Solr KPIs
that mention "master" or "slave", you will need to prepare for those metrics
now appearing with "leader" and "follower" in their names instead.
+If you rely on metrics, alerts, or monitors on Solr KPIs that use the "master"
or "slave" terminology, please update your system for those metrics to now show
up with "leader" and "follower" terminology.
=== Rolling Upgrades
@@ -61,333 +61,125 @@ A rolling upgrade from Solr 8 to Solr 9 requires the
following multiple restart
It is always strongly recommended that you fully reindex your documents after
a major version upgrade. For details, see the
xref:indexing-guide:reindexing.adoc[] section, which covers several strategies
for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via
`UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was
removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed
and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and
HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and
BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be
compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin
parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and
`SOLR_IP_BLACKLIST` are no longer supported, but replaced with
`SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements
PermissionNameProvider. Any custom or 3rd party request handler must also do
this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by
default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See
Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return
information.
-In previous versions there was a field returned in async backup status
responses, `Response`. This has now been renamed to `msg`, to better fit other
collections API responses.
-The `response` field is now a map, containing information about the backup
(`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional
fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental
backup request responses.
-Also, both snapshot and incremental backup request responses will now contain
`starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount`
and `endTime`, snapshot delete shard's response were updated to add fields
`startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of
backup and delete shard's responses are now deprecated and will be removed in
future releases.
-All date/time fields of backup and delete related shard's responses have been
updated to use `Instance` instead of `Date`, meaning the output will be in the
standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map
to return information instead of a list.
-This is only applicable for users returning information in JSON format, which
is the default behavior.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map
to represent each tag instead of an array.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr
configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section
xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change
the repository class to
`org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the
xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup
Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth
module. Existing Solr configurations do not need any Hadoop authentication
related
-changes, however the module needs to be installed - see the section
xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr
configurations do not need any SQL related
-changes, however the module needs to be installed - see the section
xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming
expression, is no longer created automatically on startup. The user must create
this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the
module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be
loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics
now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g.
"/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be
such except
-"[shard]". The default Prometheus exporter config splits that component to a
new label
-named "internal". The sample Grafana dashboard now filters to include or
exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms"
property carries per field arrays by default regardless of distrib, terms.list,
terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in
SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the
local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text,
commas between key-value pairs will now be followed by a space (e.g.,
{shape=square, color=yellow} rather than {shape=square,color=yellow}) for
consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute
standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as
StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value
irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module
irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of
"/cluster/command-status/1000" instead of
"/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to
8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an
"allowlist" to better convey the meaning of the property
-
-* SOLR-12891: MacroExpander will no longer will expand URL parameters inside
of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when
constructing streaming expressions that include user supplied data to avoid
risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with
-DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws
MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be
possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code
overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now
receives an extra boolean parameter representing the "exactness" of the
`numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to
implement this abstract method now (instead previous with the same name but
without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe
looks nicer.
-
-* SOLR-9376: The response format for field values serialized as raw XML (via
the `[xml]` raw value DocTransformer
+In Solr 8, it was possible to add docValues to a schema without re-indexing
via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* xref:query-guide:dense-vector-search.adoc[Dense Vector "Neural" Search]
through `DenseVectorField` fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and
Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud
instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the
standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations
do not need any SQL related changes, however the module needs to be installed -
see the section xref:query-guide:sql-query.adoc[].
+* JSON aggregations uses corrected sample formula to compute standard
deviation and variance. The computation of stdDev and variance in JSON
aggregation is same as StatsComponent.
+* Facet count in Json Facet module always returns a `long` value, irrespective
of number of shards.
+* `MacroExpander` will no longer will expand URL parameters inside of the
`expr` parameter (used by streaming expressions).
+Additionally, users are advised to use the `InjectionDefense` class when
constructing streaming expressions that include user supplied data to avoid
risks similar to SQL injection. The legacy behavior of expanding the `expr`
parameter can be reinstated with `-DStreamingExpressionMacros=true` passed to
the JVM at startup
+* The response format for field values serialized as raw XML (via the `[xml]`
raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as
top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure.
As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of
each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued
fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
+* Highlighting: `hl.method=unified` is the new default. Use
`hl.method=original`
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade
from prior versions of Solr may find that some requests involving complex
internal query structures (Example: long query strings using `edismax` with
many `qf` and `pf` fields that include query time synonym expansion) which
worked in the past now hit this limit and fail. Users in this situation are
advised to consider the complexity of their queries/configuration, and increase
the value of xref:configuration-g [...]
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field
to clearly show the document isn't a root document. Solr 8 would fallback on
the `\_route_` param but no longer.
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is
not found, Solr will instead use the default one from
`$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by
setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property
`-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in
`$SOLR_HOME` if started with embedded zookeeper.
-
-* SOLR-12901: Highlighting: hl.method=unified is the new default. Use
hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where
log messages may be lost in the event of some hard crash.
+=== Security
+* New xref:deployment-guide:cert-authentication-plugin.adoc[Certificate
Authentication Plugin], enabling end-to-end use of x509 client certificates for
Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions. Users may have to adapt
their `security.json`.
+* Ability to disable admin UI through a system property.
+* The property `blockUnknown` in the `BasicAuthPlugin` and the `JWTAuthPlugin`
now defaults to `true` instead of `false`. This change is backward
incompatible. If you need the pre-9.0 default behavior, you need to explicitly
set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop
users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of
the box security.
+Administrators that need Solr exposed more broadly can change the
`SOLR_JETTY_HOST` property in their Solr include (`solr.in.sh`/`solr.in.cmd`)
file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded
zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the
`clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default,
with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in version
8.x. See xref:deployment-guide:configuring-logging.adoc[Configuring Logging]
for how to change this.
+* Hadoop authentication support has been moved to the new `hadoop-auth`
module. Existing Solr configurations do not need any Hadoop authentication
related changes, however the module needs to be installed - see the section
xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* xref:deployment-guide:jwt-authentication-plugin.adoc[JWTAuthPlugin] has been
moved to a module. Users need to add the module `jwt-auth` to classpath. The
plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be
loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates removes several security
issues from dependencies, and thus make Solr more secure.
+* The allow-list defining allowed URLs for the `shards` parameter is not in
the `shardHandler` configuration anymore. It is defined by the `allowUrls`
top-level property of the `solr.xml` file. For more information, see
xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of
solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been
renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting`
package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to
`modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* xref:deployment-guide:rate-limiters.adoc[Rate limiting] provides a way to
throttle update and search requests based on usage metrics.
+* A new xref:deployment-guide:task-management.adoc[Task management] interface
allows declaring tasks as cancellable and trackable.
+* Ability to specify xref:deployment-guide:node-roles.adoc[node roles] in
Solr. This release supports `overseer` and `data` roles out of the box.
+* New API for pluggable
xref:configuration-guide:replica-placement-plugins.adoc[Replica Placement
Plugins] that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection
API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache
projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability
to create functionally identical local image.
+
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own
`log4j2.xml` and no longer shares Solr’s logging config.
+* Only `SearchHandler` and subclasses have "local" metrics now. It's now
tracked with a new `internal=true` tag, which can be filtered in e.g.
Prometheus.
+There are no longer ".distrib." named metrics; all metrics are assumed to be
such except "[shard]". The default Prometheus exporter config splits that
component to a new label named "internal". The sample Grafana dashboard now
filters to include or exclude this.
+* The default port of "Prometheus exporter" has changed from 9983 to 8989, so
you may need to adjust your configuration after upgrade.
+* Logging is now asynchronous by default. There's a small window where log
messages may be lost in the event of some hard crash.
Switch back to synchronous logging if this is unacceptable, see comments in
the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how
the `solr.xml` global
xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`]
option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of
clauses in any `BooleanQuery` object - but it was only enforced for the
_direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total
number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests
involving complex internal query structures (Example: long query strings using
`edismax` with many `qf` and `pf` fields that include query time synonym
expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their
queries/configuration, and increase the value of
xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`]
if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr
sets for use by Logging calls (such as the collection name, shard name, replica
name, etc...) have been modified to now be "bare" values, with out the special
single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would
have an MDC value with a key `collection` mapped to a value of
`c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend
these same prefixes to MDC values when included in Log messages as part of the
`<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs
are consistently formatted after upgrading will need to make similar changes to
their logging configuration files. See
link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more
details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications,
then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to
better align the stored state in Zookeeper with future versions of Solr; as of
Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client
applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will
continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr
10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special
`samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to
sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus
trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment
variables as documented by Jaeger.
-
-=== Schema Changes
-
+* Log4J configuration & Solr MDC values -
link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets
for use by Logging calls (such as the collection name, shard name, replica
name, etc...) have been modified to now be "bare" values, without the special
single character prefixes that were included in past version. The default
`log4j2.xml` configuration file for Solr has been modified to prepend these
same prefixes to MDC values when included in Log messages as part o [...]
+* xref:deployment-guide:configuring-logging.adoc#request-logging[Jetty Request
log] is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be
found under `solr/prometheus-exporter/`.
+* Solr modules (formerly known as contribs) can now easily be enabled by an
environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system
property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
+
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer
a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been
removed. If your collections use `clusterstate.json` you will need to take some
steps, described elsewhere in this document.
+* Auto-scaling framework has been removed. Please refer to
xref:configuration-guide:replica-placement-plugins.adoc[Replica Placement
Plugins] for alternate options.
* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by
their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now
defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set
`blockUnknown:false` in `security.json`.
-
-* The allow-list defining allowed URLs for the `shards` parameter is not in
the `shardHandler` configuration anymore. It is defined by the `allowUrls`
top-level property of the `solr.xml` file. For more information, see
xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of
solr.allowUrls] documentation.
-
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default
for better out of the box security.
-Administrators that need Solr exposed more broadly can change the
SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default.
Administrators that need to run Solr with Hadoop will need to disable this
feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in
one of the Solr init scripts. Other features in Solr could also break. (Robert
Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the
clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have
been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the
module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new
location.
-More information can be found in the
xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs
documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to
`modules/scripting` package instead of shipping as part of Solr, due to
security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-*
jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in
`lucene-libs/` under module directories in the binary release.
-Instead, these libraries will be included with all other module dependencies
in `lib/`.
-
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module.
It can be found under `solr/prometheus-exporter/`.
-
-* SOLR-15914: Solr modules (formerly known as contribs) can now easily be
enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or
as a system property (e.g. in `SOLR_OPTS`). Example:
`SOLR_MODULES=extraction,ltr`.
-
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at:
https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for
collections created with `stateFormat=1` as well as support for Collection API
MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false
now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at
https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache,
FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache
instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use
the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when
configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was
broken and inconsistent with other replica placement strategies.
+* Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have
been removed. Users have to modify their existing configurations to use
CaffeineCache instead.
+* `VelocityResponseWriter` is an independent project now; it is no longer a
part of Solr. This encompasses all previously included `/browse` and
`wt=velocity` examples.
+* Cross Data Center Replication has been removed.
+* SolrJ clients like `HttpSolrClient` and `LBHttpSolrClient` that lacked HTTP2
support have been deprecated. The old CloudSolrClient has been renamed as
CloudLegacySolrClient and deprecated.
+* SimpleFSDirectoryFactory is removed in favor of NIOFSDirectoryFactory
+* Removed the deprecated `HttpSolrClient.RemoteSolrException` and
`HttpSolrClient.RemoteExecutionException`. All the usages are replaced by
`BaseHttpSolrClient.RemoteSolrException` and
`BaseHttpSolrClient.RemoteExecutionException`.
+* `maxShardsPerNode` parameter has been removed because it was broken and
inconsistent with other replica placement strategies.
Other relevant placement strategies should be used instead, such as
autoscaling policy or rules-based placement.
+* The binary distribution no longer contains test-framework jars.
+* Deprecated BlockJoinFacetComponent and BlockJoinDocSetFacetComponent are
removed. Users are encouraged to migrate to uniqueBlock() in JSON Facet API.
+* Core level admin API endpoints `/admin/threads`, `/admin/properties`,
`/admin/logging` are now only available at the node level.
-* SOLR-14092: Deprecated BlockJoinFacetComponent and
BlockJoinDocSetFacetComponent are removed.
-Users are encouraged to migrate to uniqueBlock() in JSON Facet API. (Mikhail
Khludnev)
-
-* SOLR-13596: Deprecated GroupingSpecification methods are removed.
-
-* SOLR-11266: default Content-Type override for JSONResponseWriter from
`_default` configSet is removed.
-Example has been provided in `sample_techproducts_configs` to override
content-type.
-
-* `min_rf` deprecated in 7.x
-
-* hl.method=postings highlighter, deprecated in 7.0
-
-* SOLR-15124: Removed three core level admin API endpoints because they are
already registered at the node level
-where they really belong: /admin/threads, /admin/properties, /admin/logging
-
-* SOLR-15949: Docker: the official image now uses Java 17 provided by Eclipse
Temurin. Formerly it was Java 11 from OpenJDK.
-(janhoy, David Smiley)
-
-* Atomic/partial updates to nested documents now _require_ the `\_root_` field
to clearly show the document isn't a root document. Solr 8 would fallback on
the `\_route_` param but no longer.
-
-* SolrJ: The project is migrating away from the Apache HttpClient dependency
in favor of Jetty's client (supporting HTTP/2), to occur over the 9x releases.
We deprecated HttpSolrClient and friends in favor of Http2SolrClient and
equivalents. For SolrCloud, the former CloudSolrClient was renamed
CloudLegacySolrClient (deprecated), and instead BaseCloudSolrClient was renamed
to CloudSolrClient and given a Builder for the Jetty based HTTP/2 client.
-
-* SOLR-13323: The unused package org.apache.solr.internal.csv.writer and
associated classes/tests that were easily confused with but not used by
org.apache.solr.response.CSVWriter (or any other code) have been removed
-
-=== Solr Binary Release `dist/` Directory
-
-The `dist/` directory in the Solr binary release has been removed as of Solr
9.0.
-
-* The `solr-core` and `solr-solrj` jars can be found under
`server/solr-webapp/webapp/WEB-INF/lib/`.
-* The Solr module jars can be found in `modules/<module-name>/lib`, packaged
individually for each module.
-* The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the
other Server jars.
-Please refer to the SolrJ Maven artifact to see the exact dependencies you
need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and
`server/lib/ext/` if you are loading in SolrJ manually.
+=== Other
+* xref:configuration-guide:solr-modules.adoc[Contrib modules] are now just
"modules". You can easily enable module(s) through environment variable
`SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL,
Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your
`<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under
`server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in
`modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the
other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you
need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and
`server/lib/ext/` if you are loading in SolrJ manually.
If you plan on using SolrJ as a JDBC driver, please refer to the
xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the
xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs
documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See
`CloudLegacySolrClient` for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return
information instead of a list. This is only applicable for users returning
information in JSON format, which is the default behavior.
+* `SolrMetricProducer` / `SolrInfoBean` APIs have changed and third-party
components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed.
JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter
will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and
`SOLR_IP_BLACKLIST` are no longer supported, but replaced with
`SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and
return information. For collection's snapshot backup request responses
additional fields `indexVersion`, `indexFileCount`, etc. were added similar to
incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the
repository class to
`org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the
xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup
Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr
configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section
xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression,
is no longer created automatically on startup. The user must create this folder.
+* Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found,
Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`.
You can revert to the pre-9.0 behaviour by setting environment variable
`SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`.
Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded
zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to
upgrade SolrJ to 8.8.x for all of your client applications, then you can set
`-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the
stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the
`base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client
applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will
continue to store the `base_url` in Zookeeper. For background, see:
link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and
link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for
the `solr.storeBaseUrl` system property will be removed in Solr 10.x and
`base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the
field type configuration.
+* The `solr-extraction` module has been cleaned up to produce
`solr-extraction-*` jar instead of `solr-cell-*` jars.
+* Extra lucene libraries used in modules are no longer packaged in
`lucene-libs/` under module directories in the binary release.
+Instead, these libraries will be included with all other module dependencies
in `lib/`.
[#major-changes-in-earlier-8-x-versions]
== Major Changes in Earlier 8.x Versions
@@ -612,8 +404,8 @@ The default Prometheus Exporter configuration includes
metrics like queries-per-
*Solr Home*
-* The internal logic for identifying 'Solr Home' (`solr.solr.home`) has been
refactored to make testing less error prone.
-Plugin developers using `SolrPaths.locateSolrHome()` or 'new
`SolrResourceLoader`' should check deprecation warnings as existing some
existing functionality will be removed in 9.0.
+* The internal logic for identifying 'Solr Home' (`solr.solr.home`) has been
refactored to make testing less error-prone.
+Plugin developers using `SolrPaths.locateSolrHome()` or `new
SolrResourceLoader` should check deprecation warnings as existing some existing
functionality will be removed in 9.0.
https://issues.apache.org/jira/browse/SOLR-14934[SOLR-14934] has more
technical details about this change for those concerned.
*base_url removed from stored state*
diff --git
a/solr/solr-ref-guide/modules/upgrade-notes/pages/solr-upgrade-notes.adoc
b/solr/solr-ref-guide/modules/upgrade-notes/pages/solr-upgrade-notes.adoc
index 2b71cee817b..1a562844628 100644
--- a/solr/solr-ref-guide/modules/upgrade-notes/pages/solr-upgrade-notes.adoc
+++ b/solr/solr-ref-guide/modules/upgrade-notes/pages/solr-upgrade-notes.adoc
@@ -23,11 +23,10 @@
The following notes describe changes to Solr in recent releases that you
should be aware of before upgrading.
-These notes highlight the biggest changes that may impact the largest number of
-implementations.
+These notes highlight the important changes that may impact the largest number
of implementations.
It is not a comprehensive list of all changes to Solr in any release.
-When planning your Solr upgrade, consider the customizations you have made to
+When planning your Solr upgrade, consider the customizations to
your system and review the {solr-javadocs}/changes/Changes.html[`CHANGES.txt`]
file found in your Solr package.
That file includes all the changes and updates
@@ -39,19 +38,19 @@ Detailed steps for upgrading a Solr cluster are in the
section xref:deployment-g
The upgrade from 8.x to Solr 9.0 introduces several *major* changes that you
should be aware of before upgrading.
These changes are described in the section xref:major-changes-in-solr-9.adoc[].
-It's strongly recommended that you do a thorough review of that section before
starting your upgrade.
+It's strongly recommended that you thoroughly review that section before
starting your upgrade.
[NOTE]
-If you run in SolrCloud mode and use SolrJ's `CloudSolrClient`, you must first
upgrade SolrJ to version 8.10 or higher before upgrading the cluster to 9.0.
+If you run in SolrCloud mode and use SolrJ's `CloudSolrClient`, you must first
upgrade SolrJ to a version 8.10 or higher before upgrading the cluster to 9.0.
[NOTE]
Rolling upgrades from Solr 8 to Solr 9 require first upgrading the cluster to
version 8.7 or newer.
== Upgrading from Pre-8.x Versions
-Users upgrading from versions of Solr prior to 8.x are strongly encouraged to
consult {solr-javadocs}/changes/Changes.html[`CHANGES.txt`] for the details of
_all_ changes since the version they are upgrading from.
+Users upgrading from versions prior to 8.x are strongly encouraged to consult
{solr-javadocs}/changes/Changes.html[`CHANGES.txt`] for the details of _all_
changes since the version they are upgrading from.
-The upgrade from Solr 7.x to Solr 8.0 introduced several *major* changes that
you should be aware of before upgrading.
+The upgrade from 7.x to 8.0 introduced several *major* changes that you should
be aware of before upgrading.
Please do a thorough review of the section xref:major-changes-in-solr-8.adoc[]
before starting your upgrade.
A summary of the significant changes between Solr 6.x and Solr 7.0 is in the
section xref:major-changes-in-solr-7.adoc[].
