Copilot commented on code in PR #12609:
URL: https://github.com/apache/trafficserver/pull/12609#discussion_r2729624688
##########
doc/admin-guide/files/index.en.rst:
##########
@@ -80,13 +81,16 @@ Configuration Files
Configures SNI based Layer 4 routing.
:doc:`storage.config.en`
- Configures all storage devices and paths to be used for the |TS| cache.
+ Configures all storage devices and paths to be used for the |TS| cache.
(Deprecated in favor of :file:`storage.yaml`)
+
+:doc:`storage.yaml.en`
+ Configures all storage devices and paths to be used for the |TS| cache and
defines cache space usage by individual protocols.
:doc:`strategies.yaml.en`
Configures NextHop strategies used with `remap.config` and replaces
parent.config.
:doc:`volume.config.en`
- Defines cache space usage by individual protocols.
+ Defines cache space usage by individual protocols. (Deprecated
:file:`storage.yaml`)
Review Comment:
The volume.config entry description reads "(Deprecated
:file:`storage.yaml`)" which is grammatically unclear and could be misread as
deprecating ``storage.yaml``; this should say something like "(Deprecated in
favor of :file:`storage.yaml`)" to match the wording used for storage.config
above.
```suggestion
Defines cache space usage by individual protocols. (Deprecated in favor
of :file:`storage.yaml`)
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
+assignment table consistent by maintaining the mapping from physical device to
base string even in the presence of hardware changes and failures.
+
+Backwards Compatibility
+-----------------------
+
+In previous versions of |TS| it was possible to have "exclusive" spans which
were used by only one volume. This is
+now down by specifying the span in the volume and using a size of "100%". E.g.
old configuration like ::
+
+ /dev/disk2 volume=3 # storage.config
+ volume=3 scheme=http size=512 # volume.config
+
+The corresponding configuration would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 100%
+
+Because volume sizes that are percentages are computed on span storage not
already explicitly allocated, this will
+leave none of "disk.2" for such allocation and therefore "disk.2" will be used
only by volume "1". Note this
+configuration is more flexible. If it was useful to have two linear volumes,
each using exclusively half of the
+span, this would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 50%
+ - id: 2
+ spans:
+ - use: disk.2
+ size: 50%
+
+.. important::
+
+ If a span is explicitly used by any volume its storage will be allocated to
only volumes that explicitly use that span.
+
+Examples
+========
+
+The following basic example shows 128 MB of cache storage in the "/big_dir"
directory
+
+.. code-block: yaml
Review Comment:
Several of the later example blocks use the directive syntax ``..
code-block: yaml`` (single colon), e.g. on this line; for Sphinx these should
be ``.. code-block:: yaml`` (double colon) like the earlier example at the top
of the file, otherwise they will not render as code blocks.
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
+assignment table consistent by maintaining the mapping from physical device to
base string even in the presence of hardware changes and failures.
+
+Backwards Compatibility
+-----------------------
+
+In previous versions of |TS| it was possible to have "exclusive" spans which
were used by only one volume. This is
+now down by specifying the span in the volume and using a size of "100%". E.g.
old configuration like ::
+
+ /dev/disk2 volume=3 # storage.config
+ volume=3 scheme=http size=512 # volume.config
+
+The corresponding configuration would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 100%
+
+Because volume sizes that are percentages are computed on span storage not
already explicitly allocated, this will
+leave none of "disk.2" for such allocation and therefore "disk.2" will be used
only by volume "1". Note this
+configuration is more flexible. If it was useful to have two linear volumes,
each using exclusively half of the
+span, this would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 50%
+ - id: 2
+ spans:
+ - use: disk.2
+ size: 50%
+
+.. important::
+
+ If a span is explicitly used by any volume its storage will be allocated to
only volumes that explicitly use that span.
+
+Examples
+========
+
+The following basic example shows 128 MB of cache storage in the "/big_dir"
directory
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: /big_dir
+ size: 134217728
+
+By default a volume uses all spans, therefore a volume uses all of span
"store" because there are no other
+volumes. It would be equivalent is using the spans explicitly, e.g.
Review Comment:
In the example description, the phrase "It would be equivalent is using the
spans explicitly" is ungrammatical; it should be "equivalent to using the spans
explicitly".
```suggestion
volumes. It would be equivalent to using the spans explicitly, e.g.
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
Review Comment:
In the introductory YAML snippet, the comment for ``volumes[].size`` says
"optional, size in percentage", but the table below explicitly documents this
field as accepting either an absolute byte value or a percentage; please update
this comment to match the actual semantics so readers do not think only
percentages are allowed.
```suggestion
size: # optional, size in bytes or percentage
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
+assignment table consistent by maintaining the mapping from physical device to
base string even in the presence of hardware changes and failures.
+
+Backwards Compatibility
+-----------------------
+
+In previous versions of |TS| it was possible to have "exclusive" spans which
were used by only one volume. This is
+now down by specifying the span in the volume and using a size of "100%". E.g.
old configuration like ::
+
+ /dev/disk2 volume=3 # storage.config
+ volume=3 scheme=http size=512 # volume.config
+
+The corresponding configuration would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 100%
+
+Because volume sizes that are percentages are computed on span storage not
already explicitly allocated, this will
+leave none of "disk.2" for such allocation and therefore "disk.2" will be used
only by volume "1". Note this
+configuration is more flexible. If it was useful to have two linear volumes,
each using exclusively half of the
+span, this would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 50%
+ - id: 2
+ spans:
+ - use: disk.2
+ size: 50%
+
+.. important::
+
+ If a span is explicitly used by any volume its storage will be allocated to
only volumes that explicitly use that span.
+
+Examples
+========
+
+The following basic example shows 128 MB of cache storage in the "/big_dir"
directory
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: /big_dir
+ size: 134217728
+
+By default a volume uses all spans, therefore a volume uses all of span
"store" because there are no other
+volumes. It would be equivalent is using the spans explicitly, e.g.
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: /big_dir
+ size: 134217728
+ volumes:
+ - id: 1
+ size: 100%
+ spans:
+ - id: store
+
+You can use the ``.`` symbol for the current directory. Here is an example for
128 MB of cache storage in the current directory
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: "."
+ size: 134217728
+
+.. note::
+ When using on-filesystem cache disk storage, you can only have one such
+ directory specified. This will be addressed in a future version.
+
+Linux Example
+-------------
+.. note::
+
+ Rather than refer to disk devices like ``/dev/sda``, ``/dev/sdb``, etc.,
+ modern Linux supports `alternative symlinked names for disk devices
+
<https://wiki.archlinux.org/index.php/persistent_block_device_naming#by-id_and_by-path>`_
in the ``/dev/disk``
+ directory structure. As noted for the :ref:`storage-assignment-table` the
path used for the disk can effect
+ the cache if it changes. This can be ameliorated in some cases by using one
of the alternate paths
+ in via ``/dev/disk``. Note that if the ``by-id`` or ``by-path`` style is
used, replacing a failed drive will cause
+ that path to change because the new drive will have a different physical ID
or path.
+
+ If this is not sufficient then the :arg:`hash_seed` key should be used to
create a more permanent
+ assignment table. An example would be
+
+ .. code-block: yaml
+
+ cache:
+ spans:
+ - name: "span.0"
+ path: "/dev/sde"
+ hash_seed: "cache.disk.0"
+ - name: "span.1"
+ path: "/dev/sdg"
+ hash_seed: "cache.disk.1"
+
+The following example will use an entire raw disk in the Linux operating
+system
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: a
+ path: "/dev/disk/by-id/disk-A-id"
+ - name: b
+ path: "/dev/disk/by-id/disk-B-id"
+ volumes:
+ - id: 1
+ spans:
+ - use: a
+ size: 100%
+ - id: 2
+ spans:
+ - use: b
+ size: 100%
+
+In order to make sure :program:`traffic_server` will have access to this disk
+you can use :manpage:`udev(7)` to persistently set the right permissions. The
+following rules are targeted for an Ubuntu system, and stored in
+``/etc/udev/rules.d/51-cache-disk.rules``::
+
+ # Assign DiskA and DiskB to the tserver group
+ # make the assignment final, no later changes allowed to the group!
+ SUBSYSTEM=="block", KERNEL=="sd[ef]", GROUP:="tserver"
+
+In order to apply these settings, trigger a reload with
:manpage:`udevadm(8)`:::
+
+ udevadm trigger --subsystem-match=block
+
+
+FreeBSD Example
+---------------
+
+Starting with 5.1 FreeBSD dropped support for explicit raw devices. All
+devices on FreeBSD can be accessed raw now.
+
+The following example will use an entire raw disk in the FreeBSD
+operating system
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: ada.1
+ path: "/dev/ada1"
+ - name: ada.2
+ path: "/dev/ada2"
+ volumes:
+ - id: 1
+ size: 100%
+
+In order to make sure :program:`traffic_server` will have access to this disk
+you can use :manpage:`devfs(8)` to persistently set the right permissions. The
+following rules are stored in :manpage:`devfs.conf(5)`::
+
+ # Assign /dev/ada1 and /dev/ada2 to the tserver user
+ own ada[12] tserver:tserver
+
+Advanced
+--------
+
+Because relative paths in :file:`storage.yaml` are relative to the base
prefix, when using customized runroot
+it may be necessary to adjust such paths in :file:`storage.yaml` or adjust
``runroot.yaml`` itself.
+Despite the name, the cachedir value is not used for this file.
+
+Examples
+========
+
+The following example partitions the cache across 5 volumes to decreasing
single-lock pressure for a
+machine with few drives. The last volume being an example of one that might be
composed of purely
+ramdisks so that the ram cache has been disabled.
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk
+ path: "/dev/sdb"
+ volumes:
+ - id: 1
+ size: 20%
+ - id: 2
+ size: 20%
+ - id: 3
+ size: 20%
+ - id: 4
+ size: 20%
+ - id: 5
+ size: 20%
+ ram_cache: false
+
+This can be simplified by depending on the default allocation which splits
unallocated span storage across volumes.
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk
+ path: "/dev/sdb"
+ volumes:
+ - id: 1
+ - id: 2
+ - id: 3
+ - id: 4
+ - id: 5
+ ram_cache: false
+
+For a host with a physical disk and two ram disks, where the ram disks should
be split between two volumes, with a third
+volume that uses the physical disk.
+
+This depends on defaults. The spans "ram.1" and "ram.2" are split evenly
between volume "1" and volume "2" because no
+sizes are specified. Span "disk" is not used for volume "1" nor volume "2"
because it is not listed in the ``spans``.
+Volume "3" therefore gets all of span "disk".
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk
+ path: "/dev/sdb"
+ - name: ram.1
+ path: "/dev/ram.1"
+ - name: ram.2
+ path: "/dev/ram.2"
+ volumes:
+ - id: 1
+ spans:
+ - use: ram.1
+ - use: ram.2
+ - id: 2
+ spans:
+ - use: ram.1
+ - use: ram.2
+ - id: 3
+
+If one of the ram disk based volumes should be larger, this could be done as
follows by making volume "1" roughly twice
+as large as volume "2".
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk
+ path: "/dev/sdb"
+ - name: ram.1
+ path: "/dev/ram.1"
+ - name: ram.2
+ path: "/dev/ram.2"
+ volumes:
+ - id: 1
+ spans:
+ - use: ram.1
+ size: 66%
+ - use: ram.2
+ size: 66%
+ - id: 2
+ spans:
+ - use: ram.1
+ - use: ram.2
+ - id: 3
+
+Instead, suppose the physical spans ("disk.1" and "disk.2") should be split
across volumes. This can be done by adding volumes
+with only defaults, as the phisycal spans will be divided evenly among four
volumes (3 - 6), each volume allocated 25% of
Review Comment:
In the final example description, "phisycal spans" is misspelled and should
be "physical spans".
```suggestion
with only defaults, as the physical spans will be divided evenly among four
volumes (3 - 6), each volume allocated 25% of
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
Review Comment:
In the Assignment Table section, the paragraph referring to "The :arg:`id`
option" appears to be copied from storage.config and still mentions an ``id``
option that does not exist in storage.yaml, while the corresponding
functionality is now provided by the ``hash_seed`` key; this should be updated
so the text refers to the correct key name.
```suggestion
The :arg:`hash_seed` option can be used to create a fixed string that an
administrator can use to keep the
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
+assignment table consistent by maintaining the mapping from physical device to
base string even in the presence of hardware changes and failures.
+
+Backwards Compatibility
+-----------------------
+
+In previous versions of |TS| it was possible to have "exclusive" spans which
were used by only one volume. This is
+now down by specifying the span in the volume and using a size of "100%". E.g.
old configuration like ::
+
+ /dev/disk2 volume=3 # storage.config
+ volume=3 scheme=http size=512 # volume.config
+
+The corresponding configuration would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 100%
+
+Because volume sizes that are percentages are computed on span storage not
already explicitly allocated, this will
+leave none of "disk.2" for such allocation and therefore "disk.2" will be used
only by volume "1". Note this
+configuration is more flexible. If it was useful to have two linear volumes,
each using exclusively half of the
+span, this would be
+
+.. code-block:: yaml
+
+ cache:
+ spans:
+ - name: disk.2
+ path: /dev/disk2
+ volumes:
+ - id: 1
+ spans:
+ - use: disk.2
+ size: 50%
+ - id: 2
+ spans:
+ - use: disk.2
+ size: 50%
+
+.. important::
+
+ If a span is explicitly used by any volume its storage will be allocated to
only volumes that explicitly use that span.
+
+Examples
+========
+
+The following basic example shows 128 MB of cache storage in the "/big_dir"
directory
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: /big_dir
+ size: 134217728
+
+By default a volume uses all spans, therefore a volume uses all of span
"store" because there are no other
+volumes. It would be equivalent is using the spans explicitly, e.g.
+
+.. code-block: yaml
+
+ cache:
+ spans:
+ - name: store
+ path: /big_dir
+ size: 134217728
+ volumes:
+ - id: 1
+ size: 100%
+ spans:
+ - id: store
Review Comment:
In this YAML example, the span reference under ``spans:`` uses ``- id:
store``, but earlier in the document the keys for ``volumes:spans`` are defined
as ``use`` and ``size``; this example should use ``use: store`` (or otherwise
match the documented schema) to avoid confusing readers.
```suggestion
- use: store
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
Review Comment:
Typo in the important note: "Any change to this files can ..." should read
"Any change to this file can ..." to be grammatically correct.
```suggestion
Any change to this file can (and almost always will) invalidate the
existing cache in its entirety.
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
Review Comment:
In the Assignment Table description, the sentence "This can be problem when
drives fail" is missing an article; it should be "This can be a problem when
drives fail".
```suggestion
which will effectively clear most of the cache. This can be a problem when
drives fail and a system
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
+assignment table maps from an object URL to a specific stripe. The table is
initialized based on a
+pseudo-random process which is seeded by hashing a string for each stripe.
This string is composed
+of a base string, an offset (the start of the stripe on the storage element),
and the length of the
+stripe. By default the path for the storage is used as the base string. This
ensures that each
+stripe has a unique string for the assignment hash. This does make the
assignment table very
+sensitive to the path for the storage elements and changing even one can have
a cascading effect
+which will effectively clear most of the cache. This can be problem when
drives fail and a system
+reboot causes the path names to change.
+
+The :arg:`id` option can be used to create a fixed string that an
administrator can use to keep the
+assignment table consistent by maintaining the mapping from physical device to
base string even in the presence of hardware changes and failures.
+
+Backwards Compatibility
+-----------------------
+
+In previous versions of |TS| it was possible to have "exclusive" spans which
were used by only one volume. This is
+now down by specifying the span in the volume and using a size of "100%". E.g.
old configuration like ::
Review Comment:
In the Backwards Compatibility section, "This is now down by specifying the
span" appears to be a typo and should read "This is now done by specifying the
span".
```suggestion
now done by specifying the span in the volume and using a size of "100%".
E.g. old configuration like ::
```
##########
doc/admin-guide/files/storage.yaml.en.rst:
##########
@@ -0,0 +1,494 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+==============
+storage.yaml
+==============
+
+.. configfile:: storage.yaml
+
+The :file:`storage.yaml` file (by default, located in
+``/usr/local/etc/trafficserver/``) lists all the files, directories, and/or
+hard disk partitions that make up the Traffic Server cache. After you
+modify the :file:`storage.yaml` file the new settings will not be effective
until Traffic Server is restarted.
+
+Format
+======
+
+The format of the :file:`storage.yaml` file is a series of lines of the form
+
+.. code-block:: yaml
+
+ cache: # file level key
+ spans: #
+ - name: # name of the span
+ path: # path to storage
+ size: # size in bytes, required for file system storage,
optional for raw device
+ hash_seed: # optional, used to isolate lookup from path changes
+ volumes: # optional
+ - id: # identifier [1-255]
+ size: # optional, size in percentage
+ scheme: # optional, default to "http"
+ ram_cache: # optional, default to "true"
+ spans: # optional
+ - use: # Span identifier
+ size: # size allocated to this volume
+
+:code:`spans` lists the raw storage used for the cache. :code:`volumes`
organizes the storage into locations for
+storing cached objects. This is very similar to operating system partitions
and file systems.
+
+For :code:`spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| name | string | Name of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+| path | string | File system of the storage. This must be a
block device or |
+| | | directory.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Size in bytes. This is optional for devices
but required |
+| | | for directories.
|
++---------------+-------------+-------------------------------------------------------------+
+| hash_seed | string | Hashing for object location uses a seed to
randomize the |
+| | | hash. By default this is the path for the
span. |
++---------------+-------------+-------------------------------------------------------------+
+
+For :code:`volumes` the keys are
+
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=========================================================================================================+
+| id | integer | Id of the volume. Range is [1-255]. This id
can be referred |
+| | | from :file:`hosting.config`
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| size | bytes | Target size of the entire volume. This can be
an absolute |
+| | _or_ | number of bytes or a percentage.
|
+| | percentage |
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| scheme | enumeration | Protocol scheme, defaults to "http". Preserved
for future |
+| | string | use.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| ram_cache | boolean | Control of ram caching for this volume.
Default is ``true``. This may be desirable if you are using |
+| | | something like ramdisks, to avoid wasting RAM
and cpu time on double caching objects. |
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| avg_obj_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size` configuration for this
volume. |
+| | | This is useful if you have a volume that is
dedicated for say very small objects, and you need a lot of |
+| | | directory entries to store them.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| fragment_size | integer | Overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size` configuration for this volume.
|
+| | | This allows for a smaller, or larger, fragment
size for a particular volume. This may be useful |
+| | | together with ``avg_obj_size`` as well, since
a larger fragment size could reduce the number of |
+| | | directory entries needed for a large object.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+| spans | list | Spans that provide storage for this volume.
Defaults to |
+| | | all spans.
|
++---------------+-------------+---------------------------------------------------------------------------------------------------------+
+
+For :code:`volumes:spans` the keys are
+
++---------------+-------------+-------------------------------------------------------------+
+| Key | Type | Meaning
|
++===============+=============+=============================================================+
+| use | string | Name of the span to use.
|
++---------------+-------------+-------------------------------------------------------------+
+| size | bytes | Amount of the span to use. The total across
all uses of |
+| | _or_ | this specific span must be less than 100% and
less than the |
+| | percentage | total size of the span.
|
++---------------+-------------+-------------------------------------------------------------+
+
+.. important::
+
+ Any change to this files can (and almost always will) invalidate the
existing cache in its entirety.
+
+You can use any partition of any size. For best performance:
+
+- Use raw disk partitions.
+- For each disk, make all partitions the same size.
+- Group similar kinds of storage into different volumes. For example
+ split out SSD's or RAM drives into their own volume.
+
+Specify pathnames according to your operating system requirements. See
+the following examples. In the :file:`storage.yaml` file, a formatted or
+raw disk must be at least 128 MB.
+
+When using raw disk or partitions, you should make sure the :ts:cv:`Traffic
+Server user <proxy.config.admin.user_id>` used by the Traffic Server process
+has read and write privileges on the raw disk device or partition. One good
+practice is to make sure the device file is set with 'g+rw' and the Traffic
+Server user is in the group which owns the device file. However, some
+operating systems have stronger requirements - see the following examples for
+more information.
+
+As with standard ``records.yaml`` integers, human readable prefixes are also
+supported. They include
+
+ - ``K`` Kilobytes (1024 bytes)
+ - ``M`` Megabytes (1024^2 or 1,048,576 bytes)
+ - ``G`` Gigabytes (1024^3 or 1,073,741,824 bytes)
+ - ``T`` Terabytes (1024^4 or 1,099,511,627,776 bytes)
+
+Storage Allocation
+------------------
+
+Allocation of span storage to volumes is done in stages. Storage is always
allocated in multiples of 128 megabytes,
+rounded down.
+
+* Explicitly sized span storage (:code:`cache:volumes:spans:size`) is
allocated to volumes. It is an error if the total allocated is larger than the
span size.
+ * Absolute sizes are allocated first.
+ * Percentages are allocated from remaining space.
+ * Remaining storage from spans that are used without an explicit size is
divided evenly among the volumes that use the span.
+* Span storage is allocated to volumes by the :code:`cache:volumes::size`
values.
+ * Absolute sizes are allocated first.
+ * Percentages are applied to remaining space.
+ * Remaining storage is divided evenly among volumes without an explicit
size.
+
+.. _storage-assignment-table:
+
+Assignment Table
+----------------
+
+Each storage element defined in :file:`storage.yaml` is divided in to
:term:`stripes <cache stripe>`. The
Review Comment:
In the Assignment Table section, the phrase "divided in to" should be
"divided into" ("Each storage element ... is divided into :term:`stripes <cache
stripe>`") for correct grammar.
```suggestion
Each storage element defined in :file:`storage.yaml` is divided into
:term:`stripes <cache stripe>`. The
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
