Hello Suresh,
That primary storage was part of the local storage of a host that was removed.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool
WHERE id = 5
+----------------+------+------------+-------+---------------------+---------------------+
| name | uuid | pool_type | scope | created | removed
|
+----------------+------+------------+-------+---------------------+---------------------+
| cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19
02:11:51 |
+----------------+------+------------+-------+---------------------+——————————+
SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state
FROM cloud.snapshot_store_ref WHERE store_id = '5';
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | snapshot_id | store_role | install_path
| size
| state |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 155 | 5 | 109 | Primary |
/var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
| 85899345920 | Ready |
| 155 | 5 | 117 | Primary |
/var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46
| 85899345920 | Ready |
| 155 | 5 | 127 | Primary |
/var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014
| 85899345920 | Ready |
| 155 | 5 | 138 | Primary |
/var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529
| 85899345920 | Ready |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
All 155 snapshots have the same account_id and domain_id.
SELECT id, uuid, name, status, path, created, removed, location_type FROM
cloud.snapshots WHERE volume_id = '155';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| id | uuid | name |
status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 |
Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 |
BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 |
BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 |
BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 |
BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 |
BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 |
Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 |
Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 |
Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
| 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 |
Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
| 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 |
Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
| 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 |
BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
| 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 |
Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
| 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 |
Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
| 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 |
BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
| 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 |
BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
| 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 |
BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
| 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 |
BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
| 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 |
Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
| 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 |
BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
| 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 |
BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
How would you suggest I resolve the issue moving forward?
I tried the following with error…
(nimbus) 🐱 > delete snapshot id=109
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:32:08-0400",
"created": "2022-05-11T11:32:08-0400",
"jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
"jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Command failed due to Internal Server Error"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
Deleting some table rows in the right sequence?
Same issue with id=117 to id=138, but I was able to delete id=146
Regards,
Antoine
> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <[email protected]>
> wrote:
>
> Hi Antoine,
>
> The NPE error seems to be due to the primary storage with id '5', when
> checking for snapshot revertible or not during list snapshots cmd. Is that
> primary storage removed? Can you check with sql below.
>
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool
> WHERE id = 5
>
>
> I think, the account/user might not have the permissions to list all
> snapshots. You can check the account / domain details in snapshots table.
>
>
> Regards,
> Suresh
>
> On 21/04/22, 6:50 PM, "Antoine Boucher" <[email protected]> wrote:
>
> Thank you Suresh, you will find my answers below,
>
>
>
>
>
>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <[email protected]>
>> wrote:
>>
>> Hi Antoine,
>>
>> What is the CloudStack version, where you see this issue.
>
> Latest - 4.16.1
>
>
>> When you noticed this issue, is it while listing snapshots for a particular
>> volume, or all volumes? Try list snapshots from API/cmk as well.
>
> This is a customer VM, the account may have hit its snapshot limit during
> a snapshot.
>
> cmk list snapshots > only lists 26 snapshots out of 62
>
> cmk list snapshots volumeid= > Works for some volume ids, provides null
> result while the WebUI lists the correct snapshots for others and provides
> null results while show undefined error for one VM mentioned below
>
>
>>
>> Also, Check the snapshots details for the volume (with issue) in the db,
>> using the following sql queries.
>>
>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes
>> WHERE name LIKE '<volume_name>'
>>
>
> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM
> cloud.volumes WHERE name LIKE 'ROOT-110';
>
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
> | id | uuid | state | pool_id | path
> | size | removed |
>
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
> | 111 | NULL | Expunged | 5 |
> da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 |
> 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>
>
>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM
>> cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>
> mysql> SELECT id, uuid, name, status, path, created, removed,
> location_type FROM cloud.snapshots WHERE volume_id = '138';
>
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
> | id | uuid | name
> | status | path | created | removed | location_type |
>
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 |
> TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL
> | NULL |
> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 |
> TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL
> | NULL |
> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a |
> TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL
> | NULL |
> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df |
> TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL
> | NULL |
> | 138 | e9639f4a-fa09-4698-849b-572011045993 |
> TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL
> | NULL |
> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 |
> TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL
> | NULL |
> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b |
> TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL
> | NULL |
> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20
> | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 |
> TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL
> | NULL |
>
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>
>
>>
>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM
>> cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>
>
> mysql> SELECT volume_id, store_id, store_role, install_path, size, state
> FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> | volume_id | store_id | store_role | install_path
> | size | state |
>
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> | 138 | 1 | Image |
> snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 |
> Destroying |
>
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> 1 row in set (0.00 sec)
>
> mysql> SELECT volume_id, store_id, store_role, install_path, size, state
> FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | volume_id | store_id | store_role | install_path
> | size |
> state |
>
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | 138 | 5 | Primary |
> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
> | 85899345920 | Ready |
> | 138 | 1 | Image |
> snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5
> | 85899345920 | Ready |
>
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> 2 rows in set (0.00 sec)
>
> Regards,
> Antoine
>
>> Regards,
>> Suresh
>>
>> On 21/04/22, 8:45 AM, "Antoine Boucher" <[email protected]> wrote:
>>
>> Unfortunately I spoke too fast. The list is back but listing the snapshots
>> of the volume with issue still show a blank list with error “undefined”.
>>
>>
>>
>>
>> On Apr 20, 2022, at 23:11, Antoine Boucher <[email protected]> wrote:
>>
>> I was able to recover from the situation by forcing a snapshot on the
>> volume with issue.
>>
>> On Apr 20, 2022, at 22:00, Antoine Boucher <[email protected]> wrote:
>>
>> I pin-pointed the volume with the issue by going through all volumes and
>> clicking the “view snapshot"
>>
>>
>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <[email protected]> wrote:
>>>
>>> I’m no longer able to see my list of snapshots (storage > snapshots),
>>> instead I see a small popup window with the word “undefined" on an empty
>>> list snapshot list.
>>>
>>> Has anyone seen this issue? I restart and rebooted the Management server
>>> but no resolution.
>>>
>>> Here are my logs:
>>>
>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet]
>>> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1
>>> -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer]
>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from
>>> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account
>>> {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}'
>>> is allowed to perform API calls: 0.0.0.0/0,::/0
>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer]
>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled
>>> exception executing api command: [Ljava.lang.String;@42fa8f7d
>>> java.lang.NullPointerException
>>> at
>>> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>> at
>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>> at
>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>> at
>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>> at
>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>> at
>>> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>> at
>>> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>> at
>>> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>> at
>>> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>> at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> at
>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>> at
>>> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>> at
>>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>> at
>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>> at
>>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>> at
>>> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>> at
>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet]
>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END===
>>> 10.101.254.1 -- GET
>>> listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>
>>> Antoine Boucher
>>>
>>>
>>> Confidentiality Warning: This message and any attachments are intended only
>>> for the use of the intended recipient(s), are confidential, and may be
>>> privileged. If you are not the intended recipient, you are hereby notified
>>> that any review, retransmission, conversion to hard copy, copying,
>>> circulation or other use of this message and any attachments is strictly
>>> prohibited. If you are not the intended recipient, please notify the sender
>>> immediately by return e-mail, and delete this message and any attachments
>>> from your system.
>>>
>>
>>
>
>
