Hallo all,
I'm currently unable to import/activate VM due to error above.
It has been a long series of unfortunate events.
Long story short:
- VM running on storage on iscsi failing disks
- put storage and server in maint. Shut down server
- restart server.
- server has a misconfiguration on lvm/multipath rules (not due to ovirt)
- put storage and server active
- cluster goes down. Put storage in maintenance again. Detached storage from
cluster (ach! That was unfortunate).
- cluster now ok. Storage ok. Activate storage.
- trying to import VM. Ovirt replies with error in subject and attached.
How can I recover from this situation?
It would be great to recover the entire VM, but I'm more interested in disks
which are in unregistered state than VM itself.
Thank you all
2016-08-14 17:49:32,414 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] Lock Acquired to object EngineLock [exclus
iveLocks= key: Hertz value: VM_NAME
, sharedLocks= key: 49950ad2-05cd-4c5a-bde7-b702199fa250 value: REMOTE_VM
]
2016-08-14 17:49:32,446 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s
toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit =
false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId =
29b97d45-e481-4
35d-a50a-f2a890a678bf, imageId = 0225fa05-be1e-44a6-9d9e-1203b8144933), log id:
3d5b900c
2016-08-14 17:49:32,604 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand,
return: true, log id: 3d5b900c
2016-08-14 17:49:32,605 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s
toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit =
false, storageDomainId = a074b2a8-0cd3-4d28-bc29-ad37587a8777, imageGroupId =
bba3bd45-d02d-4
032-8bb0-e723b7acf23b, imageId = 20f9d5eb-b4d9-4a5d-b75e-3af013c56604), log id:
464e234c
2016-08-14 17:49:32,792 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand,
return: true, log id: 464e234c
2016-08-14 17:49:32,792 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s
toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit =
false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId =
6804229e-68f7-4
6a8-90d3-c065529f9624, imageId = 76b4f302-5182-478d-a7d3-cf4c8b6abe40), log id:
254ca74a
2016-08-14 17:49:32,815 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand,
return: true, log id: 254ca74a
2016-08-14 17:49:32,815 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s
toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit =
false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId =
217a34f9-ca8c-4a13-a2a2-9dde4c329c55, imageId =
621c0b0c-c041-46e0-8c00-e69896fcf554), log id: 4b7afe67
2016-08-14 17:49:32,842 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, return:
true, log id: 4b7afe67
2016-08-14 17:49:32,843 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand(
storagePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit =
false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId =
efb67e0b-c592-406a-949c-121102eef796, imageId =
44420cae-7c90-461a-bee1-7552758ecee9), log id: 77d442e7
2016-08-14 17:49:32,865 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand,
2016-08-14 17:49:32,908 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Running command:
ImportVmFromConfigurationCommand internal: false. Entities affected : ID:
a074b2a8-0cd3-4d28-bc29-ad37587a8777 Type: StorageAction group IMPORT_EXPORT_VM
with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type:
StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID:
2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM
with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type:
StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID:
2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM
with role type ADMIN
2016-08-14 17:49:33,079 INFO
[org.ovirt.engine.core.utils.transaction.TransactionSupport]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] transaction rolled back
2016-08-14 17:49:33,080 ERROR
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand throw exception:
org.springframework.dao.DuplicateKeyException: CallableStatementCallback; SQL
[{call insertimage(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: duplicate key
value violates unique constraint "pk_images"
Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid,
size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type,
image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 ,
$6 , $7 , $8 , $9 , $10 , $11 , $12 )"
PL/pgSQL function "insertimage" line 2 at SQL statement; nested exception is
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
constraint "pk_images"
Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid,
size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type,
image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 ,
$6 , $7 , $8 , $9 , $10 , $11 , $12 )"
PL/pgSQL function "insertimage" line 2 at SQL statement
at
org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:241)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1030)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1064)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:388)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:351)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141)
[dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:136)
[dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:74)
[dal.jar:]
at
org.ovirt.engine.core.dao.DefaultGenericDaoDbFacade.save(DefaultGenericDaoDbFacade.java:93)
[dal.jar:]
at
org.ovirt.engine.core.bll.BaseImagesCommand.saveImage(BaseImagesCommand.java:405)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand.saveImage(ImportVmCommand.java:952)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand.addVmImagesAndSnapshots(ImportVmCommand.java:917)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand$4.runInTransaction(ImportVmCommand.java:737)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand$4.runInTransaction(ImportVmCommand.java:733)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:210)
[utils.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand.processImages(ImportVmCommand.java:733)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmCommand.executeCommand(ImportVmCommand.java:702)
[bll.jar:]
at
org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand.executeCommand(ImportVmFromConfigurationCommand.java:129)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1199)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1338)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1967)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
[utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
[utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1362)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:359)
[bll.jar:]
at
org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:193)
[bll.jar:]
at
org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:160)
[bll.jar:]
at
org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:169)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90)
[utils.jar:]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[rt.jar:1.7.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_111]
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "pk_images"
Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid,
size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type,
image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 ,
$6 , $7 , $8 , $9 , $10 , $11 , $12 )"
PL/pgSQL function "insertimage" line 2 at SQL statement
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:381)
at
org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:297)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
at
org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1066)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1)
[spring-jdbc.jar:3.1.1.RELEASE]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1014)
[spring-jdbc.jar:3.1.1.RELEASE]
... 33 more
2016-08-14 17:49:33,094 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
[id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of
org.ovirt.engine.core.common.businessentities.VmStatistics; snapshot:
49950ad2-05cd-4c5a-bde7-b702199fa250.
2016-08-14 17:49:33,095 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
[id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of
org.ovirt.engine.core.common.businessentities.network.VmNetworkStatistics;
snapshot: ee0be571-554e-4a2f-9822-efc134f068fb.
2016-08-14 17:49:33,097 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
[id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of
org.ovirt.engine.core.common.businessentities.network.VmNetworkInterface;
snapshot: ee0be571-554e-4a2f-9822-efc134f068fb.
2016-08-14 17:49:33,098 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
[id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of
org.ovirt.engine.core.common.businessentities.VmDynamic; snapshot:
49950ad2-05cd-4c5a-bde7-b702199fa250.
2016-08-14 17:49:33,099 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command
[id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of
org.ovirt.engine.core.common.businessentities.VmStatic; snapshot:
49950ad2-05cd-4c5a-bde7-b702199fa250.
2016-08-14 17:49:33,108 INFO
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(org.ovirt.thread.pool-8-thread-42) [446f3b0e] Lock freed to object EngineLock
[exclusiveLocks= key: Hertz value: VM_NAME
, sharedLocks= key: 49950ad2-05cd-4c5a-bde7-b702199fa250 value: REMOTE_VM
]
2016-08-14 17:50:06,441 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain
2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L
2016-08-14 17:50:06,551 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) vds primitivo reported domain
2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting
to move the vds to status NonOperational
2016-08-14 17:50:06,552 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) vds sovana reported domain
2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting
to move the vds to status NonOperational
2016-08-14 17:50:06,552 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) vds Lambrusco reported domain
2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting
to move the vds to status NonOperational
2016-08-14 17:50:06,553 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) Removing vds
[6d37d71e-c8c5-47c8-a896-5112298bb83f, 89254f23-8748-402a-afc9-08438dca0975,
a21e6b4f-7d06-49c5-8463-680d78517c73] from the domain in maintenance cache
2016-08-14 17:50:06,554 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-26) Removing host(s)
[6d37d71e-c8c5-47c8-a896-5112298bb83f, 89254f23-8748-402a-afc9-08438dca0975,
a21e6b4f-7d06-49c5-8463-680d78517c73] from hosts unseen domain report cache
2016-08-14 17:50:06,595 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(org.ovirt.thread.pool-8-thread-38) [11443d31] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
89254f23-8748-402a-afc9-08438dca0975 Type: VDS
2016-08-14 17:50:06,607 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(org.ovirt.thread.pool-8-thread-34) [170d64d2] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
a21e6b4f-7d06-49c5-8463-680d78517c73 Type: VDS
2016-08-14 17:50:06,610 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-38) [11443d31] START,
SetVdsStatusVDSCommand(HostName = sovana, HostId =
89254f23-8748-402a-afc9-08438dca0975, status=NonOperational,
nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false),
log id: 6463356e
2016-08-14 17:50:06,621 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(org.ovirt.thread.pool-8-thread-26) [4449c225] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
6d37d71e-c8c5-47c8-a896-5112298bb83f Type: VDS
2016-08-14 17:50:06,623 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-34) [170d64d2] START,
SetVdsStatusVDSCommand(HostName = Lambrusco, HostId =
a21e6b4f-7d06-49c5-8463-680d78517c73, status=NonOperational,
nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false),
log id: 704153b7
2016-08-14 17:50:06,623 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-38) [11443d31] FINISH, SetVdsStatusVDSCommand,
log id: 6463356e
2016-08-14 17:50:06,635 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-34) [170d64d2] FINISH, SetVdsStatusVDSCommand,
log id: 704153b7
2016-08-14 17:50:06,644 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-26) [4449c225] START,
SetVdsStatusVDSCommand(HostName = primitivo, HostId =
6d37d71e-c8c5-47c8-a896-5112298bb83f, status=NonOperational,
nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false),
log id: 522d70ef
2016-08-14 17:50:06,657 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-38) [11443d31] Correlation ID: 11443d31, Job
ID: 144d8e4d-3560-4a13-9307-2c08977b6c0c, Call Stack: null, Custom Event ID:
-1, Message: Host sovana cannot access the Storage Domain(s) Hertz-dstore1-L
attached to the Data Center IRDC-INTEL. Setting Host state to Non-Operational.
2016-08-14 17:50:06,666 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-34) [170d64d2] Correlation ID: 170d64d2, Job
ID: 91f007cf-2d72-4f69-a7a0-023f150a1c59, Call Stack: null, Custom Event ID:
-1, Message: Host Lambrusco cannot access the Storage Domain(s) Hertz-dstore1-L
attached to the Data Center IRDC-INTEL. Setting Host state to Non-Operational.
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
