lokeshdevnani opened a new issue, #15047:
URL: https://github.com/apache/druid/issues/15047
Range partitioning batch Ingestion failing in phase[partial segment merge].
Error message in partial merge subtask: `java.util.zip.ZipException: zip END
header not found`
### Affected Version
v0.25
### Description
#### Ingestion spec:
```
{
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "azure",
"prefixes": [
"azure://druid-segments/raw_jsons/datadir/dp_date=2023-04-01/part-00020"
],
"filter": "*.json"
},
"inputFormat": {
"type": "json"
},
"appendToExisting": false
},
"tuningConfig": {
"type": "index_parallel",
"partitionsSpec": {
"type": "range",
"partitionDimensions": [
"event_name",
"org_id"
],
"targetRowsPerSegment": 5000000
},
"maxRowsInMemory": 5000000,
"maxNumConcurrentSubTasks": 2,
"forceGuaranteedRollup": true
},
"dataSchema": {
"dataSource": "udappevents",
"timestampSpec": {
"column": "server_timestamp",
"format": "millis"
},
"transformSpec": {
"filter": {
"type": "interval",
"dimension": "__time",
"intervals": [
"2023-03-01T00:00:00.000Z/2023-04-03T20:00:00.000Z"
]
}
},
"granularitySpec": {
"queryGranularity": "none",
"rollup": false,
"segmentGranularity": "day"
},
"dimensionsSpec": {
"dimensionExclusions": []
}
}
}
}
```
#### Error logs in **partial_index_generic_merge** task
```
2023-09-27T10:58:35,826 TRACE [ServiceClientFactory-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory - isGood =
isSucess[true] && isConnected[true] && isOpen[true]
2023-09-27T10:58:35,836 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] messageReceived:
DefaultHttpResponse(chunked: true)
HTTP/1.1 200 OK
Date: Wed, 27 Sep 2023 10:58:35 GMT
Content-Type: application/json
Vary: Accept-Encoding, User-Agent
Transfer-Encoding: chunked
2023-09-27T10:58:35,836 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] Got response: 200 OK
2023-09-27T10:58:35,837 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] messageReceived:
org.jboss.netty.handler.codec.http.DefaultHttpChunk@5ad93650
2023-09-27T10:58:35,837 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] Got chunk: 263B, last=false
2023-09-27T10:58:35,837 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] messageReceived:
org.jboss.netty.handler.codec.http.HttpChunk$1@2dd00f5a
2023-09-27T10:58:35,837 DEBUG [HttpClient-Netty-Worker-20]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://10.72.10.179:8088/druid/indexer/v1/action] Got chunk: 0B, last=true
2023-09-27T10:58:35,838 DEBUG [ServiceClientFactory-2]
org.apache.druid.rpc.ServiceClientImpl - Service [OVERLORD] request [POST
http://10.72.10.179:8088/druid/indexer/v1/action] completed.
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.NettyHttpClient - [GET
http://10.72.10.179:8088/druid/worker/v1/shuffle/task/index_parallel_udappevents_aiphkhjc_2023-09-27T10%3A56%3A53.490Z/partial_range_index_generate_udappevents_dbigclaf_2023-09-27T10%3A57%3A07.964Z/partition?startTime=2023-04-01T00:00:00.000Z&endTime=2023-04-02T00:00:00.000Z&bucketId=0]
starting
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,850 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,851 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 DEBUG [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory -
Generating: http://10.72.10.179:8088
2023-09-27T10:58:35,852 TRACE [task-runner-0-priority-0]
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory - isGood =
isSucess[true] && isConnected[true] && isOpen[true]
2023-09-27T10:58:35,856 DEBUG [HttpClient-Netty-Worker-0]
org.apache.druid.java.util.http.client.NettyHttpClient - [GET
http://10.72.10.179:8088/druid/worker/v1/shuffle/task/index_parallel_udappevents_aiphkhjc_2023-09-27T10%3A56%3A53.490Z/partial_range_index_generate_udappevents_dbigclaf_2023-09-27T10%3A57%3A07.964Z/partition?startTime=2023-04-01T00:00:00.000Z&endTime=2023-04-02T00:00:00.000Z&bucketId=0]
messageReceived: DefaultHttpResponse(chunked: false)
HTTP/1.1 404 Not Found
Content-Security-Policy: frame-ancestors 'none'
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 545
2023-09-27T10:58:35,856 DEBUG [HttpClient-Netty-Worker-0]
org.apache.druid.java.util.http.client.NettyHttpClient - [GET
http://10.72.10.179:8088/druid/worker/v1/shuffle/task/index_parallel_udappevents_aiphkhjc_2023-09-27T10%3A56%3A53.490Z/partial_range_index_generate_udappevents_dbigclaf_2023-09-27T10%3A57%3A07.964Z/partition?startTime=2023-04-01T00:00:00.000Z&endTime=2023-04-02T00:00:00.000Z&bucketId=0]
Got response: 404 Not Found
2023-09-27T10:58:35,862 INFO [task-runner-0-priority-0]
org.apache.druid.utils.CompressionUtils - Unzipping
file[/druid/data/persistent/task/partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z/work/indexing-tmp/2023-04-01T00:00:00.000Z/2023-04-02T00:00:00.000Z/0/temp_partial_range_index_generate_udappevents_dbigclaf_2023-09-27T10:57:07.964Z]
to
[/druid/data/persistent/task/partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z/work/indexing-tmp/2023-04-01T00:00:00.000Z/2023-04-02T00:00:00.000Z/0/unzipped_partial_range_index_generate_udappevents_dbigclaf_2023-09-27T10:57:07.964Z]
2023-09-27T10:58:35,862 DEBUG [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.AbstractTask - Not pushing task logs and
reports from task.
2023-09-27T10:58:35,862 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while
running
task[AbstractTask{id='partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z',
groupId='index_parallel_udappevents_aiphkhjc_2023-09-27T10:56:53.490Z',
taskResource=TaskResource{availabilityGroup='partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z',
requiredCapacity=1}, dataSource='udappevents',
context={forceTimeChunkLock=true, useLineageBasedSegmentAllocation=true}}]
java.util.zip.ZipException: zip END header not found
at java.util.zip.ZipFile$Source.zerror(ZipFile.java:1607) ~[?:?]
at java.util.zip.ZipFile$Source.findEND(ZipFile.java:1497) ~[?:?]
at java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1504) ~[?:?]
at java.util.zip.ZipFile$Source.<init>(ZipFile.java:1308) ~[?:?]
at java.util.zip.ZipFile$Source.get(ZipFile.java:1271) ~[?:?]
at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:733)
~[?:?]
at java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:850) ~[?:?]
at java.util.zip.ZipFile.<init>(ZipFile.java:248) ~[?:?]
at java.util.zip.ZipFile.<init>(ZipFile.java:177) ~[?:?]
at java.util.zip.ZipFile.<init>(ZipFile.java:191) ~[?:?]
at
org.apache.druid.utils.CompressionUtils.unzip(CompressionUtils.java:236)
~[druid-core-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.batch.parallel.HttpShuffleClient.fetchSegmentFile(HttpShuffleClient.java:92)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.batch.parallel.HttpShuffleClient.fetchSegmentFile(HttpShuffleClient.java:43)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.batch.parallel.PartialSegmentMergeTask.fetchSegmentFiles(PartialSegmentMergeTask.java:228)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.batch.parallel.PartialSegmentMergeTask.runTask(PartialSegmentMergeTask.java:173)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.batch.parallel.PartialGenericSegmentMergeTask.runTask(PartialGenericSegmentMergeTask.java:41)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449)
~[druid-indexing-service-25.0.0.jar:25.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-09-27T10:58:35,863 DEBUG [task-runner-0-priority-0]
org.apache.druid.indexing.overlord.TaskRunnerUtils - Task
[partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z]
status changed to [FAILED].
2023-09-27T10:58:35,865 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed
with status: {
"id" :
"partial_index_generic_merge_udappevents_bkajccff_2023-09-27T10:58:31.973Z",
"status" : "FAILED",
"duration" : 1010,
"errorMsg" : "java.util.zip.ZipException: zip END header not found",
"location" : {
"host" : null,
"port" : -1,
"tlsPort" : -1
}
}
2023-09-27T10:58:35,869 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS]
2023-09-27T10:58:35,869 DEBUG [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Invoking stop method[public void
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer.stop()] on
object[org.apache.druid.server.coordination.BatchDataSegmentAnnouncer@49741e80].
2023-09-27T10:58:35,870 DEBUG [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Invoking stop method[public void
org.apache.druid.curator.announcement.Announcer.stop()] on
object[org.apache.druid.curator.announcement.Announcer@60658389].
2023-09-27T10:58:35,870 DEBUG [main]
org.apache.druid.curator.announcement.Announcer - Stopping Announcer.
2023-09-27T10:58:35,870 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER]
2023-09-27T10:58:35,870 DEBUG [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Skipping
unannounce wait.
2023-09-27T10:58:35,870 DEBUG [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Stopping Jetty
Server...
2023-09-27T10:58:35,870 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
Server@31531d0d{STARTED}[9.4.48.v20220622]
2023-09-27T10:58:35,870 DEBUG [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty
lifecycle stopping [class org.eclipse.jetty.server.Server]
2023-09-27T10:58:35,870 DEBUG [main] org.eclipse.jetty.server.Server -
doStop Server@31531d0d{STOPPING}[9.4.48.v20220622]
2023-09-27T10:58:35,871 DEBUG [qtp1333721416-107]
org.eclipse.jetty.util.thread.QueuedThreadPool - ran
qtp1333721416-107-acceptor-0@594fcb22-ServerConnector@69944a90{HTTP/1.1,
(http/1.1)}{0.0.0.0:8101} in
QueuedThreadPool[qtp1333721416]@4f7ef948{STARTED,48<=48<=48,i=46,r=-1,q=0}[ReservedThreadExecutor@6d946eee{reserved=0/2,pending=0}]
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.server.handler.AbstractHandlerContainer - Graceful shutdown
Server@31531d0d{STOPPING}[9.4.48.v20220622] by
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
ServerConnector@69944a90{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
JettyMonitoringConnectionFactory@663cf5d7{STARTED}
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
HttpConnectionFactory@6f976c[HTTP/1.1]
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - STOPPED
HttpConnectionFactory@6f976c[HTTP/1.1]
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - STOPPED
JettyMonitoringConnectionFactory@663cf5d7{STOPPED}
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
SelectorManager@ServerConnector@69944a90{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2023-09-27T10:58:35,872 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
ManagedSelector@1c7fd2e0{STARTED} id=0 keys=0 selected=0 updates=0
2023-09-27T10:58:35,872 DEBUG [main] org.eclipse.jetty.io.ManagedSelector -
Queued change lazy=false
org.eclipse.jetty.io.ManagedSelector$CloseConnections@1be3a294 on
ManagedSelector@1c7fd2e0{STOPPING} id=0 keys=0 selected=0 updates=0
2023-09-27T10:58:35,873 DEBUG [main] org.eclipse.jetty.io.ManagedSelector -
Wakeup on submit ManagedSelector@1c7fd2e0{STOPPING} id=0 keys=0 selected=0
updates=1
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 woken with none selected
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 woken up from select, 0/0/0 selected
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 processing 0 keys, 1 updates
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - updateable 1
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - update
org.eclipse.jetty.io.ManagedSelector$CloseConnections@1be3a294
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Closing 0 connections on
ManagedSelector@1c7fd2e0{STOPPING} id=0 keys=0 selected=0 updates=0
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - updates 0
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 waiting with 0 keys
2023-09-27T10:58:35,873 DEBUG [main] org.eclipse.jetty.io.ManagedSelector -
Queued change lazy=false
org.eclipse.jetty.io.ManagedSelector$StopSelector@729d1428 on
ManagedSelector@1c7fd2e0{STOPPING} id=0 keys=0 selected=0 updates=0
2023-09-27T10:58:35,873 DEBUG [main] org.eclipse.jetty.io.ManagedSelector -
Wakeup on submit ManagedSelector@1c7fd2e0{STOPPING} id=0 keys=0 selected=0
updates=1
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 woken with none selected
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 woken up from select, 0/0/0 selected
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - Selector
sun.nio.ch.EPollSelectorImpl@b60cd25 processing 0 keys, 1 updates
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - updateable 1
2023-09-27T10:58:35,873 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - update
org.eclipse.jetty.io.ManagedSelector$StopSelector@729d1428
2023-09-27T10:58:35,874 DEBUG [qtp1333721416-108]
org.eclipse.jetty.io.ManagedSelector - updates 0
2023-09-27T10:58:35,875 DEBUG [main]
org.eclipse.jetty.util.component.AbstractLifeCycle - stopping
```
#### Steps to reproduce the problem
Run the batch ingestion task with range partitioning.
- I can see that the shuffle APIs are returning 404.
- No OOM issues. The ingestion size is fairly small - a couple of MBs.
- Coordinators is running as overlord.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
