[
https://issues.apache.org/jira/browse/UNOMI-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706508#comment-17706508
]
Kevan Jahanshahi edited comment on UNOMI-758 at 3/29/23 4:40 PM:
-----------------------------------------------------------------
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
This give logs like:
{noformat}
18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export ::
Collect Data'
18:36:37.683 INFO [pool-21-thread-1] Route:
82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from:
timer://collectProfile?fixedRate=true&period=1h
18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile]
Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed.
18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396
18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Connecting to ftp://[email protected]:21 using connection timeout: 10000
18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using
passive mode connections
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using
SoTimeout=300000
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Attempting to login user: kevan using password: ********
18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User
kevan logged in: true
18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
buildDirectory(data3TO/trash)
18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(/)
18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: /
18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
storeFile(data3TO/trash/profiles-export-2023032918.csv)
18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
getCurrentDirectory()
18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current
dir: /
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(data3TO/trash)
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: data3TO
18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: trash
18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
doStoreFile(profiles-export-2023032918.csv)
18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About
to store file: profiles-export-2023032918.csv using stream:
java.io.ByteArrayInputStream@3cc24cfa
18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client
appendFile: profiles-export-2023032918.csv
18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took
0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP
client returned: true
18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(/)
18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: /{noformat}
was (Author: jkevan):
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
This give logs like:
{noformat}
18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export ::
Collect Data'
18:36:37.683 INFO [pool-21-thread-1] Route:
82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from:
timer://collectProfile?fixedRate=true&period=1h
18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile]
Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed.
18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396
18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Connecting to ftp://[email protected]:21 using connection timeout: 10000
18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using
passive mode connections
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using
SoTimeout=300000
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Attempting to login user: kevan using password: ********
18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User
kevan logged in: true
18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
buildDirectory(data3TO/trash)
18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(/)
18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: /
18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
storeFile(data3TO/trash/profiles-export-2023032918.csv)
18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
getCurrentDirectory()
18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current
dir: /
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(data3TO/trash)
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: data3TO
18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: trash
18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
doStoreFile(profiles-export-2023032918.csv)
18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About
to store file: profiles-export-2023032918.csv using stream:
java.io.ByteArrayInputStream@3cc24cfa
18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client
appendFile: profiles-export-2023032918.csv
18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took
0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP
client returned: true
18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
changeCurrentDirectory(/)
18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile]
Changing directory: /{noformat}
> Bunch of fixes and improvements for Profile Export system
> ---------------------------------------------------------
>
> Key: UNOMI-758
> URL: https://issues.apache.org/jira/browse/UNOMI-758
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: David Griffon
> Assignee: David Griffon
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Issue 1: make the camel route update asynchron to avoid blocking the
> requests until camel route is killed.
> When a route is updated, we stop all execution of the current route.
> This stop can take time, and can lead to a request timeout.
> The goal of this task is to stop the execution asynchronously.
> h2. Issue 2: provide clear error messages in case something fail during camel
> route creation/update
> currently we are hiding the Exceptions:
> inside the class
> {*}org.apache.unomi.router.services.ExportConfigurationServiceImpl.java{*},
> there is no stack trace:
> {noformat}
> 2022-10-26T16:15:56,667 | ERROR | qtp824972552-99936 |
> ExportConfigurationServiceImpl | 183 - org.apache.unomi.router-service -
> 1.4.4.jahia | Error when trying to save/update running Apache Camel Route:
> d9fa4556-a967-4410-aa1f-869ad66dbda5 {noformat}
> h2. Issue 3: investigate on truncated CSV export to FTP
> Sometimes we have CSV file that is truncated when exporting to FTP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
