[ 
https://issues.apache.org/jira/browse/CAMEL-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen reassigned CAMEL-27181:
-----------------------------------

    Fix Version/s: 4.24.0
         Assignee: Claus Ibsen

> camel-azure-storage-datalake - consumer has no deleteAfterRead option
> ---------------------------------------------------------------------
>
>                 Key: CAMEL-27181
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27181
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-azure
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Minor
>             Fix For: 4.24.0
>
>
> h3. Problem
> The Data Lake consumer re-reads every file of the file system (or of 
> {{path}}) at every poll, and it has no option to delete a file once its 
> exchange completes. {{DataLakeConsumer.processBatch}} adds an on-completion 
> whose {{onComplete}} only logs at TRACE. The only way to consume each file 
> once is the Idempotent Consumer EIP, which runs after the consumer has 
> already opened the file.
> camel-azure-storage-blob has {{deleteAfterRead}} and {{moveAfterRead}} 
> ({{BlobConfiguration}}); {{DataLakeConfiguration}} has neither.
> The Kamelet {{azure-storage-datalake-source}} cannot offer a 
> delete-after-read option because of this, while {{azure-storage-blob-source}} 
> can.
> h3. Where
> {{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
>  - {{processBatch}} (the {{Synchronization.onComplete}} does nothing).
> h3. Suggested fix
> Add a consumer option {{deleteAfterRead}} (default false), as in 
> camel-azure-storage-blob:
> * take the file name when the exchange is created (the route may change the 
> {{CamelAzureStorageDataLakeFileName}} header);
> * in {{onComplete}}, delete it with {{DataLakeFileClient.deleteWithResponse}} 
> ({{DataLakeFileClientWrapper.delete}} already exists); a 404 is ignored, 
> other failures go to the exception handler;
> * nothing is deleted on failure, so the file is read again at the next poll.
> A {{moveAfterRead}} (rename to another directory) could follow the same 
> pattern.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to