Claus Ibsen created CAMEL-27182:
-----------------------------------

             Summary: camel-azure-storage-datalake - missing fileName logs a 
WARN at every poll
                 Key: CAMEL-27182
                 URL: https://issues.apache.org/jira/browse/CAMEL-27182
             Project: Camel
          Issue Type: Bug
          Components: camel-azure
            Reporter: Claus Ibsen


h3. Problem
With {{fileName}} set to a file that does not exist (yet), every poll fails and 
the scheduled consumer logs a WARN with a 
{{com.azure.storage.blob.models.BlobStorageException}} (status 404, 
BlobNotFound), every {{delay}} (500 ms by default).

{{DataLakeConsumer.poll}} means to turn a 404 into an empty poll, but it 
catches only {{DataLakeStorageException}}:
{code:java}
} catch (DataLakeStorageException e) {
    if (NOT_FOUND == e.getStatusCode()) {
        result = 0;
{code}
The read without {{fileDir}} goes through 
{{DataLakeFileClientWrapper.openInputStream}} -> 
{{DataLakeFileClient.openInputStream}}. In the Azure SDK 
(azure-storage-file-datalake 12.26.x) that method calls 
{{BlockBlobClient.openInputStream}} and does not translate the exception, so a 
missing file throws {{BlobStorageException}}, not {{DataLakeStorageException}}.

The same applies in {{createBatchExchangesFromPath}}: a listed file that is 
deleted before it is read (for example by another consumer) is meant to be 
skipped with a DEBUG log on 404, but the {{BlobStorageException}} is not caught 
there either, so the file is counted as a failure (reported to the exception 
handler, or the poll fails when it is the only file).

camel-azure-storage-blob returns an empty poll in this case 
({{BlobConsumer.poll}} catches {{BlobStorageException}} 404).

h3. How to reproduce
{noformat}
from("azure-storage-datalake:myaccount/myfs?fileName=missing.csv&dataLakeServiceClient=#client")
    .log("${body}");
{noformat}
Each poll logs a WARN with {{BlobStorageException}} ... {{Status code 404}}, 
nothing else happens. Seen while testing the Kamelet 
{{azure-storage-datalake-source}} with its new {{fileName}} option.

h3. Where
* 
{{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
 - {{poll}} (catch of {{DataLakeStorageException}} only)
* same file - {{createBatchExchangesFromPath}} (404 check for a listed file)

h3. Suggested fix
Treat a 404 of either exception type as not found, for example a small helper:
{code:java}
private static boolean isNotFound(Exception e) {
    return e instanceof DataLakeStorageException d && d.getStatusCode() == 404
            || e instanceof BlobStorageException b && b.getStatusCode() == 404;
}
{code}
and use it in both places (or wrap the read with 
{{DataLakeImplUtils.returnOrConvertException}}-style translation in 
{{DataLakeFileClientWrapper.openInputStream}}).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to