fawben opened a new issue, #110: URL: https://github.com/apache/airavata-mft/issues/110
**Describe the bug** I try to copy a file from an S3 storage to another S3 storage. Both buckets are hosted on [OVH](https://www.ovhcloud.com/en/public-cloud/object-storage/) provider. `mft ls storage-name` works and lists the objects present in the source and destination bucket. `mft cp storage-name1/file.txt storage-name2/` command creates a file on the destination bucket, but size is zero. `mft cp` command output: ``` user@mft-master-01:~$ mft cp blaze-test-s3-gra/hello.txt blaze-test-02-s3-gra/ Total number of 1 files to be transferred. Total volume is 6 bytes. Do you want to start the transfer? [Y/n]: y [------------------------------------] 0% local variable 'prev_percentage' referenced before assignment ``` There are some errors in airava.log file: ``` 2023-08-24 12:44:08,960 [pool-9-thread-20] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream {} - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2023-08-24 12:44:08,989 [pool-9-thread-20] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream {} - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2023-08-24 12:44:08,989 [pool-9-thread-20] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream {} - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2023-08-24 12:44:08,989 [pool-7-thread-8] ERROR org.apache.airavata.mft.transport.s3.S3OutgoingConnector {} - S3 failed to upload chunk to bucket blaze-test-02-s3-gra for resource path hello.txt 2023-08-24 12:44:08,990 [pool-7-thread-8] ERROR org.apache.airavata.mft.agent.TransportMediator {} - Transfer 9abb6f6c-7a06-4ed9-89cf-81bee0e7682b failed with error java.lang.NullPointerException: null at org.apache.airavata.mft.transport.s3.S3OutgoingConnector.complete(S3OutgoingConnector.java:134) ~[?:?] at org.apache.airavata.mft.agent.TransportMediator.transferSingleThread(TransportMediator.java:149) ~[mft-agent-service-0.01-SNAPSHOT.jar:0.01-SNAPSHOT] at org.apache.airavata.mft.agent.TransferOrchestrator.processTransfer(TransferOrchestrator.java:187) ~[mft-agent-service-0.01-SNAPSHOT.jar:0.01-SNAPSHOT] at org.apache.airavata.mft.agent.TransferOrchestrator.lambda$submitTransferToProcess$0(TransferOrchestrator.java:101) ~[mft-agent-service-0.01-SNAPSHOT.jar:0.01-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2023-08-24 12:44:08,995 [OkHttp http://localhost:8500/...] INFO org.apache.airavata.mft.controller.MFTController {} - Received state Key 9abb6f6c-7a06-4ed9-89cf-81bee0e7682b/local-agent/da256772-ae53-4cfa-a2b0-a71550defc19/0b8c6f040a1cdf526b05508a3bee7a10/1692881048992 val {"state":"FAILED","publisher":"local-agent","updateTimeMils":1692881048992,"percentage":0.0,"description":"Transfer failed due to java.lang.NullPointerException\n\tat org.apache.airavata.mft.transport.s3.S3OutgoingConnector.complete(S3OutgoingConnector.java:134)\n\tat org.apache.airavata.mft.agent.TransportMediator.transferSingleThread(TransportMediator.java:149)\n\tat org.apache.airavata.mft.agent.TransferOrchestrator.processTransfer(TransferOrchestrator.java:187)\n\tat org.apache.airavata.mft.agent.TransferOrchestrator.lambda$submitTransferToProcess$0(TransferOrchestrator.java:101)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.Futu reTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","childId":null} 2023-08-24 12:44:08,996 [OkHttp http://localhost:8500/...] INFO org.apache.airavata.mft.admin.MFTConsulClient {} - Saved transfer status {"state":"FAILED","publisher":"local-agent","updateTimeMils":1692881048992,"percentage":0.0,"description":"Transfer failed due to java.lang.NullPointerException\n\tat org.apache.airavata.mft.transport.s3.S3OutgoingConnector.complete(S3OutgoingConnector.java:134)\n\tat org.apache.airavata.mft.agent.TransportMediator.transferSingleThread(TransportMediator.java:149)\n\tat org.apache.airavata.mft.agent.TransferOrchestrator.processTransfer(TransferOrchestrator.java:187)\n\tat org.apache.airavata.mft.agent.TransferOrchestrator.lambda$submitTransferToProcess$0(TransferOrchestrator.java:101)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat jav a.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","childId":"0b8c6f040a1cdf526b05508a3bee7a10"} 2023-08-24 12:44:08,996 [OkHttp http://localhost:8500/...] INFO org.apache.airavata.mft.controller.MFTController {} - Deleting key mft/controller/messages/states/9abb6f6c-7a06-4ed9-89cf-81bee0e7682b/local-agent/da256772-ae53-4cfa-a2b0-a71550defc19/0b8c6f040a1cdf526b05508a3bee7a10/1692881048992 2023-08-24 12:44:08,996 [pool-7-thread-8] INFO org.apache.airavata.mft.agent.TransferOrchestrator {} - Removed transfer 9abb6f6c-7a06-4ed9-89cf-81bee0e7682b from queue with transfer success = false. Total running 13 ``` **To Reproduce** Create 2 storages of type S3 (OVH provider endpoint) with `mft storage add`. Run command: ` mft cp storage-name1/file.txt storage-name2/` **Expected behavior** The file should be present on the destination storage with the same size of the source file. **Screenshots** If applicable, add screenshots to help explain your problem. **Additional context** Add any other context about the problem here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
