Hi Jeff, None of this is applied in pipeline and FetchFile processor. It is not on cluster, it runs only on one standalone Nifi instance. Completion strategy is on None, nor deleting, nor Moving.
Only thing that can be is that someone else uses the file at the same time because that shared disk is used by other people to who are reading the files and doing some analysis. Can that be also the cause for Truncated ZIP file on UnpackContent processor? I applied loopback relationship on that processors for failure flowfiles to retry on failure. Thanks. Tom On Thu, 3 Oct 2019 at 17:18, Jeff <jtsw...@gmail.com> wrote: > Hello Tomislav, > > Are these processors running in a multi-node cluster? Is FetchFile > downstream from a ListFile processor that is scheduled to run on all nodes > versus Primary Node only? Is FetchFile's Completion Strategy set to "Move > File" or "Delete File"? Typically, source processors should be scheduled > to run on the primary node, otherwise when reading from the same source > across multiple nodes, for example a shared network drive, each source > processor might pull the same data. In a situation like this, the same > file could be listed by each node, and the FetchFile processor on each node > may attempt to fetch the same file. > > If you set the source processor to run on Primary Node only, you can > load-balance the connection between the source processor and FetchFile to > distribute the load of fetching the files across the cluster. > > On Thu, Oct 3, 2019 at 2:32 AM Tomislav Novosel <to.novo...@gmail.com> > wrote: > >> Hi all, >> >> I'm getting errors from FetchFile and UnpackContent processors. >> I have pipeline where I fetch zip files as they come continuously on >> shared network drive >> with Minimum file age set to 30 sec to avoid fetching file before it is >> written to disk completely. >> >> Sometimes I get this error from FetchFile: >> >> FetchFile[id=c741187c-1172-1166-e752-1f79197a8029] Could not fetch file >> \\avl01\ATGRZ\TestFactory\02 Dep Service\01 >> Processdata\Backup\dfs_atfexport\MANA38\ANA_12_BPE7347\ANA_12_BPE7347_TDL_HL_1\measurement_file.atf.zip >> from file system for >> StandardFlowFileRecord[uuid=e7a5e3c4-0981-4ff3-85ea-91e41f0c3c0e,claim=,offset=0,name=PEI_BPE7347_TDLHL1new_826_20191001161312.atf.zip,size=0] >> because the existence of the file cannot be verified; routing to failure >> >> >> And from UnpackContent sometimes I get this error: >> >> >> UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d] Unable to unpack >> StandardFlowFileRecord[uuid=4a019d58-fe45-4276-a161-e46cd8b1667c,claim=StandardContentClaim >> [resourceClaim=StandardResourceClaim[id=1570052741201-5000, >> container=default, section=904], offset=1651, >> length=28417768],offset=0,name=measurement.atf.zip,size=28417768] due to >> IOException thrown from >> UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d]: >> java.io.IOException: Truncated ZIP file; routing to failure: >> >> org.apache.nifi.processor.exception.ProcessException: IOException thrown >> from UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d]: >> java.io.IOException: Truncated ZIP file >> >> >> After getting this error from UnpackContent I tried to fetch file again >> and to unpack it. It went well, without any errors. >> So what does this errors mean? I spoke to colleagues who are using this >> files on the source side and they said files are ok, not corrupted or >> something. >> >> Please help or give advice. >> >> Thanks in advance. >> Tom >> >> >> >> >> >> >> >> >> >> >>
