Hi Mark,
Thanks for your valuable suggestion. It worked a lot. Now I can understand,
there is no point in load balancing between FetchSFTP and CompressContent.
After making all the changes it worked but some of the flow files are stuck
between CompressContent and putHDFS
Hi,
I am using 2 nodes cluster.
nodes config Heap(max) 48gb & 64 core machine
Processor flow
ListSFTP--->FetchSFTP(all nodes with 10 threads)--->CompressContent(all
nodes,10 threads)-->PutHDFS
Queues shows it has 96gb in queue but when I do listing it shows no flow files.
Everything seems
Hello,
It would be helpful if you could upload a screenshot of your flow
somewhere and send a link.
Thanks,
Bryan
On Tue, Dec 3, 2019 at 6:06 AM nayan sharma wrote:
>
> Hi,
> I am using 2 nodes cluster.
> nodes config Heap(max) 48gb & 64 core machine
> Processor flow
>
Sounds like PartitionRecord by month.
On Tue, Dec 3, 2019 at 9:12 AM Joe Witt wrote:
>
> I read/replied too fast - if you mean that you want them together but sorted
> by date then it makes sense we'd offer a SortRecord processor. If you wanted
> to simply group them by month then
I read/replied too fast - if you mean that you want them together but
sorted by date then it makes sense we'd offer a SortRecord processor. If
you wanted to simply group them by month then PartitionRecord should do the
trick.
On Tue, Dec 3, 2019 at 8:10 AM Joe Witt wrote:
> Sounds like a
Nayan,
Looking at the screenshot, I can see two different connections there that are
load balanced. One of them holds the nearly 100 GB of data.
There are a handful of bugs related to load-balanced connections in 1.9.2 that
were addressed in 1.10.0. If you're relying on load-balanced
Sounds like a perfect use of PartitionRecord. And if you wanted larger
bundles of such things you could then follow it with MergeRecord correlated
on that same partitioned value.
Thanks
On Tue, Dec 3, 2019 at 8:09 AM Mike Thomsen wrote:
> We need to be able to split a record set by examining
We need to be able to split a record set by examining a date field and
sorting the messages by month into new record sets. The reason is that
they're going to be fed to an Elastic cluster that uses an index template
to build new indexes based on a date convention. We have a simple solution
for now
Agreed. Is a bit unclear whether you're looking to partition by the date field
or sort by the date field, or both. If you want to partition, then you'd use
PartitionRecord. If you want to sort, we do not have a SortRecord processor.
However, what we do have is QueryRecord, which can be used to
