Re: Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0
things like accessing specific file paths > as might be needed in flow configurations themselves (GetFile/PutFile > for example) could be tricky (but doable). > > The protocols used to source data matter a lot > With all this talk of back pressure keep in mind that how data gets > into NiFi becomes really critical in these clusters. If you use > protocols which do not afford fault tolerance and load balancing then > things are not great. So protocols which have queuing semantics or > feedback mechanisms or let NiFi as the consumer control things will > work out well. Some portions of JMS are good for this. Kafka is good > for this. NiFi's own site-to-site is good for this. > > The frankencluster testing is a valuable way to force and think > through interesting issues. Maybe the frankencluster as you have it > isn't realistic but it still exposes the concepts that need to be > thought through for cases that definitely are. > > Thanks > Joe > > On Tue, Sep 27, 2016 at 7:37 AM, Joe Skora <jsk...@gmail.com> wrote: > > The images just show what the text described, 13 files queued, EmptyQueue > > returns 0 of 13 removed, and ListQueue returns the queue has no > flowfiles. > > > > There were 13 files of 1k sitting in a queue between a SegmentContent and > > ControlRate. After I sent that email I had to stop/start the processors > a > > couple of times for other things and somewhere in the midst of that the > > queue cleared. > > > > > > > > On Mon, Sep 26, 2016 at 11:05 PM, Peter Wicks (pwicks) < > pwi...@micron.com> > > wrote: > > > >> Joe, > >> > >> I didn’t get the images (might just be my exchange server). How many > files > >> are in the queue? (exact count please) > >> > >> --Peter > >> > >> From: Joe Skora [mailto:jsk...@gmail.com] > >> Sent: Monday, September 26, 2016 8:20 PM > >> To: dev@nifi.apache.org > >> Subject: Questions about heterogeneous cluster and queue > >> problem/bug/oddity in 1.0.0 > >> > >> I have a 3 node test franken-cluster that I'm abusing for the sake of > >> learning. The systems run Ubuntu 15.04, OS X 10.11.6, and Windows 10 > and > >> though far comparable each has quad-core i7 between 2.5 and 3.5 GHz and > >> 16GB of RAM. Two have SSDs and the third has a 7200RPM SATA III drive. > >> > >> 1) Is there any reason mixing operating systems with the cluster would > be > >> a bad idea. Once configured it seems to run ok. > >> 2) Will performance disparities affect reliable ability or performance > >> within the cluster? > >> 3) Are there ways to configure disparate systems such that they can all > >> perform at peak? > >> > >> The bug or issues I have run into is a queue showing files that can't be > >> remove or listed. Screen shots attached below. I don't know if it's a > >> mixed-OS issues, something I did while torturing the systems (all stayed > >> up, this time), or just a weird anomaly. > >> > >> Regards, > >> Joe > >> > >> Trying to empty queue seen in background > >> [Inline image 1] > >> > >> but the flowfiles cannot be deleted. > >> [Inline image 2] > >> > >> But try to list them and it says there are no files in the queue? > >> [Inline image 3] > >> >
Re: Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0
oe, >> >> I didn’t get the images (might just be my exchange server). How many files >> are in the queue? (exact count please) >> >> --Peter >> >> From: Joe Skora [mailto:jsk...@gmail.com] >> Sent: Monday, September 26, 2016 8:20 PM >> To: dev@nifi.apache.org >> Subject: Questions about heterogeneous cluster and queue >> problem/bug/oddity in 1.0.0 >> >> I have a 3 node test franken-cluster that I'm abusing for the sake of >> learning. The systems run Ubuntu 15.04, OS X 10.11.6, and Windows 10 and >> though far comparable each has quad-core i7 between 2.5 and 3.5 GHz and >> 16GB of RAM. Two have SSDs and the third has a 7200RPM SATA III drive. >> >> 1) Is there any reason mixing operating systems with the cluster would be >> a bad idea. Once configured it seems to run ok. >> 2) Will performance disparities affect reliable ability or performance >> within the cluster? >> 3) Are there ways to configure disparate systems such that they can all >> perform at peak? >> >> The bug or issues I have run into is a queue showing files that can't be >> remove or listed. Screen shots attached below. I don't know if it's a >> mixed-OS issues, something I did while torturing the systems (all stayed >> up, this time), or just a weird anomaly. >> >> Regards, >> Joe >> >> Trying to empty queue seen in background >> [Inline image 1] >> >> but the flowfiles cannot be deleted. >> [Inline image 2] >> >> But try to list them and it says there are no files in the queue? >> [Inline image 3] >>
Re: Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0
The images just show what the text described, 13 files queued, EmptyQueue returns 0 of 13 removed, and ListQueue returns the queue has no flowfiles. There were 13 files of 1k sitting in a queue between a SegmentContent and ControlRate. After I sent that email I had to stop/start the processors a couple of times for other things and somewhere in the midst of that the queue cleared. On Mon, Sep 26, 2016 at 11:05 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > Joe, > > I didn’t get the images (might just be my exchange server). How many files > are in the queue? (exact count please) > > --Peter > > From: Joe Skora [mailto:jsk...@gmail.com] > Sent: Monday, September 26, 2016 8:20 PM > To: dev@nifi.apache.org > Subject: Questions about heterogeneous cluster and queue > problem/bug/oddity in 1.0.0 > > I have a 3 node test franken-cluster that I'm abusing for the sake of > learning. The systems run Ubuntu 15.04, OS X 10.11.6, and Windows 10 and > though far comparable each has quad-core i7 between 2.5 and 3.5 GHz and > 16GB of RAM. Two have SSDs and the third has a 7200RPM SATA III drive. > > 1) Is there any reason mixing operating systems with the cluster would be > a bad idea. Once configured it seems to run ok. > 2) Will performance disparities affect reliable ability or performance > within the cluster? > 3) Are there ways to configure disparate systems such that they can all > perform at peak? > > The bug or issues I have run into is a queue showing files that can't be > remove or listed. Screen shots attached below. I don't know if it's a > mixed-OS issues, something I did while torturing the systems (all stayed > up, this time), or just a weird anomaly. > > Regards, > Joe > > Trying to empty queue seen in background > [Inline image 1] > > but the flowfiles cannot be deleted. > [Inline image 2] > > But try to list them and it says there are no files in the queue? > [Inline image 3] >
RE: Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0
Joe, I didn’t get the images (might just be my exchange server). How many files are in the queue? (exact count please) --Peter From: Joe Skora [mailto:jsk...@gmail.com] Sent: Monday, September 26, 2016 8:20 PM To: dev@nifi.apache.org Subject: Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0 I have a 3 node test franken-cluster that I'm abusing for the sake of learning. The systems run Ubuntu 15.04, OS X 10.11.6, and Windows 10 and though far comparable each has quad-core i7 between 2.5 and 3.5 GHz and 16GB of RAM. Two have SSDs and the third has a 7200RPM SATA III drive. 1) Is there any reason mixing operating systems with the cluster would be a bad idea. Once configured it seems to run ok. 2) Will performance disparities affect reliable ability or performance within the cluster? 3) Are there ways to configure disparate systems such that they can all perform at peak? The bug or issues I have run into is a queue showing files that can't be remove or listed. Screen shots attached below. I don't know if it's a mixed-OS issues, something I did while torturing the systems (all stayed up, this time), or just a weird anomaly. Regards, Joe Trying to empty queue seen in background [Inline image 1] but the flowfiles cannot be deleted. [Inline image 2] But try to list them and it says there are no files in the queue? [Inline image 3]
Questions about heterogeneous cluster and queue problem/bug/oddity in 1.0.0
I have a 3 node test franken-cluster that I'm abusing for the sake of learning. The systems run Ubuntu 15.04, OS X 10.11.6, and Windows 10 and though far comparable each has quad-core i7 between 2.5 and 3.5 GHz and 16GB of RAM. Two have SSDs and the third has a 7200RPM SATA III drive. 1) Is there any reason mixing operating systems with the cluster would be a bad idea. Once configured it seems to run ok. 2) Will performance disparities affect reliable ability or performance within the cluster? 3) Are there ways to configure disparate systems such that they can all perform at peak? The bug or issues I have run into is a queue showing files that can't be remove or listed. Screen shots attached below. I don't know if it's a mixed-OS issues, something I did while torturing the systems (all stayed up, this time), or just a weird anomaly. Regards, Joe Trying to empty queue seen in background [image: Inline image 1] but the flowfiles cannot be deleted. [image: Inline image 2] But try to list them and it says there are no files in the queue? [image: Inline image 3]
