Ali,
You have a lot of nice resources to work with there. I'd recommend the
series of RAID-1 configuration personally provided you keep in mind this
means you can only lose a single disk for any one partition. As long as
they're being monitored and would be quickly replaced this in practice
work
Dear Joe,
Thank you very much. That was a really great explanation.
I investigated the Nifi architecture, and it seems that most of the
read/write operations for flow file repo and provenance repo are random.
However, for content repo most of the read/write operations are sequential.
Let's say cos
Ali,
I agree with your assumption. It would be great to test that out and
provide some numbers but intuitively I agree.
I could envision certain scatter/gather data flows that could challenge
that sequential access assumption but honestly with how awesome disk
caching is in Linux these days in t
Thank you very much.
I would be more than happy to provide some benchmark results after the
implementation.
Sincerely yours,
Ali
On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt wrote:
> Ali,
>
> I agree with your assumption. It would be great to test that out and
> provide some numbers but intuitive
Stephane,
Yes, you are correct that Apache NiFi uses swagger. However, we are only
using it for keeping the documentation in sync. We use a maven plugin that
inspects the swagger annotations and generates a swagger.json. The
swagger.json is generated to nifi-web-api/target/swagger-ui/swagger.json
Hi All,
I am learning NiFi as well as trying to deploy it in production for few
use cases. One of the use case is ETL and another use case is, using NiFi
as a backup solution, where it takes the data from one source and moves to
another database|file. Is anyone using NiFi for this purpose? Does Ni
Rai,
NiFi can certainly be used for some data replication scenarios and
quite often is. If you can treat the source like a continuous data
source then there is some way to keep state about what has been pulled
already, what has changed or needs yet to be pulled, and it can just
keep running then
Rai,
There are incremental data movement processors in NiFi depending on
your source/target. For example, if your sources are files, you can
use ListFile in combination with FetchFile, the former will keep track
of which files it has found thus far, so if you put new files into the
location (or up
Thanks Joe and Matt.
@Joe, based on your comment, I need to use NiFi as a producer which puts
the data on Kafka queue and then have NiFi consumer, which writes the data
back to the destination. Is my understanding correct?
@Matt, My use case is for the DynamoDB. I will look into whether
incrementa
You'd only need to do that if you have strict ordering requirements like
reading directly from a transaction log and replicating it. If yes I'd
skip nifi unless your also doing other cases with it.
Sounds like Matts path gets you going though so that might work out just
fine.
Thanks
Joe
On Oct
Hi All,
Is there any book for apache NiFi?
Also, does Hortonworks conducts training for NiFi?
Thanks
Rai
Hi Rai,
There are some excellent documents on the Apache NiFi site [1] to help you
learn. There is an Administrator Guide [2], a User Guide [3], a Developer Guide
[4], a NiFi In-Depth document [5], an Expression Language Guide [6] and
processor and component documentation [7] as well. Currently
Thanks Andy. Appreciate your guidance.
On Thu, Oct 13, 2016 at 10:39 AM, Andy LoPresto
wrote:
> Hi Rai,
>
> There are some excellent documents on the Apache NiFi site [1] to help you
> learn. There is an Administrator Guide [2], a User Guide [3], a Developer
> Guide [4], a NiFi In-Depth document
Hi All,
I have been trying to use get and load processor for the dynamodb and I am
almost there. I am able to run the get processor and I see, data is flowing
:)
But I see the following error in my nifi-app.log file:
2016-10-13 18:02:38,823 ERROR [Timer-Driven Process Thread-9]
o.a.n.p.aws.dynamod
Hi,
I am getting the following exception in nifi-0.6.1:
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer. Increase the fetch size, or decrease
the maximum message size the broker will allow.
What is the max size? How can I increase ma
Rai,
The GetDynamoDB processor requires a hash key value to look up an item in
the table. The default setting is an Expression Language statement that
reads the hash key value from a flowfile attribute,
dynamodb.item.hash.key.value. But this is not required. You can change it
to any attribute e
Thanks James. I am looking to iterate through the table so that it takes
hash key values one by one. Do I achieve it through the expression
language? if I write an script to do that, how do I pass it to my processor?
Thanks
Niraj
On Thu, Oct 13, 2016 at 1:42 PM, James Wing wrote:
> Rai,
>
> The
Igor,
Kafka consumer properties can be found here:
http://kafka.apache.org/documentation.html#consumerconfigs
GetKafka uses the old consumer so the consumer property is:
fetch.message.max.bytes
The default for that property is ~1M.
If possible, you should limit the replica.fetch.max.bytes on th
Hi,
Thanks it helps ! Good to know there is already a java client I could use.
Nonetheless I think it would be extremely nice to use the swagger codegen
project to generate additionally sdks, I don't mind creating a github
project of my own to maintain these.
I gave it a go and it gave me a bunch
Hi Stéphane,
This is a request that has grown popular recently. NiFi was not initially
designed with environment promotion in mind, so it is something we are
currently investigating and trying to address.
The development/QA/production environment promotion process [1] (sometimes
referred to as
Investigated some more, open a JIRA issue, closed it via
https://github.com/apache/nifi/pull/1135
On Fri, Oct 14, 2016 at 9:47 AM Stéphane Maarek
wrote:
> Hi,
>
> Thanks it helps ! Good to know there is already a java client I could use.
> Nonetheless I think it would be extremely nice to use th
Hi,
I have another question regarding the hardware recommendation. As far as I
found out, Nifi uses on-heap memory currently, and it will not try to load
the whole object in memory. From the garbage collection perspective, it is
not recommended to dedicate more than 8-10 GB to JVM heap space. In t
Thanks for submitting the PR Stephane! I see that Andy has already stated
that he's reviewing. Thanks Andy!
On Thu, Oct 13, 2016 at 7:42 PM, Stéphane Maarek
wrote:
> Investigated some more, open a JIRA issue, closed it via
> https://github.com/apache/nifi/pull/1135
>
> On Fri, Oct 14, 2016 at 9:
Stéphane asked a question on the PR but as it was already closed, I wanted to
reproduce it here for visibility and to see if other community members had
something to add:
Stéphane:
good stuff. Quick question, what do you think of NiFi automating the build and
release of API clients in various
Jeff,
Many thanks. I'm now more confident NiFi could be a good fit for us.
Marcio
On Wednesday, October 12, 2016 9:06 PM, Jeff wrote:
Hello Marcio,
You're asking on the right list!
Based on the scenario you described, I think NiFi would suit your needs. To
address your 3 major steps o
Great to hear, Marcio!
On Thu, Oct 13, 2016 at 9:26 PM Márcio Faria wrote:
> Jeff,
>
> Many thanks. I'm now more confident NiFi could be a good fit for us.
>
> Marcio
>
>
> On Wednesday, October 12, 2016 9:06 PM, Jeff wrote:
>
>
> Hello Marcio,
>
> You're asking on the right list!
>
> Based on
Thanks,
FYI, I've started to host my own swagger-codegen generated Java Client on
my github: https://github.com/simplesteph/nifi-api-client-java . Check out
the docs!
If you want to start playing and get a feel for it:
public static void main(String[] args) {
ApiClient apiClient = new ApiC
27 matches
Mail list logo
