Re: Listing S3
any update on this? Was the ListS3 processor updated or are there plans to create a new processor? -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
ExecuteScript Concurrent Tasks
Hello, Concurrent tasks increased using ExecuteScript or InvokeScriptedProcessor with python/jyphon to update content has no increased throughput. If I copy the processor and run the 2 in parallel the amount of data processed does not increase. Any explanation for this? Is there a system-wide setting for how much cpu is available to the Jython engine? Would refactoring into groovy improve throughput or is it best to create a custom processor? thank you Noe
Copying an RPG
Hello, What effect does coping an RPG and using it in multiple locations (20 or more PGs)? Or is better to use one RPG that multiple data flows connect to. I also notice I am unable to disable the original RPG if a copy has been enabled. Thank you Noe >
ListenTCPRecord Max Connections
Hello, Recently moved from using ListenTCP to ListenTCPRecord, but it is rejecting connections. I keep increasing the max number, but the WARN stays. On cmd line 'netstat -punta | grep 10001 | wc -l' number stays the same around 70. In the processor, I have set the concurrency to 3 and max connections to 400 from 200 and see no difference. I did not have this problem when using ListenTCP Any ideas? Thank you Noe
Re: S2S to Specific Nodes in a Cluster
Data is on the same cluster running ver 1.9. As specific nodes can not be specified in load balance, is using TCP to move data to those specific nodes the best alternative? My use case is unique, but only a couple nodes of the 5 have an additional nic card where data needs to be sent. thank you On Tue, Feb 11, 2020 at 10:23 AM Joe Witt wrote: > Either way the answer is to use load balanced connections to single node > once data is on the desired cluster. You still cannot send to a ‘specific > node’ but you can to a single node. > > Consider the case where data was split up for various reasons but needs to > be all back together. Load balancing can handle that case beautifully. > Just note in 1.11.1 load balancing has a bug which impacts single node and > partition based load balancing. It will be fixed in a 1.11.2 release asap. > > thanks > > On Tue, Feb 11, 2020 at 7:17 AM Bryan Bende wrote: > >> Is it actually two separate clusters, or is it S2S from a cluster back >> to itself? >> >> If it's two separate clusters then I don't think there is way to >> restrict it to certain nodes. >> >> If it is S2S back to self, then a load balanced connection would be >> better and you can select "single node" as the option. >> >> On Tue, Feb 11, 2020 at 10:15 AM Noe Detore >> wrote: >> > >> > Hello, >> > >> > I have a 5 node cluster with all nodes receiving data. Using S2S is it >> possible to send that data to a specific node or nodes in the cluster? >> Otherwise, I am looking at using TCP, but are there better alternatives? >> > >> > Thank you >> > Noe >> >
S2S to Specific Nodes in a Cluster
Hello, I have a 5 node cluster with all nodes receiving data. Using S2S is it possible to send that data to a specific node or nodes in the cluster? Otherwise, I am looking at using TCP, but are there better alternatives? Thank you Noe
Re: CompressContent hadoop-snappy
In my use case, I am compressing data than storing data in S3. Unfortunately, hadoop-snappy is not able to uncompress snappy-java. So using snappy-java files in Hive is not possible. It would be nice to have the option to select hadoop-snappy from CompressContent and just add the native libs to the jvm similar to PutHdfs. I will also look into SnappyHadoopCompatibleOutputStream. I will make the effort to contribute back if I go this route. Thank you Noe On Tue, Nov 26, 2019 at 12:54 PM Bryan Bende wrote: > Not sure if this is relevant, but snappy-java has a specific > SnappyHadoopCompatibleOutputStream so CompressContent could offer a > third snappy option like "snappy-hadoop" which used that. > > Shawn is correct though that we wouldn't want to introduce Hadoop libs > into CompressContent. > > [1] > https://github.com/xerial/snappy-java/blob/73c67c70303e509be1642af5e302411d39434249/src/main/java/org/xerial/snappy/SnappyHadoopCompatibleOutputStream.java > > On Tue, Nov 26, 2019 at 11:51 AM Shawn Weeks > wrote: > > > > It uses snappy-java to get around the native class path issues that > would exist otherwise. What’s wrong with snappy-java? > > > > > > > > Thanks > > > > Shawn > > > > > > > > From: Noe Detore > > Reply-To: "users@nifi.apache.org" > > Date: Monday, November 25, 2019 at 2:16 PM > > To: "users@nifi.apache.org" > > Subject: CompressContent hadoop-snappy > > > > > > > > Hello > > > > > > > > CompressContent ver 1.9 uses snappy-java. Is there an easy way to change > it to hadoop-snappy? Or a custom processor needs to be created? > > > > > > > > thank you > > > > Noe >
CompressContent hadoop-snappy
Hello CompressContent ver 1.9 uses snappy-java. Is there an easy way to change it to hadoop-snappy? Or a custom processor needs to be created? thank you Noe
Re: Stateful Dataflow Moved to New Cluster
I may have gotten a little ahead of myself. Would I first use a template to move data flow or actual flow.xml.gz? Is it even possible to copy flow.xml.gz and what would be required? On Tue, Sep 17, 2019 at 1:04 PM Noe Detore wrote: > this is great! > > Thank you > Noe > > On Tue, Sep 17, 2019 at 11:37 AM Joe Witt wrote: > >> quick reply: There is a zookeeper state migrator utility in the toolkit I >> believe. That should be quite helpful. >> >> >> http://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#zookeeper_migrator >> >> Thanks >> >> On Tue, Sep 17, 2019 at 11:35 AM Noe Detore >> wrote: >> >>> Hello, >>> >>> I am currently using a stateful processor such as GetSplunk in an active >>> data flow. I want to move this data flow to a new Nifi cluster and preserve >>> the state of the processor. How can this be done? >>> >>> Thank you >>> Noe >>> >>
Re: Stateful Dataflow Moved to New Cluster
this is great! Thank you Noe On Tue, Sep 17, 2019 at 11:37 AM Joe Witt wrote: > quick reply: There is a zookeeper state migrator utility in the toolkit I > believe. That should be quite helpful. > > > http://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#zookeeper_migrator > > Thanks > > On Tue, Sep 17, 2019 at 11:35 AM Noe Detore > wrote: > >> Hello, >> >> I am currently using a stateful processor such as GetSplunk in an active >> data flow. I want to move this data flow to a new Nifi cluster and preserve >> the state of the processor. How can this be done? >> >> Thank you >> Noe >> >
Stateful Dataflow Moved to New Cluster
Hello, I am currently using a stateful processor such as GetSplunk in an active data flow. I want to move this data flow to a new Nifi cluster and preserve the state of the processor. How can this be done? Thank you Noe
Re: JVM Proxy in Bootstrap
Erik Sorry, this was for NIFI ver 1.5, but I see StandardProxyConfigurationService in 1.7. I am upgrading Nifi soon so I will hold off on satisfying this requirement until then. Thank you Noe On Tue, Aug 27, 2019 at 7:42 AM Erik Anderson wrote: > For NiFi I thought proxy was a controller you needed to configure then you > add that controller to, say your HTTP processors. > > Not sure why you would need to play with JVM proxy settings. > > Erik Anderson > Bloomberg > > Sent from my iPad > > On Aug 23, 2019, at 11:27 AM, Noe Detore wrote: > > Hello > > I have attempted to set authenticated proxy configuration on NIfi start-up > by setting properties in bootstrap with no success. Is it possible? > Currently getting 407 due to authentication errors. > The configs do not seem to be working > >java.arg.19=-Dhttp.proxyHost=proxy >java.arg.20=-Dhttp.proxyPort= > java.arg.23=-Dhttp.proxyUser=myuser > java.arg.24=-Dhttp.proxyPassword=mypassword > #Java version 8u111+ > java.arg.27=-Djdk.http.auth.tunneling.disabledSchemes= > java.arg.28=-Djdk.http.auth.proxying.disabledSchemes= > > Just want to double-check with the community before banging my head > against the wall for another day. > > Thank you > Noe > >
JVM Proxy in Bootstrap
Hello I have attempted to set authenticated proxy configuration on NIfi start-up by setting properties in bootstrap with no success. Is it possible? Currently getting 407 due to authentication errors. The configs do not seem to be working java.arg.19=-Dhttp.proxyHost=proxy java.arg.20=-Dhttp.proxyPort= java.arg.23=-Dhttp.proxyUser=myuser java.arg.24=-Dhttp.proxyPassword=mypassword #Java version 8u111+ java.arg.27=-Djdk.http.auth.tunneling.disabledSchemes= java.arg.28=-Djdk.http.auth.proxying.disabledSchemes= Just want to double-check with the community before banging my head against the wall for another day. Thank you Noe
Site to Site Compression
Hello, What is the best way to configure compression using site to site when sending data from one data center to another? I notice there is the ability to configure compression in a queue. What considerations need to be taken into account for different versions? DC1 Nifi 1.5 and DC2 Nifi 1.9. Thank you Noe
Nifi run.as
Hello, While using Nifi 1.5+ I want to update run.as on a linux box. Does this user require sudo. I have been finding conflicting information. If sudo is required what exact permissions are needed? Thank you N\oe
Processor Id As Attribute
Hello, Is there a way using expression language to get current processors id or something similar? I found a jira ticket related to https://issues.apache.org/jira/browse/NIFI-4284. Thank you Noe
Re: PutHiveStreaming TimelineClientImpl Exception
Shawn, That did the trick. I ended up removing all properties with reference to hook and adding hcatalog.hive.client.cache.disabled=true in the hive-site.xml file referenced by PutHiveStreaming in Hive Configuration Resources. Thank you Noe On Wed, Oct 31, 2018 at 10:47 AM Shawn Weeks wrote: > You have to either create a hive-site.xml just for NiFi without the hook > or your yarn-site.xml needs to be in the class path. Another parameter that > you might have to set to make Hive streaming less chatty is > hcatalog.hive.client.cache.disabled=true, > it was recomened by our vendor to get rid of some other error messages. > > > Thanks > > Shawn Weeks > ------ > *From:* Noe Detore > *Sent:* Wednesday, October 31, 2018 7:16:15 AM > *To:* users@nifi.apache.org > *Subject:* PutHiveStreaming TimelineClientImpl Exception > > Hello, > > Using NIFI 1.5 PutHiveStreaming processor I am seeing a lot of logs > > INFO [ATS Logger 0] o.a.h.y.c.api.impl.TimelineClientImpl Exception caught > by TimelineClientConnectionRetry, will try 1 more time(s). > Message: java.net.ConnectException: Connection refused > 2018-10-31 07:44:51,612 WARN [ATS Logger 0] > org.apache.hadoop.hive.ql.hooks.ATSHook Failed to create ATS domain > hive_6407e1d8-2d67-44af-bd0a-04288d6c587b > java.lang.RuntimeException: Failed to connect to timeline server. > Connection retries limit exceeded. The posted timeline event may be > missing... > > Data is getting into Hive, but this log is chatty. Any suggestions on how > to satisfy or remove this ATS requirement? > > Thank you > > >
PutHiveStreaming TimelineClientImpl Exception
Hello, Using NIFI 1.5 PutHiveStreaming processor I am seeing a lot of logs INFO [ATS Logger 0] o.a.h.y.c.api.impl.TimelineClientImpl Exception caught by TimelineClientConnectionRetry, will try 1 more time(s). Message: java.net.ConnectException: Connection refused 2018-10-31 07:44:51,612 WARN [ATS Logger 0] org.apache.hadoop.hive.ql.hooks.ATSHook Failed to create ATS domain hive_6407e1d8-2d67-44af-bd0a-04288d6c587b java.lang.RuntimeException: Failed to connect to timeline server. Connection retries limit exceeded. The posted timeline event may be missing... Data is getting into Hive, but this log is chatty. Any suggestions on how to satisfy or remove this ATS requirement? Thank you
Re: Escape Single Quote
that was it. Thank you
On Thu, Apr 5, 2018 at 11:12 AM, James Wing <jvw...@gmail.com> wrote:
> I believe you need something like
>
> ${my_forcast:replace("'","\\'")}
>
> Using two backslashes \\. The backslash is also used as an escape
> character in the expression language string, so you need two consecutive
> backslashes to make one literal backslash in the output.
>
> On Thu, Apr 5, 2018 at 7:31 AM, Noe Detore <ndet...@minerkasch.com> wrote:
>
>> Need to escape single quote for PutSql. 'Today's forecast ...' by a
>> backslash: 'Today\'s forecast ...', How can this be done using ReplaceText?
>> ${my_forcast:replace("'","\'")} does not work.
>>
>
>
Escape Single Quote
Need to escape single quote for PutSql. 'Today's forecast ...' by a
backslash: 'Today\'s forecast ...', How can this be done using ReplaceText?
${my_forcast:replace("'","\'")} does not work.
DBCPConnectionPool SqlServer and Kerberos
Hello Does anyone have experience or know if DBCPConnectionPool using SqlServer can be configured to authenticate with Kerberos? Thanks Noe
FlowFile UUID is not set
Hello, Having this error causing queues to back up. WARN [Timer-Driven Process Thread-48] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ReplaceText[id=85440232-015d-1000--02b489c2] due to uncaught Exception: java.lang.IllegalStateException: Cannot create Provenance Event Record because FlowFile UUID is not set running nifi 1.0.0 I assume this was caused by hard shutdown, but not sure. Similar issue seen posted here https://community.hortonworks.com/questions/84278/ javalangillegalstateexception-cannot-create-proven.html How can I correct this issue in order to move on? Update flowfile WAL? Please help Noe
