Re: Re: 答复: Re: NiFi starts very slowly
Ok...validation may not be the issue. During the startup phase at 15 sec intervals please do a series of thread dumps. Please send those dump logs with timing info. Having 1600 groups that aren’t taking advantage of nifis ability to support directed graphs and expression language and flow file attributes will provide the least performant and highest overhead processing. Is it required to be this way in your current usage? Thanks On Thu, Dec 28, 2017 at 8:54 PMwrote: > Thanks for your help. > > Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the > work. > > There are about 10 processors in each group, so there will be 16000 > processors in total. > > I understand that the processor should be validate before it starts, but > could this validation be defered until the processor (or the group it > belongs to) is to start for the first time > instead of doing this when NiFi service starts up? > > > > 发件人: > Joe Witt > 收件人: > dev@nifi.apache.org > 日期: > 2017/12/29 09:20 > 主题: > Re: 答复: Re: NiFi starts very slowly > > > > hello > > the total number of processors is probably a factor. how many are there? > can you please list how many processors of each type there are? Any > custom > processors? > > creating 1600 disjoint flows is probably going against the grain of nifi. > cant the flows be designed to operate as a directed graph amd leverage > attributes? > > keep in mind to start each processor we call validate. this could be the > cause of the slowness. > > help us collect the above info and we can give pointers. if you could > share your flow.xml.gz that would be great. > > thanks > joe > > On Dec 28, 2017 8:13 PM, wrote: > > > Thanks for your reply. > > > > We are not using a cluster, our NiFi 1.3 runs in the single node mode. > > > > > > > > 发件人: > > "Mohammed Nadeem" > > 收件人: > > dev@nifi.apache.org > > 日期: > > 2017/12/28 20:11 > > 主题: > > Re: NiFi starts very slowly > > > > > > > > Hi Luby, > > > > If you are using a cluster based setup then Nifi is unstable due to > nodes > > getting disconnected by the cluster cordinator. Cluster coordinator > > disconnects nodes from the cluster when the nodes take a little longer > to > > respond ( beyond timeout) . The cluster coordinator timeout based on the > > settings in the nifi.properties file. > > > > We need to adjust the following properties > > > > *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) > > *nifi.cluster.node.read.timeout *= 5 sec ( by default) > > > > By Increasing the timeout ,from 5 sec to higher value would probably > solve > > a > > problem > > > > > > Thanks, > > Nadeem > > > > > > > > -- > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ > > > > > > > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对 > 外 > > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发 > 件 > > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 > > > > > > This email message may contain confidential and/or privileged > information. > > If you are not the intended recipient, please do not read, save, > forward, > > disclose or copy the contents of this email or open any file attached to > > this email. We will be grateful if you could advise the sender > immediately > > by replying this email, and delete this email and any attachment or > links > > to this email completely and immediately from your computer system. > > > > > > > > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外 > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件 > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 > > > This email message may contain confidential and/or privileged information. > If you are not the intended recipient, please do not read, save, forward, > disclose or copy the contents of this email or open any file attached to > this email. We will be grateful if you could advise the sender immediately > by replying this email, and delete this email and any attachment or links > to this email completely and immediately from your computer system. > > > >
答复: Re: 答复: Re: NiFi starts very slowly
Thanks for your help. Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the work. There are about 10 processors in each group, so there will be 16000 processors in total. I understand that the processor should be validate before it starts, but could this validation be defered until the processor (or the group it belongs to) is to start for the first time instead of doing this when NiFi service starts up? 发件人: Joe Witt收件人: dev@nifi.apache.org 日期: 2017/12/29 09:20 主题: Re: 答复: Re: NiFi starts very slowly hello the total number of processors is probably a factor. how many are there? can you please list how many processors of each type there are? Any custom processors? creating 1600 disjoint flows is probably going against the grain of nifi. cant the flows be designed to operate as a directed graph amd leverage attributes? keep in mind to start each processor we call validate. this could be the cause of the slowness. help us collect the above info and we can give pointers. if you could share your flow.xml.gz that would be great. thanks joe On Dec 28, 2017 8:13 PM, wrote: > Thanks for your reply. > > We are not using a cluster, our NiFi 1.3 runs in the single node mode. > > > > 发件人: > "Mohammed Nadeem" > 收件人: > dev@nifi.apache.org > 日期: > 2017/12/28 20:11 > 主题: > Re: NiFi starts very slowly > > > > Hi Luby, > > If you are using a cluster based setup then Nifi is unstable due to nodes > getting disconnected by the cluster cordinator. Cluster coordinator > disconnects nodes from the cluster when the nodes take a little longer to > respond ( beyond timeout) . The cluster coordinator timeout based on the > settings in the nifi.properties file. > > We need to adjust the following properties > > *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) > *nifi.cluster.node.read.timeout *= 5 sec ( by default) > > By Increasing the timeout ,from 5 sec to higher value would probably solve > a > problem > > > Thanks, > Nadeem > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对 外 > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发 件 > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 > > > This email message may contain confidential and/or privileged information. > If you are not the intended recipient, please do not read, save, forward, > disclose or copy the contents of this email or open any file attached to > this email. We will be grateful if you could advise the sender immediately > by replying this email, and delete this email and any attachment or links > to this email completely and immediately from your computer system. > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 This email message may contain confidential and/or privileged information. If you are not the intended recipient, please do not read, save, forward, disclose or copy the contents of this email or open any file attached to this email. We will be grateful if you could advise the sender immediately by replying this email, and delete this email and any attachment or links to this email completely and immediately from your computer system.
Re: 答复: Re: NiFi starts very slowly
hello the total number of processors is probably a factor. how many are there? can you please list how many processors of each type there are? Any custom processors? creating 1600 disjoint flows is probably going against the grain of nifi. cant the flows be designed to operate as a directed graph amd leverage attributes? keep in mind to start each processor we call validate. this could be the cause of the slowness. help us collect the above info and we can give pointers. if you could share your flow.xml.gz that would be great. thanks joe On Dec 28, 2017 8:13 PM,wrote: > Thanks for your reply. > > We are not using a cluster, our NiFi 1.3 runs in the single node mode. > > > > 发件人: > "Mohammed Nadeem" > 收件人: > dev@nifi.apache.org > 日期: > 2017/12/28 20:11 > 主题: > Re: NiFi starts very slowly > > > > Hi Luby, > > If you are using a cluster based setup then Nifi is unstable due to nodes > getting disconnected by the cluster cordinator. Cluster coordinator > disconnects nodes from the cluster when the nodes take a little longer to > respond ( beyond timeout) . The cluster coordinator timeout based on the > settings in the nifi.properties file. > > We need to adjust the following properties > > *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) > *nifi.cluster.node.read.timeout *= 5 sec ( by default) > > By Increasing the timeout ,from 5 sec to higher value would probably solve > a > problem > > > Thanks, > Nadeem > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外 > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件 > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 > > > This email message may contain confidential and/or privileged information. > If you are not the intended recipient, please do not read, save, forward, > disclose or copy the contents of this email or open any file attached to > this email. We will be grateful if you could advise the sender immediately > by replying this email, and delete this email and any attachment or links > to this email completely and immediately from your computer system. > > > >
答复: Re: NiFi starts very slowly
Thanks for your reply. We are not using a cluster, our NiFi 1.3 runs in the single node mode. 发件人: "Mohammed Nadeem"收件人: dev@nifi.apache.org 日期: 2017/12/28 20:11 主题: Re: NiFi starts very slowly Hi Luby, If you are using a cluster based setup then Nifi is unstable due to nodes getting disconnected by the cluster cordinator. Cluster coordinator disconnects nodes from the cluster when the nodes take a little longer to respond ( beyond timeout) . The cluster coordinator timeout based on the settings in the nifi.properties file. We need to adjust the following properties *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) *nifi.cluster.node.read.timeout *= 5 sec ( by default) By Increasing the timeout ,from 5 sec to higher value would probably solve a problem Thanks, Nadeem -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 This email message may contain confidential and/or privileged information. If you are not the intended recipient, please do not read, save, forward, disclose or copy the contents of this email or open any file attached to this email. We will be grateful if you could advise the sender immediately by replying this email, and delete this email and any attachment or links to this email completely and immediately from your computer system.
Re: [VOTE] Release Apache NiFi Registry 0.1.0
+1 (binding) Release this package as nifi-registry-0.1.0 Release verification process -pull source artifacts -verify sigs -verify commit id -verify commit history reflects latest changes (based off commit id/history) -verify source README -verify source L and that it is consistent with the work done in https://issues.apache.org/jira/browse/NIFIREG-66 -verify build w/contrib check -verify binary/assembly README -verify binary/assembly L and that it is consistent with work done in https://issues.apache.org/jira/browse/NIFIREG-66 -test registry on its own (manual/user testing) -review registry docs -setup certificate based secure reg setup/test - build nifi with registry integration PR - verify interaction with secure (certs based) nifi/secure registry Tests - create empty group - make local changes / view diffs / go to - make sure nifi cannot do anything with the registry when its cert lacks proxy auth - once proxy auth added all options available - verify can publish new versions/comments/ensure input is escaped - change versions on a live flow with and without enqueued data Observations - Having a default URL handler to show the 'you may have mistyped' type page from nifi would be nice - We should improve the logic of when components are started/stopped to make version changes. For position changes only components should not start/stop - There are a lot of very cool features to add from here but this is a pretty awesome first version with a range of excellent security options/polish. This is going to be a huge step forward for the community! Thanks Joe On Thu, Dec 28, 2017 at 1:09 PM, Bryan Bendewrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi Registry 0.1.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1115/ > > The Git tag is nifi-registry-0.1.0-RC1 > The Git commit ID is 81b99e7b04491eabb72ddf30754053ca12d0fcca > https://git-wip-us.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=81b99e7b04491eabb72ddf30754053ca12d0fcca > > Checksums of nifi-registry-0.1.0-source-release.zip: > MD5: 56244c3c296cdc9c3fcc6d22590b80d1 > SHA1: 6354e91f868f40d6656ec2467bde307260ad63ca > SHA256: 2c680e441e6c4bfa2381bf004e9b19a6a79401a6a83e04597d0a714a95efd301 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/bbende.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 65 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320920=12340217 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiRegistry0.1.0 > > The vote will be open for 96 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > > The please vote: > > [ ] +1 Release this package as nifi-registry-0.1.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Apache NiFi Registry 0.1.0 RC1 Release Helper Guide
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the Apache NiFi Registry release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven artifact repository # Pull down nifi-registry-0.1.0 source release artifacts for review: wget https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.1.0/nifi-registry-0.1.0-source-release.zip wget https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.1.0/nifi-registry-0.1.0-source-release.zip.asc wget https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.1.0/nifi-registry-0.1.0-source-release.zip.md5 wget https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.1.0/nifi-registry-0.1.0-source-release.zip.sha1 wget https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.1.0/nifi-registry-0.1.0-source-release.zip.sha256 # Verify the signature gpg --verify nifi-registry-0.1.0-source-release.zip.asc # Verify the hashes (md5, sha1, sha256) match the source and what was provided in the vote email thread md5sum nifi-registry-0.1.0-source-release.zip sha1sum nifi-registry-0.1.0-source-release.zip sha256sum nifi-registry-0.1.0-source-release.zip # Unzip nifi-registry-0.1.0-source-release.zip # Verify the build works including release audit tool (RAT) checks cd nifi-registry-0.1.0 mvn clean install -Pcontrib-check # Verify the contents contain a good README, NOTICE, and LICENSE. # Verify the git commit ID is correct # Verify the RC was branched off the correct git commit ID # Look at the resulting convenience binary as found in nifi-registry-assembly/target # Make sure the README, NOTICE, and LICENSE are present and correct # Run the resulting convenience binary and make sure it works as expected # Test integration between the registry and NiFi Start the registry ./bin/nifi-registry.sh start Create a bucket in the registry - Go to the registry UI at http://localhost:18080/nifi-registry - Click the tool icon in the top right corner - Click New Bucket from the bucket table - Enter a name and click create Get the NiFi PR which adds the support for integrating with the registry https://github.com/apache/nifi/pull/2219 >From that PR, edit the root pom and change: 0.0.1-SNAPSHOT to 0.1.0 Build the PR and start NiFi NOTE: You must have already built nifi-registry with "mvn clean install" in order to build this PR because it depends on snapshot JARs being in your local Maven repo. Tell NiFi about your local registry instance - Go the controller settings for NiFi from the top-right menu - Select the Registry Clients tab - Add a new Registry Client giving it a name and the url of http://localhost:18080 Create a process group and place it under version control - Right click on the PG and select the Version menu - Select Start Version Control - Choose the registry instance and bucket you want to use - Enter a name, description, and comment Go back to the registry and refresh the main page and you should see the versioned flow you just saved Import a new PG from a versioned flow - Drag on a new PG like normal - Instead of entering a name, click the Import link - Now choose the flow you saved before You should have a second identical PG now. # Send a response to the vote thread indicating a +1, 0, -1 based on your findings. Thank you for your time and effort to validate the release!
[VOTE] Release Apache NiFi Registry 0.1.0
Hello, I am pleased to be calling this vote for the source release of Apache NiFi Registry 0.1.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1115/ The Git tag is nifi-registry-0.1.0-RC1 The Git commit ID is 81b99e7b04491eabb72ddf30754053ca12d0fcca https://git-wip-us.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=81b99e7b04491eabb72ddf30754053ca12d0fcca Checksums of nifi-registry-0.1.0-source-release.zip: MD5: 56244c3c296cdc9c3fcc6d22590b80d1 SHA1: 6354e91f868f40d6656ec2467bde307260ad63ca SHA256: 2c680e441e6c4bfa2381bf004e9b19a6a79401a6a83e04597d0a714a95efd301 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/bbende.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 65 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320920=12340217 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiRegistry0.1.0 The vote will be open for 96 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. The please vote: [ ] +1 Release this package as nifi-registry-0.1.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: DBCP Connection Pooling using multiple OJDBC Drivers
Try to avoid registering more than one connector, the issue was obviously caused by the duplicate connections which by default oracle will reuse the existing registered connector. Register the driver at runtime will solve the issue. Ex: // Oracle 7 connection File chia = new File("/path/to/oracle7.jar"); URL[] cp = new URL[1]; cp[0] = chia.toURI().toURL(); URLClassLoader ora8loader = new URLClassLoader(cp, ClassLoader.getSystemClassLoader()); Class drvClass = ora8loader.loadClass("oracle.jdbc.driver.OracleDriver"); Driver ora8driver = (Driver)drvClass.newInstance(); Properties props = new Properties(); props.setProperty("user", "chiablahblah"); props.setProperty("password", "mybankaccountpwd"); Connection ora8conn = ora8driver.connect("jdbc:oracle:thin:@host:port:sid",props); / Repeat the same blah above for 11g and you should be good. /chia -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: DBCP Connection Pooling using multiple OJDBC Drivers
Try to avoid registering more than one connector, the issue was obviously caused by the duplicate connections which by default oracle will reuse the existing registered connector. Register the driver at runtime will solve the issue. Ex: // Oracle 7 connection File chia = new File("/path/to/oracle7.jar"); URL[] cp = new URL[1]; cp[0] = chia.toURI().toURL(); URLClassLoader ora8loader = new URLClassLoader(cp, ClassLoader.getSystemClassLoader()); Class drvClass = ora8loader.loadClass("oracle.jdbc.driver.OracleDriver"); Driver ora8driver = (Driver)drvClass.newInstance(); Properties props = new Properties(); props.setProperty("user", "chiablahblah"); props.setProperty("password", "mybankaccountpwd"); Connection ora8conn = ora8driver.connect("jdbc:oracle:thin:@host:port:sid",props); / Repeat the same blah above for 11g and you should be good. /chia -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Compile avro to RecordSchema
Thanks! On Wed, Dec 27, 2017 at 8:10 PM, Koji Kawamurawrote: > Hi Mike, > > You might already have found it, but AvroTypeUtil.createSchema is > probably what you are looking for. > https://github.com/apache/nifi/blob/master/nifi-nar- > bundles/nifi-extension-utils/nifi-record-utils/nifi-avro- > record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L341 > > Thanks, > Koji > > On Wed, Dec 27, 2017 at 9:15 PM, Mike Thomsen > wrote: > > Is there an API for compiling an avro schema to a RecordSchema object, or > > is it more involved than that? > > > > Thanks, > > > > Mike >
Re: DBCP Connection Pooling using multiple OJDBC Drivers
Thanks Koji. But giving a directory of multiple drivers is throwing an error "can't load database driver " and i verified that with dbcp code and it specifically checks for ".jar extension". final ClassLoader classLoader = ClassLoaderUtils.getCustomClassLoader( locationString, this.getClass().getClassLoader(), (dir, name) -> name != null && name.endsWith(".jar") ); Your alternative approach is what the problem i wrote earlier creating multiple instances of dbcp service and making each point to different version of drivers. The issue was below error for ojdbc7.jar ( oracle 11g).. *java.sql.SQLException: Object does not wrap anything with requested interface* at oracle.jdbc.driver.OracleConnection.unwrap(OracleConnection.java:268) at org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553) at org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553) It is not able to unwrap this connection to oracle.jdbc.OracleConnection ( *because im using createOracleArray method of this OracleConnection.class* ). For this i added dependent ojdbc7.jar into my maven build pom.xml . Still I'm facing this issue Though it was working with junit testcases where i used dbcp as connection service. What could be the reason.. Please help -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: NiFi starts very slowly
Hi Luby, If you are using a cluster based setup then Nifi is unstable due to nodes getting disconnected by the cluster cordinator. Cluster coordinator disconnects nodes from the cluster when the nodes take a little longer to respond ( beyond timeout) . The cluster coordinator timeout based on the settings in the nifi.properties file. We need to adjust the following properties *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) *nifi.cluster.node.read.timeout *= 5 sec ( by default) By Increasing the timeout ,from 5 sec to higher value would probably solve a problem Thanks, Nadeem -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
NiFi starts very slowly
Hi, All, We created about 1600+ processor groups based on a NiFi template, but after creating those groups, we found tht NiFi service start very slowly, it takes about 40min ~ 1 hour to start. We are using NiFi 1.3. Does anyone know how to speed it up? Thanks Boying 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。 This email message may contain confidential and/or privileged information. If you are not the intended recipient, please do not read, save, forward, disclose or copy the contents of this email or open any file attached to this email. We will be grateful if you could advise the sender immediately by replying this email, and delete this email and any attachment or links to this email completely and immediately from your computer system.