Re: Question on ingesting HDFS batches

[Sudhindra Tirupati Nagaraj]

Thanks a lot Joe for answering my query! 

Sudhindra.

On 7/30/18, 3:47 PM, "Joe Witt"  wrote:

Sudhindra

The current ListFile processor scans through the configured directory
including any subdirectories and looks for files.  It does this by
generating a listing, comparing it to what it has seen already
(largely based on mod time) then sending out resulting listings.
These can be sent to a FetchFile process which pulls the files.

We do not offer a facility to look for the presence of a given special
'success' file.  We could, and probably at this point should since it
is a common ask, have a JIRA to add a filter to only select files in a
folder if we see a file meeting a certain name such as 'success'.

Thanks
Joe

On Mon, Jul 30, 2018 at 6:34 PM, Sudhindra Tirupati Nagaraj
 wrote:
> Hi,
>
>
>
> We just came across NIFI as a possible option for backing up our data lake
> periodically into S3. We have our pipelines that dump batches of data at
> some granularity. For example, our one-minute dumps are of the form
> “201807210617”, “201807210618”, “201807210619” etc. We are looking for a
> simple configuration based solution that reads these incoming batches
> periodically and creates a workflow for backing these up. Also, these
> batches have a “success” marker inside them that indicates that the 
batches
> are full and ready to be backed up. We came across the ListHDFS processor
> that can do this, without duplication, but we are not sure if it has the
> ability to only copy batches that have a particular state (that is, like
> having a success marker in them). We are not sure if it also works on
> “folders” and not files directly.
>
>
>
> Can I get some recommendations on whether NIFI can be used at for such a
> ingestion use-case/alternative? Thank you.
>
>
>
> Kind Regards,
>
> Sudhindra.

Re: Question on ingesting HDFS batches

[Joe Witt]

Sudhindra

The current ListFile processor scans through the configured directory
including any subdirectories and looks for files.  It does this by
generating a listing, comparing it to what it has seen already
(largely based on mod time) then sending out resulting listings.
These can be sent to a FetchFile process which pulls the files.

We do not offer a facility to look for the presence of a given special
'success' file.  We could, and probably at this point should since it
is a common ask, have a JIRA to add a filter to only select files in a
folder if we see a file meeting a certain name such as 'success'.

Thanks
Joe

On Mon, Jul 30, 2018 at 6:34 PM, Sudhindra Tirupati Nagaraj
 wrote:
> Hi,
>
>
>
> We just came across NIFI as a possible option for backing up our data lake
> periodically into S3. We have our pipelines that dump batches of data at
> some granularity. For example, our one-minute dumps are of the form
> “201807210617”, “201807210618”, “201807210619” etc. We are looking for a
> simple configuration based solution that reads these incoming batches
> periodically and creates a workflow for backing these up. Also, these
> batches have a “success” marker inside them that indicates that the batches
> are full and ready to be backed up. We came across the ListHDFS processor
> that can do this, without duplication, but we are not sure if it has the
> ability to only copy batches that have a particular state (that is, like
> having a success marker in them). We are not sure if it also works on
> “folders” and not files directly.
>
>
>
> Can I get some recommendations on whether NIFI can be used at for such a
> ingestion use-case/alternative? Thank you.
>
>
>
> Kind Regards,
>
> Sudhindra.

Re: Separating custom service API from its implementation

[Tim Dean]

Thanks Bryan - That makes sense

-Tim

> On Jul 30, 2018, at 2:45 PM, Bryan Bende  wrote:
> 
> Tim,
> 
> In the case where it is your own custom service api and service impl,
> and you know you are never going to have another implementation of the
> API, then it doesn't really matter and having them all in one NAR will
> work.
> 
> The issue is that by bundling the implementation with the API, now
> someone else who wants to provide a different implementation is forced
> to inherit the classpath of your other implementation.
> 
> By separating them, each implementation only depends on the API and
> doesn't need to worry about each other's specific implementation
> dependencies.
> 
> Thanks,
> 
> Bryan
> 
> 
> On Mon, Jul 30, 2018 at 2:35 PM, Tim Dean  wrote:
>> We’ve got a handful of custom NiFi controller services. The documentation
>> describes how to properly separate the API of a custom service from its
>> implementation using different NAR files. When we don’t do that, we see a
>> message that says the following:
>> 
>> org.apache.nifi.nar.ExtensionManager Controller Service
>> x.x.x.x.MyControllerServiceApi is bundled with its supporting APIs
>> x.x.x.x.MyControllerServiceImpl. The service APIs should not be bundled with
>> the implementations.
>> 
>> 
>> This warning makes sense, and I know how to fix it. I am trying to
>> understand if fixing it is really worth the effort. This seems to work for
>> us as is, even though it is clearly wrong according to the NiFi
>> documentation. I assume that this approach will fail under some
>> circumstances, but I don’t know when or how it will fail.
>> 
>> Can anyone provide any background on the downsides of leaving our controller
>> service implementations in the same NAR file as their APIs?
>> 
>> Thanks

Question on ingesting HDFS batches

[Sudhindra Tirupati Nagaraj]

Hi,

 

We just came across NIFI as a possible option for backing up our data lake 
periodically into S3. We have our pipelines that dump batches of data at some 
granularity. For example, our one-minute dumps are of the form “201807210617”, 
“201807210618”, “201807210619” etc. We are looking for a simple configuration 
based solution that reads these incoming batches periodically and creates a 
workflow for backing these up. Also, these batches have a “success” marker 
inside them that indicates that the batches are full and ready to be backed up. 
We came across the ListHDFS processor that can do this, without duplication, 
but we are not sure if it has the ability to only copy batches that have a 
particular state (that is, like having a success marker in them). We are not 
sure if it also works on “folders” and not files directly. 

 

Can I get some recommendations on whether NIFI can be used at for such a 
ingestion use-case/alternative? Thank you.

 

Kind Regards,

Sudhindra. 

Re: Dynamically loading JARs and using them with a controller service

[Michael Moser]

Hey Mike,

As long as it's a controller service PropertyDescriptor that uses
dynamicallyModifiesClasspath, check out the JMSConnectionFactoryProvider in
the nifi-jms-bundle.

-- Mike


On Sat, Jul 28, 2018 at 8:52 AM Mike Thomsen  wrote:

> Is there a good example somewhere that shows how to use
> dynamicallyModifiesClasspath on the PropertyDescriptor and use it add new
> JARs that are available to the controller service?
>
> Thanks,
>
> Mike
>

Re: Separating custom service API from its implementation

[Bryan Bende]

Tim,

In the case where it is your own custom service api and service impl,
and you know you are never going to have another implementation of the
API, then it doesn't really matter and having them all in one NAR will
work.

The issue is that by bundling the implementation with the API, now
someone else who wants to provide a different implementation is forced
to inherit the classpath of your other implementation.

By separating them, each implementation only depends on the API and
doesn't need to worry about each other's specific implementation
dependencies.

Thanks,

Bryan


On Mon, Jul 30, 2018 at 2:35 PM, Tim Dean  wrote:
> We’ve got a handful of custom NiFi controller services. The documentation
> describes how to properly separate the API of a custom service from its
> implementation using different NAR files. When we don’t do that, we see a
> message that says the following:
>
> org.apache.nifi.nar.ExtensionManager Controller Service
> x.x.x.x.MyControllerServiceApi is bundled with its supporting APIs
> x.x.x.x.MyControllerServiceImpl. The service APIs should not be bundled with
> the implementations.
>
>
> This warning makes sense, and I know how to fix it. I am trying to
> understand if fixing it is really worth the effort. This seems to work for
> us as is, even though it is clearly wrong according to the NiFi
> documentation. I assume that this approach will fail under some
> circumstances, but I don’t know when or how it will fail.
>
> Can anyone provide any background on the downsides of leaving our controller
> service implementations in the same NAR file as their APIs?
>
> Thanks

Separating custom service API from its implementation

[Tim Dean]

We’ve got a handful of custom NiFi controller services. The documentation 
describes how to properly separate the API of a custom service from its 
implementation using different NAR files. When we don’t do that, we see a 
message that says the following:

org.apache.nifi.nar.ExtensionManager Controller Service 
x.x.x.x.MyControllerServiceApi is bundled with its supporting APIs 
x.x.x.x.MyControllerServiceImpl. The service APIs should not be bundled with 
the implementations.

This warning makes sense, and I know how to fix it. I am trying to understand 
if fixing it is really worth the effort. This seems to work for us as is, even 
though it is clearly wrong according to the NiFi documentation. I assume that 
this approach will fail under some circumstances, but I don’t know when or how 
it will fail.

Can anyone provide any background on the downsides of leaving our controller 
service implementations in the same NAR file as their APIs?

Thanks

Re: PutHiveStreaming + Hive 3.0

[Matt Burgess]

Mike,

For your Maven command in #1, those hive.* properties are for the Hive
1 NAR (PutHiveStreaming, e.g.), you can override the hive3.* versions
of them if you want to set a particular version (they default to
Apache Hive 3.0.0).

For #2, I didn't see anything in the new Hive Streaming API that
accepts just username and password, rather it tries to get the
information from the UserGroupInformation which I believe needs
Kerberos. If Kerberos is not enabled, I believe it uses the
"user.name" system property for username, but I still don't see where
authorization is happening (even behind the scenes). I assume it has
to do with being authenticated and passing the username to Ranger, but
I'm not sure. Having said that, there may be some properties you could
set in hive-site.xml to specify a username/password. I'll ask around
and follow up if I find out anything.

Regards,
Matt

On Mon, Jul 30, 2018 at 10:40 AM Michal Tomaszewski
 wrote:
>
> Hi Matt,
>
> #1
> Thank you very much!!!
> PutHive3Streaming works after compilation :)
>
> BTW: compilation with options
> mvn -T C2.0 clean install -Phortonworks -Dhive.version=3.1.0.3.0.0.0-1634 
> -Dhive.hadoop.version=3.1.0.3.0.0.0-1634 -Dhadoop.version=3.1.0.3.0.0.0-1634 
> -Pinclude-hive3 -DskipTests -e
> throws errors like
> [ERROR] symbol:   class MemoryManager
> [ERROR] location: class NiFiOrcUtils
> [ERROR] 
> /home/compiler/nifi/nifi-master/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java:[478,32]
>  error: cannot find symbol
>
> so I compiled using:
> mvn -T C2.0 clean install -Phortonworks -Pinclude-hive3 -DskipTests -e
> I hope its proper command.
>
> #2
> Please advice how to make PutHive3Streaming use specified username and 
> password for authorization when contacting with hive?
> Currently all requests are made as anonymous.
> I have ranger for authorization, but cluster is not kerberised.
>
> In HiveConnectionPool (e.g. for HiveQL) there is a place for username and 
> password for query that works as required. In PutHive3Streaming there is no 
> place to put credentials.
>
> Regards,
> Mike
>
> > -Original Message-
> > From: Matt Burgess 
> > Sent: Monday, July 30, 2018 2:45 PM
> > To: users@nifi.apache.org
> > Subject: Re: PutHiveStreaming + Hive 3.0
> >
> > Mike,
> >
> > That error usually indicates a Thrift version mismatch, which in this case 
> > is
> > pretty much expected since PutHiveStreaming uses version 1.2.1 and HDP 3.0
> > uses 3.0.0+. As of NiFi 1.7.0 you can add the "-Pinclude-hive3" profile in 
> > your
> > Maven build and it will add a full set of Hive 3-compatible processors to 
> > the
> > assembly, or you can just go into nifi-nar-bundles/nifi-hive-bundle and 
> > build
> > from there, there will be a nifi-hive3-nar created in 
> > nifi-hive3-nar/target. The
> > processors include SelectHive3QL, PutHive3QL, PutHive3Streaming, and
> > PutORC, as well as a Hive3ConnectionPool for the JDBC processors
> > (Select/Put).
> >
> > Regards,
> > Matt
> >
> > On Mon, Jul 30, 2018 at 8:18 AM Michal Tomaszewski
> >  wrote:
> > >
> > > Hi Pierre,
> > > nifi-app.log enclosed.
> > >
> > > Regards,
> > >Mike
> > >
> > >
> > > From: Pierre Villard 
> > > Sent: Monday, July 30, 2018 11:40 AM
> > > To: users@nifi.apache.org
> > > Subject: Re: PutHiveStreaming + Hive 3.0
> > >
> > > Hi Mike,
> > >
> > > By any chance, could you share the full stack trace from nifi-app.log?
> > >
> > > Thanks,
> > > Pierre
> > >
> > > 2018-07-30 11:19 GMT+02:00 Michal Tomaszewski
> > :
> > > Hello,
> > > Is PutHiveStreaming processor working with Hive 3.0?
> > > We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and
> > got errors:
> > > Failed connecting to Hive endpoint table: hivetest3 at 
> > > thrift://servername.
> > > NiFi has access to all site.xml's. Configuration is exactly the same as 
> > > with
> > previous Hive server, that worked properly.
> > >
> > > Regards,
> > > Mike
>  Uwaga: Treść niniejszej wiadomości 
> może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej 
> wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem 
> upoważnionym do jej przekazania adresatowi, informujemy że wszelkie 
> rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest 
> zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie 
> odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. 
>  Note: The information contained in this 
> message may be privileged and confidential and protected from disclosure. If 
> the reader of this message is not the intended recipient, or an employee or 
> agent responsible for delivering this message to the intended recipient, you 
> are hereby notified that any dissemination, distribution or copying of this 
> communication is strictly 

RE: PutHiveStreaming + Hive 3.0

[Michal Tomaszewski]

Hi Matt,

#1
Thank you very much!!!
PutHive3Streaming works after compilation :)

BTW: compilation with options
mvn -T C2.0 clean install -Phortonworks -Dhive.version=3.1.0.3.0.0.0-1634 
-Dhive.hadoop.version=3.1.0.3.0.0.0-1634 -Dhadoop.version=3.1.0.3.0.0.0-1634 
-Pinclude-hive3 -DskipTests -e
throws errors like
[ERROR] symbol:   class MemoryManager
[ERROR] location: class NiFiOrcUtils
[ERROR] 
/home/compiler/nifi/nifi-master/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java:[478,32]
 error: cannot find symbol

so I compiled using:
mvn -T C2.0 clean install -Phortonworks -Pinclude-hive3 -DskipTests -e
I hope its proper command.

#2
Please advice how to make PutHive3Streaming use specified username and password 
for authorization when contacting with hive?
Currently all requests are made as anonymous.
I have ranger for authorization, but cluster is not kerberised.

In HiveConnectionPool (e.g. for HiveQL) there is a place for username and 
password for query that works as required. In PutHive3Streaming there is no 
place to put credentials.

Regards,
Mike

> -Original Message-
> From: Matt Burgess 
> Sent: Monday, July 30, 2018 2:45 PM
> To: users@nifi.apache.org
> Subject: Re: PutHiveStreaming + Hive 3.0
>
> Mike,
>
> That error usually indicates a Thrift version mismatch, which in this case is
> pretty much expected since PutHiveStreaming uses version 1.2.1 and HDP 3.0
> uses 3.0.0+. As of NiFi 1.7.0 you can add the "-Pinclude-hive3" profile in 
> your
> Maven build and it will add a full set of Hive 3-compatible processors to the
> assembly, or you can just go into nifi-nar-bundles/nifi-hive-bundle and build
> from there, there will be a nifi-hive3-nar created in nifi-hive3-nar/target. 
> The
> processors include SelectHive3QL, PutHive3QL, PutHive3Streaming, and
> PutORC, as well as a Hive3ConnectionPool for the JDBC processors
> (Select/Put).
>
> Regards,
> Matt
>
> On Mon, Jul 30, 2018 at 8:18 AM Michal Tomaszewski
>  wrote:
> >
> > Hi Pierre,
> > nifi-app.log enclosed.
> >
> > Regards,
> >Mike
> >
> >
> > From: Pierre Villard 
> > Sent: Monday, July 30, 2018 11:40 AM
> > To: users@nifi.apache.org
> > Subject: Re: PutHiveStreaming + Hive 3.0
> >
> > Hi Mike,
> >
> > By any chance, could you share the full stack trace from nifi-app.log?
> >
> > Thanks,
> > Pierre
> >
> > 2018-07-30 11:19 GMT+02:00 Michal Tomaszewski
> :
> > Hello,
> > Is PutHiveStreaming processor working with Hive 3.0?
> > We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and
> got errors:
> > Failed connecting to Hive endpoint table: hivetest3 at thrift://servername.
> > NiFi has access to all site.xml's. Configuration is exactly the same as with
> previous Hive server, that worked properly.
> >
> > Regards,
> > Mike
 Uwaga: Treść niniejszej wiadomości 
może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości 
nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym 
do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, 
rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli 
otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a 
samą wiadomość usunąć z komputera. Dziękujemy.  
Note: The information contained in this message may be privileged and 
confidential and protected from disclosure. If the reader of this message is 
not the intended recipient, or an employee or agent responsible for delivering 
this message to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited.If you have received this communication in error, please notify the 
sender immediately by replying to the message and deleting it from your 
computer. Thank you. 

Re: nifi-hive-nar-1.7.1.nar will not load

[Matt Burgess]

Yep could do that too, the argument could just be something like:

java.arg.tmp=-Djava.io.tmpdir=/path/to/tmpdir

Regards,
Matt
On Fri, Jul 27, 2018 at 8:00 AM Otto Fowler  wrote:
>
> Why not have the java tmp dir be configurable and make sure this doesn’t 
> happen for any other possible nar?
> It is a more generic solution.
>
>
> On July 26, 2018 at 17:23:31, Matt Burgess (mattyb...@apache.org) wrote:
>
> This has been biting a few users lately, not sure when it changed
> exactly, but the Hive NAR uses a version of Snappy that tries to
> extract the native Snappy library into a location pointed to by the
> "java.io.tmpdir" variable, which IIRC is /tmp/. The /tmp
> directory sometimes has a noexec restriction on it, and/or the OS user
> running NiFi does not have permissions to read/write/execute that
> directory. I haven't tried this workaround but I believe it has worked
> for other folks: Add a line under your other "java.args.X" lines in
> bootstrap.conf such as:
>
> java.arg.snappy=-Dorg.xerial.snappy.tempdir=/path/to/nifi/lib/
>
> Where the directory is the full path to NiFi's lib/ directory. This
> will cause the native Snappy library to be extracted there, but since
> the Snappy Java class is the only one attempting to load the library,
> it shouldn't cause any issues by being there.
>
> Have other folks run into this? I wonder if we should just add this
> argument to bootstrap.conf to avoid any potential issues, but of
> course we'd want to make sure that it doesn't introduce any issues
> either.
>
> Regards,
> Matt
> On Thu, Jul 26, 2018 at 2:49 PM geoff.craig  wrote:
> >
> > Here is the error:
> >
> > 2018-07-26 18:48:49,013 ERROR [main] org.apache.nifi.NiFi Failure to launch
> > NiFi due to java.util.ServiceConfigurationError:
> > org.apache.nifi.processor.Processor: Provider
> > org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated
> > java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor:
> > Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be
> > instantiated
> > at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> > at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> > at
> > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> > at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> > at
> > org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
> > at
> > org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:832)
> > at org.apache.nifi.NiFi.(NiFi.java:157)
> > at org.apache.nifi.NiFi.(NiFi.java:71)
> > at org.apache.nifi.NiFi.main(NiFi.java:292)
> > Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY]
> > null
> > at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)
> > at org.xerial.snappy.Snappy.(Snappy.java:48)
> > at
> > org.apache.nifi.processors.hive.PutHiveStreaming.(PutHiveStreaming.java:152)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > at java.lang.Class.newInstance(Class.java:442)
> > at
> > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> > ... 8 common frames omitted
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutHiveStreaming + Hive 3.0

[Matt Burgess]

Mike,

That error usually indicates a Thrift version mismatch, which in this
case is pretty much expected since PutHiveStreaming uses version 1.2.1
and HDP 3.0 uses 3.0.0+. As of NiFi 1.7.0 you can add the
"-Pinclude-hive3" profile in your Maven build and it will add a full
set of Hive 3-compatible processors to the assembly, or you can just
go into nifi-nar-bundles/nifi-hive-bundle and build from there, there
will be a nifi-hive3-nar created in nifi-hive3-nar/target. The
processors include SelectHive3QL, PutHive3QL, PutHive3Streaming, and
PutORC, as well as a Hive3ConnectionPool for the JDBC processors
(Select/Put).

Regards,
Matt

On Mon, Jul 30, 2018 at 8:18 AM Michal Tomaszewski
 wrote:
>
> Hi Pierre,
> nifi-app.log enclosed.
>
> Regards,
>Mike
>
>
> From: Pierre Villard 
> Sent: Monday, July 30, 2018 11:40 AM
> To: users@nifi.apache.org
> Subject: Re: PutHiveStreaming + Hive 3.0
>
> Hi Mike,
>
> By any chance, could you share the full stack trace from nifi-app.log?
>
> Thanks,
> Pierre
>
> 2018-07-30 11:19 GMT+02:00 Michal Tomaszewski 
> :
> Hello,
> Is PutHiveStreaming processor working with Hive 3.0?
> We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and got 
> errors:
> Failed connecting to Hive endpoint table: hivetest3 at thrift://servername.
> NiFi has access to all site.xml's. Configuration is exactly the same as with 
> previous Hive server, that worked properly.
>
> Regards,
> Mike
>
>
>  Uwaga: Treść niniejszej wiadomości 
> może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej 
> wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem 
> upoważnionym do jej przekazania adresatowi, informujemy że wszelkie 
> rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest 
> zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie 
> odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. 
>  Note: The information contained in this 
> message may be privileged and confidential and protected from disclosure. If 
> the reader of this message is not the intended recipient, or an employee or 
> agent responsible for delivering this message to the intended recipient, you 
> are hereby notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited.If you have received this communication 
> in error, please notify the sender immediately by replying to the message and 
> deleting it from your computer. Thank you. 
>
>  Uwaga: Treść niniejszej wiadomości 
> może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej 
> wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem 
> upoważnionym do jej przekazania adresatowi, informujemy że wszelkie 
> rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest 
> zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie 
> odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. 
>  Note: The information contained in this 
> message may be privileged and confidential and protected from disclosure. If 
> the reader of this message is not the intended recipient, or an employee or 
> agent responsible for delivering this message to the intended recipient, you 
> are hereby notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited.If you have received this communication 
> in error, please notify the sender immediately by replying to the message and 
> deleting it from your computer. Thank you. 

RE: PutHiveStreaming + Hive 3.0

[Michal Tomaszewski]

Hi,
Of course.
I can even make a separate nifi server having only Puthivestreaming processor 
to narrow logs only to those important.
Is it ok for you?
Would you like me to switch on any additional debugging or anything?

Regards,
Mike

From: Pierre Villard 
Sent: Monday, July 30, 2018 11:40 AM
To: users@nifi.apache.org
Subject: Re: PutHiveStreaming + Hive 3.0

Hi Mike,

By any chance, could you share the full stack trace from nifi-app.log?

Thanks,
Pierre

2018-07-30 11:19 GMT+02:00 Michal Tomaszewski 
:
Hello,
Is PutHiveStreaming processor working with Hive 3.0?
We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and got 
errors:
Failed connecting to Hive endpoint table: hivetest3 at thrift://servername.
NiFi has access to all site.xml's. Configuration is exactly the same as with 
previous Hive server, that worked properly.

Regards,
Mike


 Uwaga: Treść niniejszej wiadomości 
może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości 
nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym 
do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, 
rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli 
otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a 
samą wiadomość usunąć z komputera. Dziękujemy.  
Note: The information contained in this message may be privileged and 
confidential and protected from disclosure. If the reader of this message is 
not the intended recipient, or an employee or agent responsible for delivering 
this message to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited.If you have received this communication in error, please notify the 
sender immediately by replying to the message and deleting it from your 
computer. Thank you. 

Re: PutHiveStreaming + Hive 3.0

[Pierre Villard]

Hi Mike,

By any chance, could you share the full stack trace from nifi-app.log?

Thanks,
Pierre

2018-07-30 11:19 GMT+02:00 Michal Tomaszewski :

> Hello,
> Is PutHiveStreaming processor working with Hive 3.0?
> We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and
> got errors:
> Failed connecting to Hive endpoint table: hivetest3 at thrift://servername.
> NiFi has access to all site.xml's. Configuration is exactly the same as
> with previous Hive server, that worked properly.
>
> Regards,
> Mike
>
>
>  Uwaga: Treść niniejszej
> wiadomości może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik
> tej wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub
> pośrednikiem upoważnionym do jej przekazania adresatowi, informujemy że
> wszelkie rozprowadzanie, rozpowszechnianie lub powielanie niniejszej
> wiadomości jest zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę
> bezzwłocznie odesłać ją nadawcy, a samą wiadomość usunąć z komputera.
> Dziękujemy.  Note: The information
> contained in this message may be privileged and confidential and protected
> from disclosure. If the reader of this message is not the intended
> recipient, or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited.If you
> have received this communication in error, please notify the sender
> immediately by replying to the message and deleting it from your computer.
> Thank you. 
>

PutHiveStreaming + Hive 3.0

[Michal Tomaszewski]

Hello,
Is PutHiveStreaming processor working with Hive 3.0?
We installed Hortonworks HDP 3.0, compiled newest NiFi 1.8 snapshot and got 
errors:
Failed connecting to Hive endpoint table: hivetest3 at thrift://servername.
NiFi has access to all site.xml's. Configuration is exactly the same as with 
previous Hive server, that worked properly.

Regards,
Mike


 Uwaga: Treść niniejszej wiadomości 
może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości 
nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym 
do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, 
rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli 
otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a 
samą wiadomość usunąć z komputera. Dziękujemy.  
Note: The information contained in this message may be privileged and 
confidential and protected from disclosure. If the reader of this message is 
not the intended recipient, or an employee or agent responsible for delivering 
this message to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited.If you have received this communication in error, please notify the 
sender immediately by replying to the message and deleting it from your 
computer. Thank you. 

Re: Re: Pushing to bitbucket from nifi registry

[Krish Kumar]

Kevin,

Thanks for the reply, I've gone with the password ssh option and its working perfectly. 

Thanks

Krish
On 29/07/2018, 16:33 Kevin Doran  wrote:
Hi, Glad to hear you are finding the NiFi Registry features useful. Regarding your question, the “Remote Access User” and “Remote Access Password” properties are only used when the remote URL is an HTTPS url. When it is an SSH url, it is expected that password-less SSH has been configured on your NiFi Registry instance, and the user/password properties are ignored. If it is possible, I would suggest you try one of the following:Setup password-less SSH (e.g., if you can ensure no one else has access to the private ssh key through some other mechanism, such as file permissions)If your remote bitbucket accepts username/password authentication over https, change your remote to be https based rather than ssh based. I hope this helps! If you have any other trouble/questions let me know. Regards,KevinOn Sun, Jul 29, 2018 at 11:18 AM, Krish  wrote:



Hello,I've just upgraded from NiFi 1.2.0 to 1.7.1 and been playing around with the NiFi registry features which are great!I've been trying to get it to push automatically to my bitbucket repository but I keep getting the following error: "Failed to push commits to origin due to org.eclipse.jgit.api.errors.TransfportException ssh://git@bitbucketserver:1234/nifi-registry.git Auth Fail" In my configuration I have the folllowing setttings in my providers.xml:        org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProviderclass>        ./storage/nifi_registry_repo        origin        ssh_key_user        ssh key passwordThe nifi_registry_repo is a git repo which I have cloned down from my bitbucket server. I've setup a SSH and loaded into the bitbucket repo, I can successfully push and pull from the command line on the nifi registry server using the ssh key.When I change a flow in the NiFi registry UI it commits successfully to the repo but fails to push the changes up to the bitbucket server.Has anyone else done this successfully? Or is there mistake I've made in my configs?ThanksK