Re: StoreInKiteDataset cannot recognize hive dataset

[Bryan Bende]

Hello,

I'm not that familiar with Kite, but is it possible that you need to create
the Kite dataset using the Kite CLI before StoreInKiteDataset tries to
write data to it?

It looks like that is how the test cases for this processor work:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestKiteProcessorsCluster.java#L85

It uses "dataset:hive:ns/test", but calls Datasets.create(...) before
running the processor.

-Bryan


On Mon, Nov 9, 2015 at 5:06 AM, panfei  wrote:

> Hi all:
> I use StoreInKiteDataset processor to try to store data in hive by
> configuring the target URI to:
>
> *dataset:hive:default/sandwiches*
>
> but the processor reports that* the URI is invalid*. but after replacing
> the URI to
>
> *dataset:file:/tmp/sandwiches*
>
> everything works OK.
>
>
> Is there any way to resolve the Hive  dataset issue ?  or it is not
> supported at all ?
>
>
> Thank you very much
> --
> 不学习，不知道
>

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[Joe Witt]

Mans,

Being new to NiFi or even contributing to open source at all is not a
problem.  We're here to help.  Ask questions as needed and we can help
you contribute.

Thanks
Joe

On Mon, Nov 9, 2015 at 9:38 AM, M Singh  wrote:
> Hi Joe:
>
> I was looking at the InvokeHttp code and I can work on enhancing the 3xx
> issue based on the pattern for handling other statuses.  However, I would
> like to add a newbie caveat here.
>
> Let me know if that would help.
>
> Thanks
>
> Mans
>
>
>
> On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
>
>
> Hi Joe:
>
> You are right - setting follow-redirects did not work and I did mix retry
> with redirect.
>
> I will wait for your enhancements.
>
> Thanks again for your help.
>
>
>
> On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
> wrote:
>
>
> Hello,
>
>
> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status
> code is for redirecting to another url and 5xx is to try again. The property
> we have is "Follow Redirects". Retrying doesn't involve a location header
> but the redirect does. That being said, I did a bit of digging I don't think
> InvokeHttp was handling redirects properly. All we were doing was setting
> the "setInstanceFollowRedirects" to true, which according to this site [1]
> doesn't fully handle it.
>
> I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's
> refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test
> that out and see if that it solves the redirect and location header problem
> that would be awesome.
>
> [1]
> http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
> [2] https://issues.apache.org/jira/browse/NIFI-1086
>
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Sunday, November 8, 2015 9:53 PM, Joe Witt  wrote:
>
>
>
> Joe P,
>
> Do you have any other recommendations light of the work you're doing
> now to Invoke HTTP?
>
> Thanks
> Joe
>
>
> On Sun, Nov 8, 2015 at 9:49 PM, M Singh  wrote:
>> Thanks Joe.
>>
>> When I look at the provenance of the flow file, it shows the status as 301
>> as shown below but no Location attribute.
>>
>> If I curl the url and check the headers it does show the Location
>> attribute.
>>
>> invokehttp.status.code  301
>>
>>
>>
>>
>> On Sunday, November 8, 2015 4:58 PM, Joe Witt  wrote:
>>
>>
>> Hello
>>
>> You can use provenance or a LogAttributes processor to see what the
>> headers are of the flow file after InvokeHTTP executes.  You may find
>> the location header present as one of the attributes.  If so then you
>> should be able to use that attribute to establish the URL it will hit
>> next time.  Does that help?
>>
>> Thanks
>> Joe
>>
>> On Sun, Nov 8, 2015 at 7:51 PM, M Singh  wrote:
>>> Hi:
>>>
>>> I am trying to use InvokeHTTP and have set the follow-retry to true, and
>>> have associated a self referencing relation for the InvokeHttp for
>>> no-retry
>>> (1xx, 2xx, and 3xx) and retry (5xx) relations.  But it looks like it
>>> retires
>>> only status 500 codes requests and for 3xx does not pick up the Location
>>> header for retrying.
>>>
>>> However, since I am new to nifi - I might be missing something or using
>>> the
>>> wrong settings or wrong processor.
>>>
>>> If anyone has any suggestion, please let me know.
>>>
>>> Thanks
>>>
>>
>>
>
>
>
>

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[M Singh]

Hi Joe:
I was looking at the InvokeHttp code and I can work on enhancing the 3xx issue 
based on the pattern for handling other statuses.  However, I would like to add 
a newbie caveat here.
Let me know if that would help.
Thanks
Mans 


 On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
   

 Hi Joe:
You are right - setting follow-redirects did not work and I did mix retry with 
redirect.
I will wait for your enhancements.
Thanks again for your help. 


 On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
 wrote:
   

 Hello,


Firstly, I think you're mixing up "retry" and "redirect". The 3xx status code 
is for redirecting to another url and 5xx is to try again. The property we have 
is "Follow Redirects". Retrying doesn't involve a location header but the 
redirect does. That being said, I did a bit of digging I don't think InvokeHttp 
was handling redirects properly. All we were doing was setting the 
"setInstanceFollowRedirects" to true, which according to this site [1] doesn't 
fully handle it. 

I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's 
refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test that 
out and see if that it solves the redirect and location header problem that 
would be awesome.

[1] http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
[2] https://issues.apache.org/jira/browse/NIFI-1086
 
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Sunday, November 8, 2015 9:53 PM, Joe Witt  wrote:



Joe P,

Do you have any other recommendations light of the work you're doing
now to Invoke HTTP?

Thanks
Joe


On Sun, Nov 8, 2015 at 9:49 PM, M Singh  wrote:
> Thanks Joe.
>
> When I look at the provenance of the flow file, it shows the status as 301
> as shown below but no Location attribute.
>
> If I curl the url and check the headers it does show the Location attribute.
>
> invokehttp.status.code  301
>
>
>
>
> On Sunday, November 8, 2015 4:58 PM, Joe Witt  wrote:
>
>
> Hello
>
> You can use provenance or a LogAttributes processor to see what the
> headers are of the flow file after InvokeHTTP executes.  You may find
> the location header present as one of the attributes.  If so then you
> should be able to use that attribute to establish the URL it will hit
> next time.  Does that help?
>
> Thanks
> Joe
>
> On Sun, Nov 8, 2015 at 7:51 PM, M Singh  wrote:
>> Hi:
>>
>> I am trying to use InvokeHTTP and have set the follow-retry to true, and
>> have associated a self referencing relation for the InvokeHttp for
>> no-retry
>> (1xx, 2xx, and 3xx) and retry (5xx) relations.  But it looks like it
>> retires
>> only status 500 codes requests and for 3xx does not pick up the Location
>> header for retrying.
>>
>> However, since I am new to nifi - I might be missing something or using
>> the
>> wrong settings or wrong processor.
>>
>> If anyone has any suggestion, please let me know.
>>
>> Thanks
>>
>
>


   

  

Re: Reading file

[Aldrin Piri]

Hi Naveen,

The treatment of that file as one FlowFile is as intended.  If you wanted
to break that out to separate FlowFiles, you could use the SplitText
processor with one line per split and have a FlowFile per object.

On Mon, Nov 9, 2015 at 4:27 PM, Madhire, Naveen <
naveen.madh...@capitalone.com> wrote:

> Hi All,
>
> I am using GetFile processor to read a “\n” line delimiter text file. Each
> line contains a JSON structure.
> The GetFile processor is treating the whole file as one flow file instead
> of each line as a single flow file. Is there any other processor which I
> can use to read each line as one single flow file?
>
>
> Thanks,
> Naveen
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[M Singh]

Joe:
No problem and thanks for your support in fixing the issue.
Mans 


 On Monday, November 9, 2015 12:35 PM, Joe Percivall 
 wrote:
   

 Mans,

I did a bit of testing using my refactor and I understand what the issue is. I 
was able to successfully complete the first use-case where "follow redirect" is 
true and within the same ontrigger follow the redirect.

The second use-case where the "follow redirect" is false didn't work as 
intended. It ends up routing to "No Retry" and emitting no response flowfile. 
The problem arises that the location header is not added to the request 
flowfile so there is no way to know where it was supposed to redirect to later. 

This actually brings up a bigger issue that is at the core of this issue. That 
is when the processor reaches out to a website and routes to any of the not 2xx 
relationships, the headers (other than status code and message) that are 
returned by the response aren't stored anywhere. I'm thinking we have a 
property that is a regular expression that allows users to match specific 
headers to add as attributes to the request flowfile. Thoughts?

Not trying to discourage you from contributing but I'm going to knock this out 
as part of 1086. There are plenty of other tickets that could use contributors!

Thanks for bringing this up,
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Monday, November 9, 2015 2:05 PM, Joe Percivall  
wrote:
Hey Mans,

Thanks for creating the ticket. First, have you tried the patch I supplied in 
1086 [1]? I changed the underlying implementation  of InvokeHttp so it very 
well could already be fixed.

The functionality should work where if you have "follow redirects" to true then 
it should follow the 3xx response to hit the other server automatically within 
the same ontrigger call. If "follow redirects" is false then it would not 
follow the redirect to the  location set in the header and instead route the 
request to the 3xx relationship.


[1] https://issues.apache.org/jira/browse/NIFI-1086
Thanks,
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com





On Monday, November 9, 2015 1:54 PM, M Singh  wrote:



Hi:

I could not find Jira ticket for InvokeHTTP processor not saving location 
header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP 
Processor does not save Location header for 3xx responses - ASF JIRA.

My understanding is that once the http status is saved and a relationship for 
3xx is available then we can route the flow file with appropriate attribute 
(based on location header) to the invoke http processor so it can make a new 
request.


Please let me know if it ok and if I can start working on it.

Thanks 
  
          
[NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx 
responses - ASF JIRA
InvokeHTTP processor does not save the Location header for 3xx responses. So, 
we can cannot hook up the processor to route based on redirect location header 
values.  
View on issues.apache.org Preview by Yahoo 
  




On Monday, November 9, 2015 6:51 AM, Joe Witt  wrote:



Mans,

Being new to NiFi or even contributing to open source at all is not a
problem.  We're here to help.  Ask questions as needed and we can help
you contribute.

Thanks
Joe


On Mon, Nov 9, 2015 at 9:38 AM, M Singh  wrote:
> Hi Joe:
>
> I was looking at the InvokeHttp code and I can work on enhancing the 3xx
> issue based on the pattern for handling other statuses.  However, I would
> like to add a newbie caveat here.
>
> Let me know if that would help.
>
> Thanks
>
> Mans
>
>
>
> On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
>
>
> Hi Joe:
>
> You are right - setting follow-redirects did not work and I did mix retry
> with redirect.
>
> I will wait for your enhancements.
>
> Thanks again for your help.
>
>
>
> On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
> wrote:
>
>
> Hello,
>
>
> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status
> code is for redirecting to another url and 5xx is to try again. The property
> we have is "Follow Redirects". Retrying doesn't involve a location header
> but the redirect does. That being said, I did a bit of digging I don't think
> InvokeHttp was handling redirects properly. All we were doing was setting
> the "setInstanceFollowRedirects" to true, which according to this site [1]
> doesn't fully handle it.
>
> I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's
> refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test
> that out and see if that it solves the redirect and location header problem
> that would be awesome.
>
> [1]
> http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
> [2] https://issues.apache.org/jira/browse/NIFI-1086
>
> 

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[M Singh]

Hi:
I could not find Jira ticket for InvokeHTTP processor not saving location 
header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP 
Processor does not save Location header for 3xx responses - ASF JIRA.
My understanding is that once the http status is saved and a relationship for 
3xx is available then we can route the flow file with appropriate attribute 
(based on location header) to the invoke http processor so it can make a new 
request.

Please let me know if it ok and if I can start working on it.
Thanks 
|   |
|   |   |   |   |   |
| [NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx 
responses - ASF JIRAInvokeHTTP processor does not save the Location header for 
3xx responses. So, we can cannot hook up the processor to route based on 
redirect location header values.  |
|  |
| View on issues.apache.org | Preview by Yahoo |
|  |
|   |


 


 On Monday, November 9, 2015 6:51 AM, Joe Witt  wrote:
   

 Mans,

Being new to NiFi or even contributing to open source at all is not a
problem.  We're here to help.  Ask questions as needed and we can help
you contribute.

Thanks
Joe

On Mon, Nov 9, 2015 at 9:38 AM, M Singh  wrote:
> Hi Joe:
>
> I was looking at the InvokeHttp code and I can work on enhancing the 3xx
> issue based on the pattern for handling other statuses.  However, I would
> like to add a newbie caveat here.
>
> Let me know if that would help.
>
> Thanks
>
> Mans
>
>
>
> On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
>
>
> Hi Joe:
>
> You are right - setting follow-redirects did not work and I did mix retry
> with redirect.
>
> I will wait for your enhancements.
>
> Thanks again for your help.
>
>
>
> On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
> wrote:
>
>
> Hello,
>
>
> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status
> code is for redirecting to another url and 5xx is to try again. The property
> we have is "Follow Redirects". Retrying doesn't involve a location header
> but the redirect does. That being said, I did a bit of digging I don't think
> InvokeHttp was handling redirects properly. All we were doing was setting
> the "setInstanceFollowRedirects" to true, which according to this site [1]
> doesn't fully handle it.
>
> I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's
> refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test
> that out and see if that it solves the redirect and location header problem
> that would be awesome.
>
> [1]
> http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
> [2] https://issues.apache.org/jira/browse/NIFI-1086
>
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Sunday, November 8, 2015 9:53 PM, Joe Witt  wrote:
>
>
>
> Joe P,
>
> Do you have any other recommendations light of the work you're doing
> now to Invoke HTTP?
>
> Thanks
> Joe
>
>
> On Sun, Nov 8, 2015 at 9:49 PM, M Singh  wrote:
>> Thanks Joe.
>>
>> When I look at the provenance of the flow file, it shows the status as 301
>> as shown below but no Location attribute.
>>
>> If I curl the url and check the headers it does show the Location
>> attribute.
>>
>> invokehttp.status.code  301
>>
>>
>>
>>
>> On Sunday, November 8, 2015 4:58 PM, Joe Witt  wrote:
>>
>>
>> Hello
>>
>> You can use provenance or a LogAttributes processor to see what the
>> headers are of the flow file after InvokeHTTP executes.  You may find
>> the location header present as one of the attributes.  If so then you
>> should be able to use that attribute to establish the URL it will hit
>> next time.  Does that help?
>>
>> Thanks
>> Joe
>>
>> On Sun, Nov 8, 2015 at 7:51 PM, M Singh  wrote:
>>> Hi:
>>>
>>> I am trying to use InvokeHTTP and have set the follow-retry to true, and
>>> have associated a self referencing relation for the InvokeHttp for
>>> no-retry
>>> (1xx, 2xx, and 3xx) and retry (5xx) relations.  But it looks like it
>>> retires
>>> only status 500 codes requests and for 3xx does not pick up the Location
>>> header for retrying.
>>>
>>> However, since I am new to nifi - I might be missing something or using
>>> the
>>> wrong settings or wrong processor.
>>>
>>> If anyone has any suggestion, please let me know.
>>>
>>> Thanks
>>>
>>
>>
>
>
>
>


  

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[Joe Percivall]

Mans,

I did a bit of testing using my refactor and I understand what the issue is. I 
was able to successfully complete the first use-case where "follow redirect" is 
true and within the same ontrigger follow the redirect.

The second use-case where the "follow redirect" is false didn't work as 
intended. It ends up routing to "No Retry" and emitting no response flowfile. 
The problem arises that the location header is not added to the request 
flowfile so there is no way to know where it was supposed to redirect to later. 

This actually brings up a bigger issue that is at the core of this issue. That 
is when the processor reaches out to a website and routes to any of the not 2xx 
relationships, the headers (other than status code and message) that are 
returned by the response aren't stored anywhere. I'm thinking we have a 
property that is a regular expression that allows users to match specific 
headers to add as attributes to the request flowfile. Thoughts?

Not trying to discourage you from contributing but I'm going to knock this out 
as part of 1086. There are plenty of other tickets that could use contributors!

Thanks for bringing this up,
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Monday, November 9, 2015 2:05 PM, Joe Percivall  
wrote:
Hey Mans,

Thanks for creating the ticket. First, have you tried the patch I supplied in 
1086 [1]? I changed the underlying implementation  of InvokeHttp so it very 
well could already be fixed.

The functionality should work where if you have "follow redirects" to true then 
it should follow the 3xx response to hit the other server automatically within 
the same ontrigger call. If "follow redirects" is false then it would not 
follow the redirect to the  location set in the header and instead route the 
request to the 3xx relationship.


[1] https://issues.apache.org/jira/browse/NIFI-1086
Thanks,
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com





On Monday, November 9, 2015 1:54 PM, M Singh  wrote:



Hi:

I could not find Jira ticket for InvokeHTTP processor not saving location 
header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP 
Processor does not save Location header for 3xx responses - ASF JIRA.

My understanding is that once the http status is saved and a relationship for 
3xx is available then we can route the flow file with appropriate attribute 
(based on location header) to the invoke http processor so it can make a new 
request.


Please let me know if it ok and if I can start working on it.

Thanks 
  
  
[NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx 
responses - ASF JIRA
InvokeHTTP processor does not save the Location header for 3xx responses. So, 
we can cannot hook up the processor to route based on redirect location header 
values.  
View on issues.apache.org Preview by Yahoo 
  




On Monday, November 9, 2015 6:51 AM, Joe Witt  wrote:



Mans,

Being new to NiFi or even contributing to open source at all is not a
problem.  We're here to help.  Ask questions as needed and we can help
you contribute.

Thanks
Joe


On Mon, Nov 9, 2015 at 9:38 AM, M Singh  wrote:
> Hi Joe:
>
> I was looking at the InvokeHttp code and I can work on enhancing the 3xx
> issue based on the pattern for handling other statuses.  However, I would
> like to add a newbie caveat here.
>
> Let me know if that would help.
>
> Thanks
>
> Mans
>
>
>
> On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
>
>
> Hi Joe:
>
> You are right - setting follow-redirects did not work and I did mix retry
> with redirect.
>
> I will wait for your enhancements.
>
> Thanks again for your help.
>
>
>
> On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
> wrote:
>
>
> Hello,
>
>
> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status
> code is for redirecting to another url and 5xx is to try again. The property
> we have is "Follow Redirects". Retrying doesn't involve a location header
> but the redirect does. That being said, I did a bit of digging I don't think
> InvokeHttp was handling redirects properly. All we were doing was setting
> the "setInstanceFollowRedirects" to true, which according to this site [1]
> doesn't fully handle it.
>
> I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's
> refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test
> that out and see if that it solves the redirect and location header problem
> that would be awesome.
>
> [1]
> http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
> [2] https://issues.apache.org/jira/browse/NIFI-1086
>
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Sunday, November 8, 2015 9:53 PM, Joe Witt  

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

[Mark Payne]

Joe,

FWIW, I would personally avoid adding a Regex property. I think we use regexes 
already too much. I would just add the HTTP Headers or not... or you could even 
add a property that specifies whether or not to add the headers as attributes.

This is very much a preference here, though. I think regexes get messy too 
quickly though and are very difficult to write and maintain.

-Mark


> On Nov 9, 2015, at 3:35 PM, Joe Percivall  wrote:
> 
> Mans,
> 
> I did a bit of testing using my refactor and I understand what the issue is. 
> I was able to successfully complete the first use-case where "follow 
> redirect" is true and within the same ontrigger follow the redirect.
> 
> The second use-case where the "follow redirect" is false didn't work as 
> intended. It ends up routing to "No Retry" and emitting no response flowfile. 
> The problem arises that the location header is not added to the request 
> flowfile so there is no way to know where it was supposed to redirect to 
> later. 
> 
> This actually brings up a bigger issue that is at the core of this issue. 
> That is when the processor reaches out to a website and routes to any of the 
> not 2xx relationships, the headers (other than status code and message) that 
> are returned by the response aren't stored anywhere. I'm thinking we have a 
> property that is a regular expression that allows users to match specific 
> headers to add as attributes to the request flowfile. Thoughts?
> 
> Not trying to discourage you from contributing but I'm going to knock this 
> out as part of 1086. There are plenty of other tickets that could use 
> contributors!
> 
> Thanks for bringing this up,
> Joe
> - - - - - - 
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> On Monday, November 9, 2015 2:05 PM, Joe Percivall  
> wrote:
> Hey Mans,
> 
> Thanks for creating the ticket. First, have you tried the patch I supplied in 
> 1086 [1]? I changed the underlying implementation  of InvokeHttp so it very 
> well could already be fixed.
> 
> The functionality should work where if you have "follow redirects" to true 
> then it should follow the 3xx response to hit the other server automatically 
> within the same ontrigger call. If "follow redirects" is false then it would 
> not follow the redirect to the  location set in the header and instead route 
> the request to the 3xx relationship.
> 
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1086
> Thanks,
> Joe
> - - - - - - 
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> 
> On Monday, November 9, 2015 1:54 PM, M Singh  wrote:
> 
> 
> 
> Hi:
> 
> I could not find Jira ticket for InvokeHTTP processor not saving location 
> header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP 
> Processor does not save Location header for 3xx responses - ASF JIRA.
> 
> My understanding is that once the http status is saved and a relationship for 
> 3xx is available then we can route the flow file with appropriate attribute 
> (based on location header) to the invoke http processor so it can make a new 
> request.
> 
> 
> Please let me know if it ok and if I can start working on it.
> 
> Thanks 
> 
> 
> [NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx 
> responses - ASF JIRA
> InvokeHTTP processor does not save the Location header for 3xx responses. So, 
> we can cannot hook up the processor to route based on redirect location 
> header values.  
> View on issues.apache.org Preview by Yahoo 
> 
> 
> 
> 
> 
> On Monday, November 9, 2015 6:51 AM, Joe Witt  wrote:
> 
> 
> 
> Mans,
> 
> Being new to NiFi or even contributing to open source at all is not a
> problem.  We're here to help.  Ask questions as needed and we can help
> you contribute.
> 
> Thanks
> Joe
> 
> 
> On Mon, Nov 9, 2015 at 9:38 AM, M Singh  wrote:
>> Hi Joe:
>> 
>> I was looking at the InvokeHttp code and I can work on enhancing the 3xx
>> issue based on the pattern for handling other statuses.  However, I would
>> like to add a newbie caveat here.
>> 
>> Let me know if that would help.
>> 
>> Thanks
>> 
>> Mans
>> 
>> 
>> 
>> On Sunday, November 8, 2015 8:44 PM, M Singh  wrote:
>> 
>> 
>> Hi Joe:
>> 
>> You are right - setting follow-redirects did not work and I did mix retry
>> with redirect.
>> 
>> I will wait for your enhancements.
>> 
>> Thanks again for your help.
>> 
>> 
>> 
>> On Sunday, November 8, 2015 8:31 PM, Joe Percivall 
>> wrote:
>> 
>> 
>> Hello,
>> 
>> 
>> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status
>> code is for redirecting to another url and 5xx is to try again. The property
>> we have is "Follow Redirects". Retrying doesn't involve a location header
>> but the redirect does. That being said, I did a bit of digging I don't think
>> InvokeHttp was 

Re: Help me figure out why this wont work

[Joe Percivall]

Hey Chris,

I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work 
with InvokeHttp already so I should be able to get this done pretty quick.

 
Joe- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Monday, November 9, 2015 3:23 PM, Christopher Hamm 
 wrote:



Ugg
https://issues.apache.org/jira/browse/NIFI-993



On Mon, Nov 9, 2015 at 3:10 PM, Christopher Hamm  
wrote:

Ok tried that and still wont work. I think it as something to do with message 
of "supports expression language: false" for the url. Attached error and xml 
template. Any help is appreciated. I need to understand how the dynamic 
querying should work in order to complete an actual project. Many thanks
>
>
>
>​
>
>
>
>On Fri, Nov 6, 2015 at 8:04 AM, Aldrin Piri  wrote:
>
>Christopher,
>>
>>
>>For this setup, you want to use GetHTTP. InvokeHTTP operates off of FlowFile 
>>input.  
>>
>>
>>
>>
>>On Fri, Nov 6, 2015 at 2:33 AM, Christopher Hamm  
>>wrote:
>>
>>I have a dataFlow setup that looks up api call using current date. I turn it 
>>on and nothing seems to happen.. Help appreciated.
>>>
>>>
>>>
>>>
>>>
>>>-- 
>>>
>>>Sincerely,
>>>Chris Hamm
>>>(E) ceham...@gmail.com
>>>(Twitter) http://twitter.com/webhamm
>>>(Linkedin) http://www.linkedin.com/in/chrishamm
>>
>
>
>
>-- 
>
>Sincerely,
>Chris Hamm
>(E) ceham...@gmail.com
>(Twitter) http://twitter.com/webhamm
>(Linkedin) http://www.linkedin.com/in/chrishamm


-- 

Sincerely,
Chris Hamm
(E) ceham...@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm

Re: Help me figure out why this wont work

[Aldrin Piri]

Just for awareness, the processor at issue and to be addressed is GetHTTP.
InvokeHTTP supports EL but requires an input.

On Mon, Nov 9, 2015 at 3:26 PM, Joe Percivall 
wrote:

> Hey Chris,
>
> I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of
> work with InvokeHttp already so I should be able to get this done pretty
> quick.
>
>
> Joe- - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Monday, November 9, 2015 3:23 PM, Christopher Hamm <
> em...@christopherhamm.com> wrote:
>
>
>
> Ugg
> https://issues.apache.org/jira/browse/NIFI-993
>
>
>
> On Mon, Nov 9, 2015 at 3:10 PM, Christopher Hamm <
> em...@christopherhamm.com> wrote:
>
> Ok tried that and still wont work. I think it as something to do with
> message of "supports expression language: false" for the url. Attached
> error and xml template. Any help is appreciated. I need to understand how
> the dynamic querying should work in order to complete an actual project.
> Many thanks
> >
> >
> >
> >​
> >
> >
> >
> >On Fri, Nov 6, 2015 at 8:04 AM, Aldrin Piri  wrote:
> >
> >Christopher,
> >>
> >>
> >>For this setup, you want to use GetHTTP. InvokeHTTP operates off of
> FlowFile input.
> >>
> >>
> >>
> >>
> >>On Fri, Nov 6, 2015 at 2:33 AM, Christopher Hamm <
> em...@christopherhamm.com> wrote:
> >>
> >>I have a dataFlow setup that looks up api call using current date. I
> turn it on and nothing seems to happen.. Help appreciated.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>--
> >>>
> >>>Sincerely,
> >>>Chris Hamm
> >>>(E) ceham...@gmail.com
> >>>(Twitter) http://twitter.com/webhamm
> >>>(Linkedin) http://www.linkedin.com/in/chrishamm
> >>
> >
> >
> >
> >--
> >
> >Sincerely,
> >Chris Hamm
> >(E) ceham...@gmail.com
> >(Twitter) http://twitter.com/webhamm
> >(Linkedin) http://www.linkedin.com/in/chrishamm
>
>
> --
>
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>

Re: Help me figure out why this wont work

[Mark Payne]

Christopher,

You can look into the EvaluateXQuery, EvaluateXPath, and TransformXml 
processors to extract data from XML files.

You can then use ReplaceText to build the SQL statement that you want to use 
and PutSQL to update the database.

Thanks
-Mark


> On Nov 9, 2015, at 3:42 PM, Christopher Hamm  wrote:
> 
> I just tried with 0B size set and it worked fine. Now I just need to look 
> into parsing the xml to extract fields and update a MySql db.
> 
> Any guidance on that?
> 
> On Nov 9, 2015 3:32 PM, "Joe Witt"  > wrote:
> "Did you try using an invokeHTTP processor with a FlowFile input file created 
> using the GenerateFlowFile processor?" -- Which we should admit is super 
> hacky and we're fixing in 0.4.0 :-)
> 
> On Mon, Nov 9, 2015 at 3:30 PM, Matthew Clarke  > wrote:
> Did you try using an invokeHTTP processor with a FlowFile input file created 
> using the GenerateFlowFile processor?
> 
> On Mon, Nov 9, 2015 at 3:23 PM, Christopher Hamm  > wrote:
> Ugg
> https://issues.apache.org/jira/browse/NIFI-993 
> 
> 
> On Mon, Nov 9, 2015 at 3:10 PM, Christopher Hamm  > wrote:
> Ok tried that and still wont work. I think it as something to do with message 
> of "supports expression language: false" for the url. Attached error and xml 
> template. Any help is appreciated. I need to understand how the dynamic 
> querying should work in order to complete an actual project. Many thanks
> 
> 
> ​
> 
> On Fri, Nov 6, 2015 at 8:04 AM, Aldrin Piri  > wrote:
> Christopher,
> 
> For this setup, you want to use GetHTTP. InvokeHTTP operates off of FlowFile 
> input.  
> 
> 
> On Fri, Nov 6, 2015 at 2:33 AM, Christopher Hamm  > wrote:
> I have a dataFlow setup that looks up api call using current date. I turn it 
> on and nothing seems to happen.. Help appreciated.
> 
> 
> 
> -- 
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com 
> (Twitter) http://twitter.com/webhamm 
> (Linkedin) http://www.linkedin.com/in/chrishamm 
> 
> 
> 
> 
> -- 
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com 
> (Twitter) http://twitter.com/webhamm 
> (Linkedin) http://www.linkedin.com/in/chrishamm 
> 
> 
> 
> -- 
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com 
> (Twitter) http://twitter.com/webhamm 
> (Linkedin) http://www.linkedin.com/in/chrishamm 
> 
> 

Re: Help me figure out why this wont work

[Joe Percivall]

Chris,

Just posted a patch to allow expression language for the URL property[1]. I 
also enabled expression language for the filename property since there didn't 
seem to be a reason not to. 

[1] https://issues.apache.org/jira/browse/NIFI-993

Hope that helps,
Joe

- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Monday, November 9, 2015 3:26 PM, Joe Percivall  
wrote:
Hey Chris,

I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work 
with InvokeHttp already so I should be able to get this done pretty quick.


Joe- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com





On Monday, November 9, 2015 3:23 PM, Christopher Hamm 
 wrote:



Ugg
https://issues.apache.org/jira/browse/NIFI-993



On Mon, Nov 9, 2015 at 3:10 PM, Christopher Hamm  
wrote:

Ok tried that and still wont work. I think it as something to do with message 
of "supports expression language: false" for the url. Attached error and xml 
template. Any help is appreciated. I need to understand how the dynamic 
querying should work in order to complete an actual project. Many thanks
>
>
>
>​
>
>
>
>On Fri, Nov 6, 2015 at 8:04 AM, Aldrin Piri  wrote:
>
>Christopher,
>>
>>
>>For this setup, you want to use GetHTTP. InvokeHTTP operates off of FlowFile 
>>input.  
>>
>>
>>
>>
>>On Fri, Nov 6, 2015 at 2:33 AM, Christopher Hamm  
>>wrote:
>>
>>I have a dataFlow setup that looks up api call using current date. I turn it 
>>on and nothing seems to happen.. Help appreciated.
>>>
>>>
>>>
>>>
>>>
>>>-- 
>>>
>>>Sincerely,
>>>Chris Hamm
>>>(E) ceham...@gmail.com
>>>(Twitter) http://twitter.com/webhamm
>>>(Linkedin) http://www.linkedin.com/in/chrishamm
>>
>
>
>
>-- 
>
>Sincerely,
>Chris Hamm
>(E) ceham...@gmail.com
>(Twitter) http://twitter.com/webhamm
>(Linkedin) http://www.linkedin.com/in/chrishamm


-- 

Sincerely,
Chris Hamm
(E) ceham...@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm