Thank you all for your help I think Koji's thread at JIRA will be the best solution when it is implemented
Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find much examples about using NIFI REST APIs. Is it possible to POST the WEB UI (including all inputs from multipart form and uploaded files in a POST request to a NIFI input port of a remote NIFI instance? because, I think, this will avoid using "handlehttprequest" processor and allow users to communicate with the remote NIFI instance both from WEB UI and/or NIFI instance (e.g., site-to-site). Regards On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <[email protected]> wrote: > If the multipart contains only text data, it might be possible to use > those processors Andy suggested. However, if we support multipart > natively, it can be done much easier and safer. I created a JIRA for > this improvement before, but haven't been able to find a time to > implement it myself. > If anyone interested in tackling this enhancement, it'd be greatly > appreciated. > https://issues.apache.org/jira/browse/NIFI-3469 > > Thanks, > Koji > > On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <[email protected]> > wrote: > > Absolutely. Use some combination of SplitText, SplitContent, SplitJSON, > and > > ExtractText to split the content into multiple flowfiles on regular > > expressions, new lines, and/or byte sequences and then RouteOnContent to > > send the flowfiles which reference a file to a processor to > retrieve/parse > > the file contents. > > > > Andy LoPresto > > [email protected] > > [email protected] > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <[email protected]> > > wrote: > > > > This is an example of the output file from "handlehttprequest": > > > > " > > -----------------------------831805444208556434014280960^M > > Content-Disposition: form-data; name="userid"^M > > ^M > > jpl_user^M > > -----------------------------831805444208556434014280960^M > > Content-Disposition: form-data; name="scripts"; filename="a.py"^M > > Content-Type: text/x-python^M > > ^M > > ^M > > -----------------------------831805444208556434014280960^M > > Content-Disposition: form-data; name="scripts"; filename="b.py"^M > > Content-Type: text/x-python^M > > ^M > > ^M > > -----------------------------831805444208556434014280960^M > > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M > > Content-Type: application/json^M > > ^M > > { > > "userid":"jpl_user_1", > > "scripts":["a.py","b.py"] > > } > > ^M > > -----------------------------831805444208556434014280960--^M > > " > > > > Can I extract different components of this file without script at server > > side? > > > > Regards > > > > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <[email protected]> > > wrote: > >> > >> Mohammed, > >> > >> So from what I understand you have a separate web application which > >> accepts a variety of data (a combination of form fields and file > uploads) > >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what > problem > >> you had extracting these “components” from the output flowfile from the > >> HandleHTTPRequest processor. Can you change your web interface to send > >> separate POST requests to NiFi and logically separate these components > from > >> the beginning? Can you modify the format of the request (i.e. make it > JSON, > >> etc.) so it is properly structured for NiFi to parse and operate on this > >> data? > >> > >> NiFi treats the content of the flowfile the same once it is in the > system > >> regardless of the source processor, so you should be able to split, > extract, > >> or manipulate the contents of a single flowfile containing multiple form > >> elements and file bodies. > >> > >> If you are referencing NiFi’s Input Port components as a destination for > >> an arbitrary HTTP request, this will not work. The input ports are > designed > >> to work with NiFi’s Site to Site protocol (either over raw sockets or > >> HTTP(S)). > >> > >> > >> Andy LoPresto > >> [email protected] > >> [email protected] > >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> > >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <[email protected]> > >> wrote: > >> > >> Thanks Andy > >> > >> I have a user web interface that send some form information (e.g., user > >> id), as well as upload multiple files, to a remote NIFI instance. When > using > >> "handlhttprequest" processor, it results in a single file that contains > >> "form inputs" and bodies of uploaded files. > >> > >> I want to extract individual components of this file (i.e., download all > >> uploaded files and extract other form information (e.g., userid) in a > JSON > >> or other text file. I had a similar problem before and I learnt that > >> "httphandlerequest" has a problem to extract individual compoents from, > so I > >> have to write my own scripts to extract these components (form inputs > and > >> uploaded files). > >> > >> As NIFI input ports does not face this problem of "handlehttprequest", I > >> though if I can send form inputs and upload files directly to "NIFI > input > >> port" at remote site, then I don't have to write my own scripts to > extract > >> individual components. > >> > >> Regards > >> > >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <[email protected]> > >> wrote: > >>> > >>> Mohammed, > >>> > >>> I’m not sure I understand your question. NiFi has a REST API which > >>> supports the UI and allows scripts/other applications to make requests > of > >>> the platform. This runs on the input port specified in nifi.properties. > >>> > >>> If you want to submit form data and uploaded files as data to be > ingested > >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the > processors > >>> you would want to use. > >>> > >>> If you can clarify your use case, we should be able to offer more > >>> assistance. > >>> > >>> Andy LoPresto > >>> [email protected] > >>> [email protected] > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>> > >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <[email protected]> > >>> wrote: > >>> > >>> Hi > >>> > >>> Is it possible to send http request (including form data and uploaded > >>> files) to NIFI input port instead of "HandleHTTPRequest" processor? > >>> > >>> Regards > >>> > >>> -- > >>> Mohammed > >>> > >>> > >> > >> > >> > >> -- > >> Mohammed > >> > >> > > > > > > > > -- > > Mohammed > > > > > -- Mohammed
