Re: [Dev] Accessing original file name from Smooks ESB Mediator
Ok. Thank you. Is there another WSO2 ESB implementation pattern I can use to split a large (> 1 GB) file into smaller files without using the Smooks mediator? (where I can avoid loading the entire file into memory and also have the split files be named based on the original file name) Regards, - Rudy From: Shiva Balachandran [mailto:sh...@wso2.com] Sent: Tuesday, August 25, 2015 11:12 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Accessing original file name from Smooks ESB Mediator Hi Rudy, AFAIK, we cannot use that property inside smooks, cause we only pass the stream to the smooks mediator. Thank you. Regards, Shiva Balachandran On Tue, Aug 25, 2015 at 8:38 PM, Rudy Hilado wrote: Shiva, Yes, I can access the file name and put it into the property “Processing_File_Name”, but how can I access that property from within the Smooks mediator configuration (), particularly from within the element? Thanks, - Rudy From: Shiva Balachandran [mailto:sh...@wso2.com] Sent: Tuesday, August 25, 2015 10:37 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Accessing original file name from Smooks ESB Mediator Have you tried setting a property to capture the file name that's being processed? Check this link <https://shivabalachandran.wordpress.com/2015/08/06/quick-note-5-wso2-findout-access-lookup-identify-the-name-of-the-current-file-being-processed-by-the-vfs-listener/> . On Mon, Aug 24, 2015 at 11:21 PM, Rudy Hilado wrote: Hello (I thought I’d post this again to see if anyone has some insight) In the WSO2 ESB, we are trying to process very large files and I’ve reviewed the online examples that explain the Smooks mediator should be used to split the larger file into smaller files. Using the smooks configuration , the output file name is defined in the . All the examples show the file name pattern derived using data from the record being processed. Is it possible to access the source file name? I need the split files to have a name that includes the original file name. (I’m beginning to suspect that, oddly, it’s not possible to access the original file name within Smooks) Thanks for any help, - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Shiva Balachandran Software Engineer WSO2 Inc. Mobile - +94 774445788 Blog - https://shivabalachandran.wordpress.com/ -- Shiva Balachandran Software Engineer WSO2 Inc. Mobile - +94 774445788 Blog - https://shivabalachandran.wordpress.com/ ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Accessing original file name from Smooks ESB Mediator
Shiva, Yes, I can access the file name and put it into the property “Processing_File_Name”, but how can I access that property from within the Smooks mediator configuration (), particularly from within the element? Thanks, - Rudy From: Shiva Balachandran [mailto:sh...@wso2.com] Sent: Tuesday, August 25, 2015 10:37 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Accessing original file name from Smooks ESB Mediator Have you tried setting a property to capture the file name that's being processed? Check this link <https://shivabalachandran.wordpress.com/2015/08/06/quick-note-5-wso2-findout-access-lookup-identify-the-name-of-the-current-file-being-processed-by-the-vfs-listener/> . On Mon, Aug 24, 2015 at 11:21 PM, Rudy Hilado wrote: Hello (I thought I’d post this again to see if anyone has some insight) In the WSO2 ESB, we are trying to process very large files and I’ve reviewed the online examples that explain the Smooks mediator should be used to split the larger file into smaller files. Using the smooks configuration , the output file name is defined in the . All the examples show the file name pattern derived using data from the record being processed. Is it possible to access the source file name? I need the split files to have a name that includes the original file name. (I’m beginning to suspect that, oddly, it’s not possible to access the original file name within Smooks) Thanks for any help, - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Shiva Balachandran Software Engineer WSO2 Inc. Mobile - +94 774445788 Blog - https://shivabalachandran.wordpress.com/ ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Accessing original file name from Smooks ESB Mediator
Hello (I thought I’d post this again to see if anyone has some insight) In the WSO2 ESB, we are trying to process very large files and I’ve reviewed the online examples that explain the Smooks mediator should be used to split the larger file into smaller files. Using the smooks configuration , the output file name is defined in the . All the examples show the file name pattern derived using data from the record being processed. Is it possible to access the source file name? I need the split files to have a name that includes the original file name. (I’m beginning to suspect that, oddly, it’s not possible to access the original file name within Smooks) Thanks for any help, - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Pause/restart when processing a large file, accessing original file name from Smooks
Thanks again for the information! We are trying to process very large files and I’ve reviewed the online examples that explain that the Smooks mediator should be used to split the larger file into smaller files using the smooks configuration , with the output file name defined in the . All the examples show the file name pattern derived using data from the record being processed. Is it possible to access the source file name? I need the split files to have a name that includes the original file name. The expression you mention below returns the original file name, but how can this property be access from within the Smooks configuration ? Regards, - Rudy From: Malaka Silva [mailto:mal...@wso2.com] Sent: Thursday, August 06, 2015 11:58 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Pause/restart when processing a large file Hi Rudy, With [1] you can retrieve the processing file name, and with [2] you can set the response (out) file name. [1] [2] On Thu, Aug 6, 2015 at 2:29 PM, Rudy Hilado wrote: Malaka, Thanks for the information! When splitting the larger file into smaller files, how do you create the smaller files with a name based on the parent file? For example, if the source file is named “documentABC.txt”, I would like the smaller files to be named “documentABC_0001.txt” and “documentABC_0002.txt” etc. I’m having a difficult time figuring out how to access the name of the current file being processed by the VFS listener. Thanks, - Rudy From: Malaka Silva [mailto:mal...@wso2.com] Sent: Thursday, August 06, 2015 1:57 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Pause/restart when processing a large file Hi Rudy, I am afraid currently this is not supported out of the box. However you can achieve your requirement with the following solution. You can split the file and put in a separate folder (One entry in a file). eg:- VFS Listerner [3] -> Iterator mediator [4] -> File Connector [1][2] Take one file and process it. eg:- VFS listener [3] Following doc links will help you to do this. [1] https://storepreview.wso2.com/store/assets/esbconnector/f0550184-88c4-4d5e-b73f-ea51b1b5ab98 [2] https://docs.wso2.com/display/ESBCONNECTORS/Working+with+the+File+Connector#WorkingwiththeFileConnector-create [3] https://docs.wso2.com/display/ESB481/VFS+Transport [4] https://docs.wso2.com/display/ESB481/Iterate+Mediator On Wed, Aug 5, 2015 at 8:52 PM, Rudy Hilado wrote: Is it possible in the WSO2 ESB to process a large file (with each line representing a single record/message), and stop or pause the processing within the file, then restart where it left off? (using the Smooks mediator and/or Iterate mediator, or any other mechanism) It appears that if you are processing a large file (say with 10K entries) you cannot stop or pause the processing (or if the ESB is brought down in the middle of processing the file), and then restart where it was stopped. Upon restarting either the whole file has to be reprocessed, or the file is dumped to the error folder and skipped. Is this correct? Thanks for any help on this. - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Best Regards, Malaka Silva Senior Tech Lead M: +94 777 219 791 Tel : 94 11 214 5345 Fax :94 11 2145300 Skype : malaka.sampath.silva LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 Blog : http://mrmalakasilva.blogspot.com/ WSO2, Inc. lean . enterprise . middleware http://www.wso2.com/ http://www.wso2.com/about/team/malaka-silva/ <http://wso2.com/about/team/malaka-silva/> Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary. -- Best Regards, Malaka Silva Senior Tech Lead M: +94 777 219 791 Tel : 94 11 214 5345 Fax :94 11 2145300 Skype : malaka.sampath.silva LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 Blog : http://mrmalakasilva.blogspot.com/ WSO2, Inc. lean . enterprise . middleware http://www.wso2.com/ http://www.wso2.com/about/team/malaka-silva/ <http://wso2.com/about/team/malaka-silva/> Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Pause/restart when processing a large file
Malaka, Thanks for the information! When splitting the larger file into smaller files, how do you create the smaller files with a name based on the parent file? For example, if the source file is named “documentABC.txt”, I would like the smaller files to be named “documentABC_0001.txt” and “documentABC_0002.txt” etc. I’m having a difficult time figuring out how to access the name of the current file being processed by the VFS listener. Thanks, - Rudy From: Malaka Silva [mailto:mal...@wso2.com] Sent: Thursday, August 06, 2015 1:57 AM To: Rudy Hilado Cc: WSO2 Developers' List Subject: Re: [Dev] Pause/restart when processing a large file Hi Rudy, I am afraid currently this is not supported out of the box. However you can achieve your requirement with the following solution. You can split the file and put in a separate folder (One entry in a file). eg:- VFS Listerner [3] -> Iterator mediator [4] -> File Connector [1][2] Take one file and process it. eg:- VFS listener [3] Following doc links will help you to do this. [1] https://storepreview.wso2.com/store/assets/esbconnector/f0550184-88c4-4d5e-b73f-ea51b1b5ab98 [2] https://docs.wso2.com/display/ESBCONNECTORS/Working+with+the+File+Connector#WorkingwiththeFileConnector-create [3] https://docs.wso2.com/display/ESB481/VFS+Transport [4] https://docs.wso2.com/display/ESB481/Iterate+Mediator On Wed, Aug 5, 2015 at 8:52 PM, Rudy Hilado wrote: Is it possible in the WSO2 ESB to process a large file (with each line representing a single record/message), and stop or pause the processing within the file, then restart where it left off? (using the Smooks mediator and/or Iterate mediator, or any other mechanism) It appears that if you are processing a large file (say with 10K entries) you cannot stop or pause the processing (or if the ESB is brought down in the middle of processing the file), and then restart where it was stopped. Upon restarting either the whole file has to be reprocessed, or the file is dumped to the error folder and skipped. Is this correct? Thanks for any help on this. - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Best Regards, Malaka Silva Senior Tech Lead M: +94 777 219 791 Tel : 94 11 214 5345 Fax :94 11 2145300 Skype : malaka.sampath.silva LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 Blog : http://mrmalakasilva.blogspot.com/ WSO2, Inc. lean . enterprise . middleware http://www.wso2.com/ http://www.wso2.com/about/team/malaka-silva/ <http://wso2.com/about/team/malaka-silva/> Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Pause/restart when processing a large file
Is it possible in the WSO2 ESB to process a large file (with each line representing a single record/message), and stop or pause the processing within the file, then restart where it left off? (using the Smooks mediator and/or Iterate mediator, or any other mechanism) It appears that if you are processing a large file (say with 10K entries) you cannot stop or pause the processing (or if the ESB is brought down in the middle of processing the file), and then restart where it was stopped. Upon restarting either the whole file has to be reprocessed, or the file is dumped to the error folder and skipped. Is this correct? Thanks for any help on this. - Rudy ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
