Thanks Priyanka, I would try with caching in my upcoming project. Do you have any sample that can be helpful to me?
Regards, Surya Vamshi From: Priyanka Gugale [mailto:[email protected]] Sent: 2016, August, 11 12:53 AM To: [email protected] Subject: Re: Information Needed IMO, it's okay to keep the files in HDFS, at startup you can load them in memory which will be one time activity in application lifetime. Considering they are format/config files, it won't be difficult to cache it in memory of your current operator (no need to caching operator separately). -Priyanka On Wed, Aug 10, 2016 at 7:06 PM, Mukkamula, Suryavamshivardhan (CWM-NR) <[email protected]<mailto:[email protected]>> wrote: Hello Team, I have a use case where I am trying to read number of input files with different formats(may be 100 source systems) and I have to parse each record. To achieve this we need to have metadata(configuration files) of the input file structures. Can you please let me know that, having this metadata on HDFS is good or having the metadata in some sql/oracle database is good? Assume that I don’t have caching operator for now, I need to get the configuration file for each source and I have to parse them at run time. To do this going to HDFS is good or going to database is good? Regards, Surya Vamshi _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future. _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
