Hi Devendra, They change frequently, so I would prefer to use SQL database.
Thank you for the suggestion. I will use subject line accordingly. Regards, Surya Vamshi From: Yogi Devendra [mailto:[email protected]] Sent: 2016, August, 11 2:04 AM To: [email protected] Subject: Re: Information Needed Surya, How often will these metadata (configuration files) of input structures change? If they are not changed frequently then HDFS is fine. Otherwise, better to go for something which is more edit friendly (may be SQL as you suggested) or you can also consider options like FTP, NFS etc. ~ Yogi On 11 August 2016 at 11:29, Yogi Devendra <[email protected]<mailto:[email protected]>> wrote: @Surya On a side note, I have seen multiple emails with subject line "Information needed". I urge you to use more informative subject line for your subsequent posts. This would help future users to look through the mailing list archive. ~ Yogi On 11 August 2016 at 10:23, Priyanka Gugale <[email protected]<mailto:[email protected]>> wrote: IMO, it's okay to keep the files in HDFS, at startup you can load them in memory which will be one time activity in application lifetime. Considering they are format/config files, it won't be difficult to cache it in memory of your current operator (no need to caching operator separately). -Priyanka On Wed, Aug 10, 2016 at 7:06 PM, Mukkamula, Suryavamshivardhan (CWM-NR) <[email protected]<mailto:[email protected]>> wrote: Hello Team, I have a use case where I am trying to read number of input files with different formats(may be 100 source systems) and I have to parse each record. To achieve this we need to have metadata(configuration files) of the input file structures. Can you please let me know that, having this metadata on HDFS is good or having the metadata in some sql/oracle database is good? Assume that I don’t have caching operator for now, I need to get the configuration file for each source and I have to parse them at run time. To do this going to HDFS is good or going to database is good? Regards, Surya Vamshi _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future. _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
