Surya, How often will these metadata (configuration files) of input structures change? If they are not changed frequently then HDFS is fine. Otherwise, better to go for something which is more edit friendly (may be SQL as you suggested) or you can also consider options like FTP, NFS etc.
~ Yogi On 11 August 2016 at 11:29, Yogi Devendra <[email protected]> wrote: > @Surya > > On a side note, I have seen multiple emails with subject line "Information > needed". > I urge you to use more informative subject line for your subsequent posts. > > This would help future users to look through the mailing list archive. > > ~ Yogi > > On 11 August 2016 at 10:23, Priyanka Gugale <[email protected]> wrote: > >> IMO, it's okay to keep the files in HDFS, at startup you can load them in >> memory which will be one time activity in application lifetime. Considering >> they are format/config files, it won't be difficult to cache it in memory >> of your current operator (no need to caching operator separately). >> >> -Priyanka >> >> On Wed, Aug 10, 2016 at 7:06 PM, Mukkamula, Suryavamshivardhan (CWM-NR) < >> [email protected]> wrote: >> >>> Hello Team, >>> >>> I have a use case where I am trying to read number of input files with >>> different formats(may be 100 source systems) and I have to parse each >>> record. >>> >>> To achieve this we need to have metadata(configuration files) of the >>> input file structures. Can you please let me know that, having this >>> metadata on HDFS is good or having the metadata in some sql/oracle database >>> is good? >>> >>> Assume that I don’t have caching operator for now, I need to get the >>> configuration file for each source and I have to parse them at run time. To >>> do this going to HDFS is good or going to database is good? >>> >>> Regards, >>> Surya Vamshi >>> >>> >>> _______________________________________________________________________ >>> >>> If you received this email in error, please advise the sender (by return >>> email or otherwise) immediately. You have consented to receive the attached >>> electronically at the above-noted email address; please retain a copy of >>> this confirmation for future reference. >>> >>> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur >>> immédiatement, par retour de courriel ou par un autre moyen. Vous avez >>> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à >>> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de >>> cette confirmation pour les fins de reference future. >>> >>> >> >
