Hi Ram,
I would like to add the parameters for each partition like below. Each operator
to be given with its own configuration file and source identifier. If there is
any other way please let me know ?
In my current definepartition() method , I am doing similarly like below, but I
have to add setter and getter methods in AbstractFileInputOperator class.
for (int j = 0; j < numDirs; ++j) {
int first = sliceFirstIndex[j];
int last = first + partitionCounts[j];
String dir = directories[j];
LOG.info("definePartitions: first = {}, last = {}, dir = {}", first,
last, dir);
for (int i = first; i < last; ++i) {
AbstractFileInputOperator<String> oper = cloneObject(kryo, this);
oper.setDirectory(dir);
oper.setSourceId(<sourceId>);
oper.setConfigFile(<fileName>);
//oper.setpIndex(i);
SlicedDirectoryScanner scn = (SlicedDirectoryScanner) scanners.get(i);
scn.setStartIndex(first);
scn.setEndIndex(last);
scn.setDirectory(dir);
oper.setScanner(scn);
newPartitions.add(new DefaultPartition<>(oper));
newManagers.add(oper.getIdempotentStorageManager());
}
}
Regards,
Surya Vamshi
From: Munagala Ramanath [mailto:[email protected]]
Sent: 2016, June, 28 2:03 PM
To: [email protected]
Subject: Re: Reading Multiple directories in parallel
Not sure I fully understand the question but you can add whatever fields you
need
to your class that extends AbstractFileInputOperator. For example,
https://github.com/DataTorrent/examples/blob/master/tutorials/fileIO-multiDir/src/main/java/com/example/fileIO/FileReaderMultiDir.java
defines fields directories and partitionCounts.
You can then set these fields as needed in definePartitions.
Ram
On Tue, Jun 28, 2016 at 10:31 AM, Mukkamula, Suryavamshivardhan (CWM-NR)
<[email protected]<mailto:[email protected]>>
wrote:
Hi Ram,
Can you please suggest , how would I add another variable (like ‘directory’)
while creating multiple partitions of AbstractFileInputOperator in the define
partition method.
I have currently added variables in the AbstractFileInputOperator , which I
guess not a better way.
These variables are basically used to scan directories in parallel differently.
Regards,
Surya Vamshi
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email
or otherwise) immediately. You have consented to receive the attached
electronically at the above-noted email address; please retain a copy of this
confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email
or otherwise) immediately. You have consented to receive the attached
electronically at the above-noted email address; please retain a copy of this
confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
