Thanks. I thought the purpose of below method was to supply that
information?
@Override
*public* *boolean* reachedEnd() *throws* IOException {
*logger*.info("Reached " + reached);
*return* reached;
}
On Wed, Aug 2, 2017 at 1:43 AM, Fabian Hueske <[email protected]> wrote:
> FileInputFormat cannot know about the reached variable that you added in
> your class. So there is no way it could reset it to false.
> An alternative implementation without overriding open() could be to change
> the reachedEnd method to check if the stream is still at offset 0.
>
> 2017-08-01 20:22 GMT+02:00 Mohit Anchlia <[email protected]>:
>
>> Thanks that worked. However, what I don't understand is wouldn't the open
>> call that I am inheriting have this logic already inbuilt? I am inheriting
>> FileInputFormat.
>>
>> On Tue, Aug 1, 2017 at 1:42 AM, Fabian Hueske <[email protected]> wrote:
>>
>>> An InputFormat processes multiple InputSplits. open() is called for each
>>> InputSplit.
>>> If you don't reset reached to false in open() you will only read a
>>> single (i.e., the first) InputSplit and skip all others.
>>>
>>> I'd override open as follows:
>>>
>>> public void open(FileInputSplit fileSplit) throws IOException {
>>> super.open();
>>> reached = false;
>>> }
>>>
>>> Cheers, Fabian
>>>
>>>
>>> 2017-08-01 8:08 GMT+02:00 Mohit Anchlia <[email protected]>:
>>>
>>>> I didn't override open. I am using open that got inherited from
>>>> FileInputFormat . Am I supposed to specifically override open?
>>>>
>>>> On Mon, Jul 31, 2017 at 9:49 PM, Fabian Hueske <[email protected]>
>>>> wrote:
>>>>
>>>>> Do you set reached to false in open()?
>>>>>
>>>>>
>>>>> Am 01.08.2017 2:44 vorm. schrieb "Mohit Anchlia" <
>>>>> [email protected]>:
>>>>>
>>>>> And here is the inputformat code:
>>>>>
>>>>> public class PDFFileInputFormat extends FileInputFormat<String> {
>>>>> /**
>>>>> *
>>>>> */
>>>>> private static final long serialVersionUID = -4137283038479003711L;
>>>>> private static final Logger logger = LoggerFactory
>>>>> .getLogger(PDFInputFormat.class.getName());
>>>>> private boolean reached = false;
>>>>> @Override
>>>>> public boolean reachedEnd() throws IOException {
>>>>> logger.info("called reached " + reached);
>>>>> // TODO Auto-generated method stub
>>>>> return reached;
>>>>> }
>>>>> @Override
>>>>> public String nextRecord(String reuse) throws IOException {
>>>>> logger.info("This is where you parse PDF");
>>>>> String content = new String(
>>>>> Files.readAllBytes(Paths.get(this.currentSplit.getPath()
>>>>> .getPath())));
>>>>> logger.info("Content " + content);
>>>>> reached = true;
>>>>> return content;
>>>>> }
>>>>> }
>>>>>
>>>>> On Mon, Jul 31, 2017 at 5:09 PM, Mohit Anchlia <[email protected]
>>>>> > wrote:
>>>>>
>>>>>> I have a very simple program that just reads all the files in the
>>>>>> path. However, flink is not working as expected.
>>>>>>
>>>>>> Everytime I execute this job I only see flink reading 2 files, even
>>>>>> though there are more in that directory. On closer look it appears that
>>>>>> it
>>>>>> might be related to:
>>>>>>
>>>>>> [flink-akka.actor.default-dispatcher-3] INFO
>>>>>> org.apache.flink.runtime.taskmanager.TaskManager - TaskManager has 2
>>>>>> task slot(s).
>>>>>>
>>>>>> My question is, isn't flink supposed to iterate over the directory
>>>>>> after those 2 slots become free again? I am assuming this problem is
>>>>>> caused
>>>>>> because there are only 2 slots.
>>>>>>
>>>>>>
>>>>>> Code ---
>>>>>>
>>>>>> PDFFileInputFormat format = new PDFFileInputFormat();
>>>>>> format.setFilePath(args[0]);
>>>>>> format.setNestedFileEnumeration(true);
>>>>>> logger.info("Number of splits " + format.getNumSplits());
>>>>>>
>>>>>> // logger.info(Paths.get(".").toAbsolutePath().normalize().toSt
>>>>>> ring());
>>>>>>
>>>>>> env.createInput(format, TypeInformation.of(StringValue
>>>>>> .class)).print();
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>