Hi,
what where the steps you took? By the way, are you running this on yarn or in
standalone mode? How are you starting the Flink job? Do you still don’t see
DEBUG entries in the log?
Cheers,
Aljoscha
> On 23 Mar 2016, at 14:32, Vijay <vijikar...@yahoo.com> wrote:
>
> I have changed the properties file but it did not help.
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
>> On Mar 23, 2016, at 5:39 AM, Aljoscha Krettek <aljos...@apache.org> wrote:
>>
>> Ok, then you should be able to change the log level to DEBUG in
>> conf/log4j.properties.
>>
>>> On 23 Mar 2016, at 12:41, Vijay <vijikar...@yahoo.com> wrote:
>>>
>>> I think only the ERROR category gets displayed in the log file
>>>
>>> Regards,
>>> Vijay
>>>
>>> Sent from my iPhone
>>>
>>>> On Mar 23, 2016, at 2:30 AM, Aljoscha Krettek <aljos...@apache.org> wrote:
>>>>
>>>> Hi,
>>>> are you seeing the regular log output from the RollingSink in the
>>>> TaskManager logs?
>>>>
>>>> Cheers,
>>>> Aljoscha
>>>>> On 22 Mar 2016, at 20:03, Vijay Srinivasaraghavan <vijikar...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>> I have tried both log4j logger as well as System.out.println option but
>>>>> none of these worked.
>>>>>
>>>>> From what I have seen so far is the Filesystem streaming connector
>>>>> classes are not packaged in the grand jar
>>>>> (flink-dist_2.10-1.1-SNAPSHOT.jar) that is copied under
>>>>> <FLINK_HOME>/build-target/lib location as part of Flink maven build step.
>>>>>
>>>>> So, I manually copied (overwrite) the compiled class files from
>>>>> org.apache.flink.streaming.connectors.fs package to the my "Flink job"
>>>>> distribution jar (otherwise it was using standard jars that are defined
>>>>> as mvn dependency in Articatory) and then uploaded the jar to Job Manager.
>>>>>
>>>>> Am I missing something? How do I enable logging for the RollingSink class?
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.flink</groupId>
>>>>> <artifactId>flink-connector-filesystem_2.11</artifactId>
>>>>> <version>${flink.version}</version>
>>>>> <scope>provided</scope>
>>>>> </dependency>
>>>>>
>>>>>
>>>>> On Tuesday, March 22, 2016 3:04 AM, Aljoscha Krettek
>>>>> <aljos...@apache.org> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>> how are you printing the debug statements?
>>>>>
>>>>> But yeah all the logic of renaming in progress files and cleaning up
>>>>> after a failed job happens in restoreState(BucketState state). The steps
>>>>> are roughly these:
>>>>>
>>>>> 1. Move current in-progress file to final location
>>>>> 2. truncate the file if necessary (if truncate is not available write a
>>>>> .valid-length file)
>>>>> 3. Move pending files to final location that where part of the checkpoint
>>>>> 4. cleanup any leftover pending/in-progress files
>>>>>
>>>>> Cheers,
>>>>> Aljoscha
>>>>>
>>>>>> On 22 Mar 2016, at 10:08, Vijay Srinivasaraghavan
>>>>>> <vijikar...@yahoo.com.INVALID> wrote:
>>>>>>
>>>>>> Hello,
>>>>>> I have enabled checkpoint and I am using RollingSink to sink the data to
>>>>>> HDFS (2.7.x) from KafkaConsumer. To simulate failover/recovery, I
>>>>>> stopped TaskManager and the job gets rescheduled to other Taskmanager
>>>>>> instance. During this momemnt, the current "in-progress" gets closed and
>>>>>> renamed to part-0-1 from _part-0-1_in-progress.
>>>>>> I was hoping to see the debug statement that I have added to
>>>>>> "restoreState" method but none of my debug statement gets printed. I am
>>>>>> not sure if the restoreState() method gets invoked during this scenario.
>>>>>> Could you please help me understand the flow during "failover" scenario?
>>>>>> P.S: Functionally the code appears to be working fine but I am trying to
>>>>>> understand the underlying implementation details. public void
>>>>>> restoreState(BucketState state)
>>>>>> Regards
>>>>>> Vijay
>>
