I have tried both log4j logger as well as System.out.println option but none of
these worked.
>From what I have seen so far is the Filesystem streaming connector classes are
>not packaged in the grand jar (flink-dist_2.10-1.1-SNAPSHOT.jar) that is
>copied under <FLINK_HOME>/build-target/lib location as part of Flink maven
>build step.
So, I manually copied (overwrite) the compiled class files from
org.apache.flink.streaming.connectors.fs package to the my "Flink job"
distribution jar (otherwise it was using standard jars that are defined as mvn
dependency in Articatory) and then uploaded the jar to Job Manager.
Am I missing something? How do I enable logging for the RollingSink class?
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-filesystem_2.11</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
On Tuesday, March 22, 2016 3:04 AM, Aljoscha Krettek <aljos...@apache.org>
wrote:
Hi,
how are you printing the debug statements?
But yeah all the logic of renaming in progress files and cleaning up after a
failed job happens in restoreState(BucketState state). The steps are roughly
these:
1. Move current in-progress file to final location
2. truncate the file if necessary (if truncate is not available write a
.valid-length file)
3. Move pending files to final location that where part of the checkpoint
4. cleanup any leftover pending/in-progress files
Cheers,
Aljoscha
> On 22 Mar 2016, at 10:08, Vijay Srinivasaraghavan
> <vijikar...@yahoo.com.INVALID> wrote:
>
> Hello,
> I have enabled checkpoint and I am using RollingSink to sink the data to HDFS
> (2.7.x) from KafkaConsumer. To simulate failover/recovery, I stopped
> TaskManager and the job gets rescheduled to other Taskmanager instance.
> During this momemnt, the current "in-progress" gets closed and renamed to
> part-0-1 from _part-0-1_in-progress.
> I was hoping to see the debug statement that I have added to "restoreState"
> method but none of my debug statement gets printed. I am not sure if the
> restoreState() method gets invoked during this scenario. Could you please
> help me understand the flow during "failover" scenario?
> P.S: Functionally the code appears to be working fine but I am trying to
> understand the underlying implementation details. public void
> restoreState(BucketState state)
> Regards
> Vijay
