Hi Piotr, What was the version you were using before 1.7.1? How do you deploy your cluster, e.g., YARN, standalone? Can you attach full TM and JM logs?
Best, Gary On Fri, Jan 18, 2019 at 3:05 PM Piotr Szczepanek <piotr.szczepa...@gmail.com> wrote: > Hello, > we have scenario with running Data Processing jobs that generates export > files on demand. Our first approach was using ClusterClient, but recently > we switched to REST API for job submittion. In the meantime we switched to > flink 1.7.1 and that started to cause a problems. > Some of our jobs are stuck, not processing any data. Task Managers have > info that Chain is switching to RUNNING, and then nothing happenes. > In TM's stdout logs we can see that for some reason log is cut, e.g.: > > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: RecordReader > initialized will read a total of 615 records. > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: at row 0. reading > next block > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: block read in memory > in 63 ms. row count = 615 > Jan 10, 2019 4:28:33 PM WARNING: > org.apache.parquet.hadoop.ParquetRecordReader: Can not initialize counter > due to context is not a instance of TaskInputOutputContext, but is > org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: RecordReader > initialized will read a total of 140 records. > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: at row 0. reading > next block > Jan 10, 2019 4:28:33 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordReader: block read in memory > in 2 ms. row count = 140 > Jan 10, 2019 4:28:33 PM WARNING: > org.apache.parquet.hadoop.ParquetRecordReader: Can not initialize counter > due to context is not a instance of TaskInputOutputContext, but is > org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl > Jan 10, 2019 4:28:33 PM INFO: or > > As you can see, last line is cut in the middle, and nothing happenes later > on. > None of counters ( records/bytes sent/read) are increased. > We switched debug on on both TMs and JM but only thing they are showing up > are sending heartbeats between each other. > Do you have any idea what could be a problem? and how we could deal with > them or at least try to investigate? Is there any timeout/config that we > could try to enable? >
