Hi Oran, can you check your ES logs / metrics? Most issues we see with the ES sink are around incorrect batching and/or overloaded clusters. Could it be your ES write queue is building up?
On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <oran.shus...@houzz.com> wrote: > Flink version 1.13.1 > ES Version 7.12.0 > Flink deployment type - session cluster > Scala version 2.12 > > Same job used to run with elasticsearch6 connector with about the same > load and had no issues > Since moving to elasticsearch7 more and more exceptions were being thrown > about reactor being stopped in our logs > > [IllegalStateException] with message [Request cannot be executed; I/O > reactor status: STOPPED] with rest status -1 > > Looking online for this type of error gave me some results from people > instantiating the ES client on their own which is irrelevant for our case. > Also found that it might be caused by an uncaught exception in the failure > handler, so i wrapped all the code in try..catch but found no uncaught > exceptions > > Next thing I tried was to force using the updated version of ES rest > client with the same version as my cluster 7.12. That didn't seem to fix > any issues > > Problem is that once this happens it does not fix itself. That instance of > the sink will continuously get those exceptions until we restart the job. I > tried to look for any logs before that but couldn't find anything > -- [image: logo] <https://bigdataboutique.com/> Itamar Syn-Hershko CTO, Founder Email: ita...@bigdataboutique.com Website: https://bigdataboutique.com <https://twitter.com/synhershko> <https://github.com/synhershko> <https://www.linkedin.com/in/itamar-syn-hershko-78b25013/> <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>
