Re: Small-files source - partitioning based on prefix of file

Thanks Vino. Yes, I can do that after the source function. But that means data would be shuffled - sending from every source to the right partition. I think that by doing the partition from within the file source would help to save that shuffling. Thanks. Averell. -- Sent from: http://apache

Small-files source - partitioning based on prefix of file

und-robin. However, I could not find the call of "rebalancing" method, but "transform" is called. Not much information about that "transform" method though. Would it possible for me to ask for some guideline on this? Thanks for your help. Averell -- Sent from: http

Re: Small-files source - partitioning based on prefix of file

Thank you Vino. Yes, I went thru that official guide before posting this question. The problem was that I could not see any call to one of those mentioned partitioning methods (partitionCustom, shuffle, rebalance, rescale, or broadcast) in the original readFile function. I'm still trying to look

Re: S3 file source - continuous monitoring - many files missed

Here is my https://github.com/lvhuyen/flink implementation of the change. 3 files were updated: StreamExecutionEnvironment.java, StreamExecutionEnvironment.scala, and ContinuousFileMonitoringFunction.java. All the thanks to Fabian. -- Sent from:

Re: Small-files source - partitioning based on prefix of file

Oh, Thank you Vino. I was not aware of that reshuffling after every custom partitioning. Why would that needed though? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

Hi Vino, I'm a little bit confused. If I can do the partitioning from within the source function, using the same hash function on the key to identify the partition, would that be sufficient to avoid shuffling in the next byKey call? Thanks. Averell -- Sent from: http://apache-flink-user

Re: Detect late data in processing time

Hi Soheil, Why don't you just use the processing time as event time? Simply overriding extractTimestamp to return your processing time. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

that yet. Thank you very much for your support. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

Thank you Fabian. "/In either case, some record will be read twice but if reading position can be reset, you can still have exactly-once state consistency because the state is reset as well./" I do not quite understand this statement. If I have read 30 lines from the checkpoint and sent those 30

Re: CoFlatMapFunction with more than two input streams

ding using Either wrapper, as my understanding, I would need to use that both in my sources (stream_A and B) and in the CoProcess/CoFlatMapFunction. Then using a super class Animal would be more convenient, wouldn't it? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-

Re: CoFlatMapFunction with more than two input streams

cat").map(_ match { case Left(c) = c case _ = NON_EXIST_CAT }) Thanks! Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

CoFlatMapFunction with more than two input streams

ent on duplicating the CoFlatMapFunction (*). With option 2, there's additional cost coming from unioning, splitting/selecting, and type-casting at the final streams. Is there any better option for me? Thank you very much for your support. Regards, Averell (*) I am using Scala, and I tried to create

Initial value of ValueStates of primitive types

Hi, In Flink's documents, I couldn't find any example that uses primitive type when working with States. What would be the initial value of a ValueState of type Int/Boolean/...? The same question apply for MapValueState like [String, Int] Thanks and regards, Averell -- Sent from: http

Re: Initial value of ValueStates of primitive types

conversion: val y:Integer = getRuntimeContext.getState(new ValueStateDescriptor[Int]("Triggered", classOf[Int])).value() >> y = {Integer@5795} "0" Thanks for your time. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

Thank you Fabian. It is clear to me now. Thanks a lot for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

Thank you Vino and Fabien for your help in answering my questions. As my files are small, I think there would not be much benefit in checkpointing file offset state. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

/stream/state/state.html> . However, I have not been able to find the information I am looking for. Please help point me to the right place. Thanks and best regards, Averell. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Can I only use checkpoints instead of savepoints in production?

of: (1) flink application killed (2) cluster crashed (3) one of the servers in the cluster crashed (4) unhandled exception raised when abnormal data received ... Could you please help explain? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.

Re: Small-files source - partitioning based on prefix of file

Thank you Fabian. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

per minute) * I set the checkpointing interval = 1 minute. In this example, /will the 1st barrier be injected into the stream of file-splits 50 seconds after the 10,000th split, or after the 2,000th one?/ Sorry for being confusing. Thanks and best regards, Averell -- Sent from: http

Re: Can I only use checkpoints instead of savepoints in production?

Thank you Vino. I sometimes got the error message like the one below. It looks like my executors got overloaded. Here I have another question: is there any existing solution that allows me to have the job restored automatically? Thanks and best regards, Averell -- Sent from: http://apache

Re: Can I only use checkpoints instead of savepoints in production?

automatically. I am trying to test my scenarios now. Found some issues, and I think it would be better to ask in a separate thread. Thanks and regards, Averell = org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 457d8f370ef8a50bb462946e1f12b80e

Re: Can I only use checkpoints instead of savepoints in production?

Hi Vino, Could you please tell where I should find the JM and TM logs? I'm running on an AWS EMR using yarn. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Small-files source - partitioning based on prefix of file

e expected behaviour, as my current checkpoint config is to checkpoint every 10s, and it took only a second or two for the listing of those 20K files. Am I correct here? And do we have a solution for this? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.

Raising a bug in Flink's unit test scripts

-9940 bug fix, or I have to raise a separated bug? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

S3 file source - continuous monitoring - many files missed

Good day everyone, I have a Flink job that has an S3 folder as a source, and we keep putting thousands of small (around 1KB each) gzip files into that folder, with the rate of about 5000 files per minute. Here is how I created that source in Scala: / val my_input_format = new

Re: S3 file source - continuous monitoring - many files missed

Hi Jörn, Thanks. I had missed that EMRFS strong consistency configuration. Will try that now. We also had a backup solution - using Kinesis instead of S3 (I don't see Kinesis in your suggestion, but hope that it would be alright). "/The small size and high rate is not suitable for S3 or HDFS/"

Re: S3 file source - continuous monitoring - many files missed

Just some update: I tried to enable "EMRFS Consistent View" option, but it didn't help. Not sure whether that's what you recommended, or something else. Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: S3 file source - continuous monitoring - many files missed

Could you please help explain more details on "/try read after write consistency (assuming the files are not modified) /"? I guess that the problem I got comes from the inconsistency in S3 files listing. Otherwise, I would have got exceptions on file not found. My use case is to read output

Re: S3 file source - continuous monitoring - many files missed

Thank you Fabian. I tried to implement a quick test basing on what you suggested: having an offset from system time, and I did get improvement: with offset = 500ms - the problem has completely gone. With offset = 50ms, I still got around 3-5 files missed out of 10,000. This number might come

Re: S3 file source - continuous monitoring - many files missed

Hello Fabian, I created the JIRA bug https://issues.apache.org/jira/browse/FLINK-9940 BTW, I have one more question: Is it worth to checkpoint that list of processed files? Does the current implementation of file-source guarantee exactly-once? Thanks for your support. -- Sent from:

Re: S3 file source - continuous monitoring - many files missed

Thank you Fabian for the guide to implement the fix. I'm not quite clear about the best practice of creating JIRA ticket. I modified its priority to Major as you said that it is important. What would happen next with that issue then? Someone (anyone) will pick it and create a fix, then include

Re: S3 file source - continuous monitoring - many files missed

Hello Jörn. Thanks for your help. "/Probably the system is putting them to the folder and Flink is triggered before they are consistent./" <<< yes, I also guess so. However, if Flink is triggered before they are consistent, either (a) there should be some error messages, or (b) Flink should be

ElasticSearch 6 - error with UpdateRequest

Indexer: RequestIndexer): Unit = { requestIndexer.add(upsertRequest(element)) } What could be the issue here? Thanks for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearch 6 - error with UpdateRequest

project? If you have some spare time, please help explain. I also would like to know the other way to fix that issue (that you implemented in your branch). Thanks a lot for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

ot; >From Flink GUI, all checkpoints were shown as completed successfully. How could I debug further? Thanks a lot for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

typeInfo, reader);/ Does this create two different operators? If yes, then it seems impossible to assign a UID to the 1st operator. And might it be the cause for my problem? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: When does Trigger.clear() get called?

Thank you Fabian. All my doubts are cleared now. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

that I notice is the error doesn't stay on one single operator but changes from time to time (even within the same build). For example, the previous exception I quoted was with a Window operator, while the one below is with CoStreamFlatMap. Thanks and best regards, Averell Caused

Re: Error restoring from savepoint while there's no modification to the job

s. However, as I mentioned from the 1st email, I got errors when restoring savepoint created by the same version of my application. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Identifying missing events in keyed streams

Thank you Fabian. Tried (2), and it's working well. I found one more benefit of (2) over (3) is that it allow me to easily raise multiple levels of alarms for each keyed stream (i.e: minor: missed 2 cycles, major: missed 5 cycles,...) Thanks for your help. Regards, Averell -- Sent from: http

Re: When does Trigger.clear() get called?

. And one related question: for keyed streams, if I know that some keys would never have new events anymore, should/could I remove those streams corresponding to those keys so that I can save some memory allocated to the metadata? Thanks and best regards, Averell -- Sent from: http://apache-flink

Re: When does Trigger.clear() get called?

clear those streams? Thank you very much for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

flink/flink-docs-master/dev/connectors/streamfile_sink.html> , buck-enconding can only combined with OnCheckpointRollingPolicy, which rolls on every checkpoint. So setting that CheckInterval makes no difference. So why should we expose that withBucketCheckInterval method? Thanks and be

Re: Streaming to Parquet Files in HDFS

ready-to-use solution that supports copying/moving file from HDFS to S3 (something like a trigger from Flink after it has finished writing to HDFS). Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

What a great news. Thanks for that, Kostas. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

to HADOOP configurations? Thanks and best regards, Averell java.lang.Exception: unable to establish the security context at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:73) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1118) Caused

Error restoring from savepoint while there's no modification to the job

please help give a look? Thanks and best regards, Averell taskmanager.gz <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/taskmanager.gz> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 606ad5239f5e23cedb85d3e75b

Re: Error restoring from savepoint while there's no modification to the job

multiple times), and had not been moved. Is there any kind of improper user code can cause such error? Thanks for your support. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

ing as keys are of types either String or Long. For this, I don't have to define equals and hashcode method, do I? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

Hi Kostas, No, the same code was used. I (1) started the job, (2) created a savepoint, (3) cancelled the job, (4) restored the job with the same command as in (1) with the addition "-s ". Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Metrics: (1) Histogram in prometheus and (2) meter by event_time

toring one) will be separated? Is that possible to disable checkpointing for that 2nd branch? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

t=2018-09-20/part-7-5: org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool/ 4. Where is the temporary folder that you store the parquet file before uploading to S3? Thanks a lot for your help. Best rega

Re: Identifying missing events in keyed streams

to the minimum level. Could you please explain why (2) is better? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Difference between BucketingSink and StreamingFileSink

to parquet? Is that possible to partition my output basing on event-time? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Difference between BucketingSink and StreamingFileSink

Hi, https://issues.apache.org/jira/browse/FLINK-9749 <<< as per this ticket, StreamingFileSink is a newer option, which is better than BucketingSink for Parquet. Would love to see some example one using that. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mai

Re: Difference between BucketingSink and StreamingFileSink

rmat(new java.util.Date(in.getTimestamp))}" } override def getSerializer = SimpleVersionedStringSerializer.INSTANCE } Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Identifying missing events in keyed streams

defining patterns of missing events. Could you please give some advices? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Utilising EMR's master node

Hi Gary, Thanks for the information. I didn't know that -yn is obsolete :( I am using Flink 1.6. Not sure whether that's a bug when I tried to set -yn explicitly, but I started only 1 cluster. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4

"unable to establish the security context" with shaded Hadoop S3

stacktrace below. The shading of hadoop jars started from this ticket FLINK-10366 <https://issues.apache.org/jira/browse/FLINK-10366> . Googling the error didn't help. Could someone please help me? Thanks and best regards, Averell /Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no H

Re: Streaming to Parquet Files in HDFS

closing in-progress part file for bucket id=dt=2018-09-22 on checkpoint. 2018-10-06 14:40:18.254 [Sink: Unnamed (1/1)] DEBUG o.a.flink.streaming.api.functions.sink.filesystem.Buckets - Subtask 0 checkpointing: BucketState for bucketId=dt=2018-09-22 and bucketPath=s3a://assn-averell/Test/output/dt

Promethus - custom metrics at job level

-flink-user-mailing-list-archive.2336050.n4.nabble.com/Service-discovery-for-Prometheus-on-YARN-td21988.html, which looks like a "no" answer to my question. However, I still hope for some easy to use solution. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-ma

Re: Promethus - custom metrics at job level

Thank you Reza. I will try your repo first :) Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Utilising EMR's master node

Hi Gary, Thanks for your help. Regarding TM configurations, in term of performance, when my 2 servers have 16 vcores each, should I have 2 TMs with 16GB mem, 16 task slots each, or 8 TMs with 4GB mem and 4 task slots each? Thanks and regards, Averell -- Sent from: http://apache-flink-user

Re: Strange behaviour with checkpointing and custom FilePathFilter

Please refer to this version: === import java.util.Date import org.apache.flink.api.common.io.FilePathFilter import org.apache.flink.core.fs.Path import org.slf4j.LoggerFactory object SdcFilePathFilter { private val TIME_FORMAT = new java.text.SimpleDateFormat("MMdd

Re: Strange behaviour with checkpointing and custom FilePathFilter

on at the global level would be too heavy. Thanks and regards, Averell == import java.util.Date import org.apache.flink.api.common.io.FilePathFilter import org.apache.flink.core.fs.Path import org.slf4j.LoggerFactory object SdcFilePathFilter { private val TIME_FORMAT = new

Strange behaviour with checkpointing and custom FilePathFilter

monitor generates 0 records during that 15-20 minutes period) Could someone please help with this? Thank you very much for your time. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Metrics: (1) Histogram in prometheus and (2) meter by event_time

ogram.png> I'm using Prometheus with Grafana. Is that possible to do what I mentioned? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Strange behaviour with checkpointing and custom FilePathFilter

to ongoing checkpoint. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Scheduling sources

give some help? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Strange behaviour with checkpointing and custom FilePathFilter

this 30s timer? Thanks and regards, Averell 2018-09-25 12:01:13.222 [Canceler/Interrupts for Source: Custom File Source (1/1) (a5f5434070044510eafc9103bc24af43).] WARN org.apache.flink.runtime.taskmanager.Task - Task 'Source: Custom File Source (1/1)' did not react to cancelling signal for 30

Re: Strange behaviour with checkpointing and custom FilePathFilter

, checkpointing process is not triggered though. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Scheduling sources

) gets enriched properly, I want to have (2a) loaded properly into memory before starting to process (1). Is there any walkaround solution for me in this case? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Utilising EMR's master node

configurations then still 4 TMs created, but the occupied ones are always on two different servers. I'm not sure whether that's EMR's issue, or YARN's or Flink's. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Scheduling sources

Hi Kostas, So that means my 2a will be broadcasted to all TMs? Is that possible to partition that? As I'm using CoProcessFunction to join 1 with 2. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Scheduling sources

Hi Tison, "/setting a timer on schedule start and trigger (1) after a fixed delay/" would be quite sufficient for me. Looking forward to the change of that Jira ticket's status. Thanks for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.

Re: Strange behaviour with checkpointing and custom FilePathFilter

help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Metrics: (1) Histogram in prometheus and (2) meter by event_time

e-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/Histo2.png> Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

No resource available error while testing HA

ed, or a new JobManager will try to connect to the running TMs to resume the job? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: No resource available error while testing HA

to zookeeper and the problem was solved. Then I have another question: when JM cannot start/connect to the JM on .88, why didn't it try on .82 where resource are still available? Thanks and regards, Averell Here is the JM log (from /mnt/var/log/hadoop-yarn/.../jobmanager.log on .82) (it seems

Logging metrics from within Elasticsearch ActionRequestFailureHandler

solution. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Logging metrics from within Elasticsearch ActionRequestFailureHandler

and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Utilising EMR's master node

least for the JM only? If that is not possible, should I have different hardware configurations between the master node and core nodes (smaller server for the master)? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Utilising EMR's master node

that be an expected behaviour? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

S3 parquet sink - failed with S3 connection exception

checkpoint, but it could not make any further checkpoint - all subsequent checkpoints failed with the same reason. Searching on Internet I could only find one explanation: S3Object has not been closed properly. Could someone please help? Thanks and regards, Averell /The program finished

Re: Broadcast state before events stream consumption

ate is kept in-memory at runtime and memory provisioning should be done accordingly. This holds for all operator states."/ I am using RocksDB state backend, and is confused by that statement and yours. Could you please help clarify? Thanks and regards, Averell -- Sent from: http://apache-fl

Re: Stream enrichment with static data, side inputs for DataStream

for every record from the main stream, you read the enrichment data from the saved ValueState to enrich that mainstream record. If no, then I am having the same issue :D Looking at Broadcast State, but there is still something that doesn't look right for me. Regards, Averell -- Sent from: htt

Re: S3 parquet sink - failed with S3 connection exception

that broadcast stream (as mentioned in the document, it doesn't use RocksDB). But not quite sure. Thanks and regards, Averell logs.gz <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/logs.gz> -- Sent from: http://apache-flink-user-mailing-list-archive.2336

Re: S3 parquet sink - failed with S3 connection exception

that Flink GUI's Exception tab is reading from? Thanks and regards, Averell java.lang.ArrayIndexOutOfBoundsException: 122626 at org.apache.parquet.column.values.dictionary.DictionaryValuesWriter$PlainLongDictionaryValuesWriter.fallBackDictionaryEncodedData(DictionaryValuesWriter.java:397

Where does the logs in Flink GUI's Exception tab come from?

Hi everyone, I am running Flink in EMR YARN cluster, and when the job failed and restarted, I could see some logs in the Exception tab of Flink GUI. I could not find this piece of

Re: S3 parquet sink - failed with S3 connection exception

ink yet. * reduced the parallelism for my S3 continuous files reader. However, the problem still randomly occurred (random by job executions. When it occurred, the only solution is to cancel the job and restart from the last successful checkpoint). Thanks and regards, Averell [1] Hadoop-AWS

Re: No resource available error while testing HA

and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Thank you Gordon. That's my exact problem. Will try the fix in 1.7.2 now. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Gordon, Sorry for a noob question: How can I get the RC 1.7.2 build / code to build? I could not find any branch like that in Github. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

doesn't have that new class. Maybe Gordon meant 1.7.2 rc2? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

solution is to use the version field of each ER request - increase it for every time I retried putting the request into the queue. This works for me until now, but it doesn't look right. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How to use ExceptionUtils.findThrowable() in Scala

flict_engine_exception, reason=[_doc][...]: version conflict, document already exists (current version [1])] / Thanks and best regards, Averell [1] handling-failing-elasticsearch-requests <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html#handli

Re: No resource available error while testing HA

Hi Gary, I am trying to reproduce that problem. BTW, is that possible to change log level (I'm using logback) for a running job? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to use ExceptionUtils.findThrowable() in Scala

P/S: This is the full stack trace 2019-02-07 01:53:12.790 [I/O dispatcher 16] ERROR o.a.f.s.connectors.elasticsearch.ElasticsearchSinkBase - Failed Elasticsearch item request: [...][[...][1]] ElasticsearchException[Elasticsearch exception [type=version_conflict_engine_exception,

  1   2   >