Re: Job is still in ACTIVE state after /jobs/:jobid/stop

I have some updates. Some weird behaviours were found. Please refer to the attached photo. All requests were sent via REST API The status of the savepoint triggered by that stop request (ID 11018) is "COMPLETED [Savepoint]", however, no checkpoint data has been persisted (in S3). The folder

Job is still in ACTIVE state after /jobs/:jobid/stop

now": 1605182656731, "timestamps": { "CANCELLING": 0, "FAILING": 0, "CANCELED": 0, "FINISHED": 0, "RUNNING": 1604016319495, "FAILED": 0, "RESTARTING": 0, "CREATED": 160401631

Re: HA on AWS EMR

Hello Robert, Thanks for the info. That makes sense. I will save and cancel my jobs with 1.10, upgrade to 1.11, and restore the jobs from the savepoints. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: KryoException UnsupportedOperationException when writing Avro GenericRecords to Parquet

Hello Till, Adding GenericRecordAvroTypeInfo(schema) does help. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

KryoException UnsupportedOperationException when writing Avro GenericRecords to Parquet

th both Flink 1.10.0 and 1.11.0, and currently stuck at this. Could you please help? Thanks and regards, Averell /com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException Serialization trace: reserved (org.apache.avro.Schema$Field) fieldMap (org.apache.avro.Schema

Re: HA on AWS EMR

messages attached below. Not sure that's a bug or expected behaviour. Thanks and best regards, Averell /07:39:33.906 [main-EventThread] ERROR org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState - Authentication failed 07:40:11.585 [flink-akka.actor.default-dispatcher-2

Re: HA on AWS EMR

dataDir, which is by default stored in the local storage of the EMR's master node. I'm trying to move this one to an EFS, in hope that it would help. Not sure whether this is a right approach. Thanks for your help. Regards, Averell [1] http://apache-flink-user-mailing-list-archive.2336050.n

HA on AWS EMR

I miss something here? Thanks for your help. Regards, Averell [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: SAX2 driver class org.apache.xerces.parsers.SAXParser not found

Averell <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/FlinkFileSink.png> -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: SAX2 driver class org.apache.xerces.parsers.SAXParser not found

. However, intermittenly my checkpoints still fail (about 10%). And whenever it fails, there are non-completed files left in S3 (attached screenshot below). I'm not sure whether those uncompleted files are expected, or is that a bug? Thanks and regards, Averell <http://apache-flink-user-mailing-l

Re: JSON to Parquet

now. Thanks for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

SAX2 driver class org.apache.xerces.parsers.SAXParser not found

Hello, I have a Flink 1.10 job which runs in AWS EMR, checkpointing to S3a as well as writing output to S3a using StreamingFileSink. The job runs well until I add the Java Hadoop properties: /-Dfs.s3a.acl.default= BucketOwnerFullControl/. Since after that, the checkpoint process fails to

JSON to Parquet

-> AVRO (GenericRecord) -> Parquet. Is that an option? I would want to be able to quickly/dynamically (as less code change as possible) change the JSON schema. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Automatically resuming failed jobs in K8s

Thank you very much, Yang. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Automatically resuming failed jobs in K8s

the job, I need to provide the full path of the last checkpoint (/s3:chk-2345//). Is there any option to just provide the base_path? 3. Store the info to restore the jobs in the K8s deployment config Thanks a lot. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive

Re: Flink Metrics in kubernetes

Hi Gary, Sorry for the false alarm. It's caused by a bug in my deployment - no metrics were added into the registry. Sorry for wasting your time. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Metrics in kubernetes

.n4.nabble.com/file/t1586/k8xDump.txt> . Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Flink Metrics in kubernetes

need to care for when running in Kubernetes? Thanks a lot. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: K8s native - checkpointing to S3 with RockDBStateBackend

Hi David, Yang, Thanks. But I just tried to submit the same job on a YARN cluster using that same uberjar, and it was successful. I don't have flink-s3-fs-hadoop.jar anywhere in the lib or plugin folder. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive

Re: K8s native - checkpointing to S3 with RockDBStateBackend

Thank you Yun Tang. Building my own docker image as suggested solved my problem. However, I don't understand why I need that while I already had that s3-hadoop jar included in my uber jar? Thanks. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4

Re: Question about Scala Case Class and List in Flink

O fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance./" I imported /org.apache.flink.streaming.api.scala._/ << is this enough to tell that I am using Scala API? Thanks and regar

K8s native - checkpointing to S3 with RockDBStateBackend

(is it already too late? should that be there before the JM is started?) Thanks for your help. Averell / Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to create checkpoint storage at checkpoint coordinator side. at org.apache.flink.runtime.checkpoint.CheckpointCoordinator

Re: Change to StreamingFileSink in Flink 1.10

Thanks @Seth Wiesman and all. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Change to StreamingFileSink in Flink 1.10

Hello Leonard, Sivaprasanna, But my code was working fine with Flink v1.8. I also tried with a simple String DataStream, and got the same error. /StreamingFileSink .forRowFormat(new Path(path), new SimpleStringEncoder[String]())

Re: Change to StreamingFileSink in Flink 1.10

)*.asInstanceOf[RowFormatBuilder[IN, String, _]]* .build()/ Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Change to StreamingFileSink in Flink 1.10

, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Change to StreamingFileSink in Flink 1.10

der[_]] .withRollingPolicy(...) returns a RowFormatBuilder[_] .withBucketAssigner(...) returns Any/ I'm using Maven 3.6.0, Java 1.8.0_242, and Scala 2.11.12. Tried with/without IntelliJ, no difference. Not sure/understand what's wrong Thanks! Averell -- Sent from: http://apache-flink-user-mailing-l

Re: Preserving (best effort) messages order between operators

. Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to stream intermediate data that is stored in external storage?

se options. Writing a custom sink function to write to your own SQL server is also a not-so-difficult solution. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to stream intermediate data that is stored in external storage?

Hi Kant, I wonder why you need to "source" your intermediate state from files? Why not "source" it from the previous operator? I.e. instead of (A join B) -> State -> files -> (C), why not do (A join B) -> State -> (files + C)? -- Sent from:

Preserving (best effort) messages order between operators

uld throttle the 1st operator when back-pressure is high, then I could mitigate the mentioned problem. But I could not find any guide on doing that. Could you please help? Thanks. Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How to generate a sequential watermark which increases by one unit each time

for each file. I thought of extending the AssignerWithPeriodicWatermarks interface with a member variable holding that sequence value. However, it seems to me that it is not possible to persist that value during checkpoints. Are there any options for me? Thanks and best regards, Averell -- Sent

Re: How to export all not-null keyed ValueState

Thank you very much, Fabian. Regards, Averell On Fri, May 10, 2019 at 9:46 PM Fabian Hueske wrote: > Hi Averell, > > I'd go with your approach any state access (given that you use RocksDB > keyed state) or deduplication of messages is going to be more expensive > than a simple

Re: How to export all not-null keyed ValueState

. There's no use of applyToKeyedState in this implementation. And the problem I got is I received duplicated output (one from each parallelism-instance). Is there any option to modify the keyed state from within the processBroadcastElement() method? Thanks a lot for your help. Regards, Averell

Re: How to export all not-null keyed ValueState

collector: Collector[MyEvent]): Unit = { context.applyToKeyedState(toggleStateDescriptor, (k: Key, s: ValueState[Boolean]) => if (s != null) context.output(outputTag, (k, s.value( } } Thanks for your help. Regards, Averell On Thu, May 9, 2019 at 7:31 PM Fabian H

Re: How to export all not-null keyed ValueState

that RichCoFlatMapFunction with a new KeyedBroadcastProcessFunction, which has both functionalities: filter and export? Doing this would require unioning Toggle and Data into one single keyed stream. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How to export all not-null keyed ValueState

regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: I want to use MapState on an unkeyed stream

>From my understanding, having a fake keyBy (stream.keyBy(r => "dummyString")) means there would be only one slot handling the data. Would a broadcast function [1] work for your case? Regards, Averell [1] https://ci.apache.org/projects/flink/flink-docs-stable/

IllegalArgumentException with CEP & reinterpretAsKeyedStream

this error, and not sure whether I should use keyed streams with CEP? Thanks and best regards, Averell My code: / val cepInput = streamA.keyBy(r => (r.id1, r.id2)) .connect(streamB.keyBy(r => (r.id1, r.id2))) .flatMap(new MyCandidateFilterFu

Re: CoProcessFunction vs Temporal Table to enrich multiple streams with a slow stream

that you mentioned, is there any summary page in Flink docs for that? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

CoProcessFunction vs Temporal Table to enrich multiple streams with a slow stream

this way of implementation (using Flink Table) be better than the option no.1 mentioned in my other thread: creating two different (though similar) CoProcessFunction's, maintaining two state tables (for the enrichment stream, one in each function)? Thanks and best regards, Averell [1] https

Re: Timestamp and key preservation over operators

Thank you Fabian. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Timestamp and key preservation over operators

. Is there any way to accomplish that? Currently, I have an assignTimestampsAndWatermarks after my window function, but, as you said, it is against the best practice. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Timestamp and key preservation over operators

, but the watermark stays at 10:00) Thus, my question: what is the easiest way to check the timestamp of a message? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Timestamp and key preservation over operators

MARK for those idle sources. Not sure whether doing that is recommended? Thanks for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Timestamp and key preservation over operators

Could you please recommend a solution/good-practice here? I have one more question about the recommendation [2] to emit timestamp and watermark from within the source function. Is there any way to do that with the file sources? Thanks and best regards, Averell [1] https://ci.apache.org/projects/f

Timestamp and key preservation over operators

), and I have been trying to use DataStreamUtils.reinterpretAsKeyedStream. Is it safe to assume that as long as I dont do transformation on key, I could use that reinterpretAsKeyedStream function? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050

Sending FileInputSplit to the next operator

point. Could you please help tell me the wrong in that 2nd implementation? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: FileInputFormat that processes files in chronological order

FileInputFormat has anything to do here. Use that when your files are in a format not currently supported by Flink. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Serialising null value in case-class

Thank you Timo. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Write batch function for apache flink

Hi Anurag, Something like this one: https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/ Is it what you are looking for? Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Serialising null value in case-class

many places in Flink documents that Java primitive types are recommended. But how are Scala types? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Identify orphan records after joining two streams

use the out of the box function sideOutputLateData() Not sure whether I would really be benefited from that. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Serialising null value in case-class

to hold a value./ I am confused. Why there's the difference between a null String and a null Integer? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Identify orphan records after joining two streams

this apply to the low-level-API functions as well? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Identify orphan records after joining two streams

-of-memory. Would back-pressure kicks in for this case? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: No resource available error while testing HA

Hi Gary, Thanks for the answer. I missed your most recent answer in this thread too. However, my last question Averell wrote > How about changing the configuration of the Flink job itself during > runtime? > What I have to do now is to take a savepoint, stop the job, change the > c

Re: Where does the logs in Flink GUI's Exception tab come from?

Hi Gary, Thanks a lot for the explanation, and sorry for missing your earlier message. I am clear now. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Where does the logs in Flink GUI's Exception tab come from?

Hi everyone, I am running Flink in EMR YARN cluster, and when the job failed and restarted, I could see some logs in the Exception tab of Flink GUI. I could not find this piece of

Re: S3 parquet sink - failed with S3 connection exception

that Flink GUI's Exception tab is reading from? Thanks and regards, Averell java.lang.ArrayIndexOutOfBoundsException: 122626 at org.apache.parquet.column.values.dictionary.DictionaryValuesWriter$PlainLongDictionaryValuesWriter.fallBackDictionaryEncodedData(DictionaryValuesWriter.java:397

Re: S3 parquet sink - failed with S3 connection exception

ink yet. * reduced the parallelism for my S3 continuous files reader. However, the problem still randomly occurred (random by job executions. When it occurred, the only solution is to cancel the job and restart from the last successful checkpoint). Thanks and regards, Averell [1] Hadoop-AWS

Re: S3 parquet sink - failed with S3 connection exception

that broadcast stream (as mentioned in the document, it doesn't use RocksDB). But not quite sure. Thanks and regards, Averell logs.gz <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/logs.gz> -- Sent from: http://apache-flink-user-mailing-list-archive.2336

S3 parquet sink - failed with S3 connection exception

checkpoint, but it could not make any further checkpoint - all subsequent checkpoints failed with the same reason. Searching on Internet I could only find one explanation: S3Object has not been closed properly. Could someone please help? Thanks and regards, Averell /The program finished

Re: Broadcast state before events stream consumption

ate is kept in-memory at runtime and memory provisioning should be done accordingly. This holds for all operator states."/ I am using RocksDB state backend, and is confused by that statement and yours. Could you please help clarify? Thanks and regards, Averell -- Sent from: http://apache-fl

Re: Stream enrichment with static data, side inputs for DataStream

for every record from the main stream, you read the enrichment data from the saved ValueState to enrich that mainstream record. If no, then I am having the same issue :D Looking at Broadcast State, but there is still something that doesn't look right for me. Regards, Averell -- Sent from: htt

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

solution is to use the version field of each ER request - increase it for every time I retried putting the request into the queue. This works for me until now, but it doesn't look right. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

doesn't have that new class. Maybe Gordon meant 1.7.2 rc2? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Gordon, Sorry for a noob question: How can I get the RC 1.7.2 build / code to build? I could not find any branch like that in Github. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: No resource available error while testing HA

and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Thank you Gordon. That's my exact problem. Will try the fix in 1.7.2 now. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

9a59ff9da6dafa1 expired before completing. 2019-02-09 04:39:30.961 [Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 26 @ 1549687170776 for job 1a1438ca23387c4ef9a59ff9da6dafa1./ Thanks and best regards, Averell [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasti

Re: How to use ExceptionUtils.findThrowable() in Scala

P/S: This is the full stack trace 2019-02-07 01:53:12.790 [I/O dispatcher 16] ERROR o.a.f.s.connectors.elasticsearch.ElasticsearchSinkBase - Failed Elasticsearch item request: [...][[...][1]] ElasticsearchException[Elasticsearch exception [type=version_conflict_engine_exception,

Re: No resource available error while testing HA

Hi Gary, I am trying to reproduce that problem. BTW, is that possible to change log level (I'm using logback) for a running job? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How to use ExceptionUtils.findThrowable() in Scala

flict_engine_exception, reason=[_doc][...]: version conflict, document already exists (current version [1])] / Thanks and best regards, Averell [1] handling-failing-elasticsearch-requests <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html#handli

Re: No resource available error while testing HA

logs collected via "yarn logs --applicationId here. But it seems I still missed something. I am using Flink 1.7.1, with yarn-site configuration yarn.resourcemanager.am.max-attempts=5. Flink configurations are all of the default values. Thanks and best regards, Averell flink.log <http://apa

Re: No resource available error while testing HA

t;Job [...] switched from state" followed by a > stacktrace. If you cannot find the exception, the problem might be rooted > in > your log4j or logback configuration. Thanks. I got the point. I am using logback. Tried to configure rolling logs, but not yet success yet. Will ne

Re: Checkpoints failed with NullPointerException

I tried to create a savepoint on HDFS, and got the same exception: The program finished with the following exception: org.apache.flink.util.FlinkException: Triggering a savepoint for the job 028e392d02bd229ed08f50a2da5227e2 failed.

Checkpoints failed with NullPointerException

pshotState(AbstractStreamOperator.java:407) ... 13 more Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: No resource available error while testing HA

age (not in jobmanager.log or in taskmanager.log in EMR's hadoop-yarn logs folder). <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/Screen_Shot_2019-01-25_at_22.png> Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: No resource available error while testing HA

to zookeeper and the problem was solved. Then I have another question: when JM cannot start/connect to the JM on .88, why didn't it try on .82 where resource are still available? Thanks and regards, Averell Here is the JM log (from /mnt/var/log/hadoop-yarn/.../jobmanager.log on .82) (it seems

No resource available error while testing HA

ed, or a new JobManager will try to connect to the running TMs to resume the job? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

s. However, as I mentioned from the 1st email, I got errors when restoring savepoint created by the same version of my application. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

that I notice is the error doesn't stay on one single operator but changes from time to time (even within the same build). For example, the previous exception I quoted was with a Window operator, while the one below is with CoStreamFlatMap. Thanks and best regards, Averell Caused

Re: When does Trigger.clear() get called?

Thank you Fabian. All my doubts are cleared now. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

typeInfo, reader);/ Does this create two different operators? If yes, then it seems impossible to assign a UID to the 1st operator. And might it be the cause for my problem? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: When does Trigger.clear() get called?

clear those streams? Thank you very much for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: When does Trigger.clear() get called?

. And one related question: for keyed streams, if I know that some keys would never have new events anymore, should/could I remove those streams corresponding to those keys so that I can save some memory allocated to the metadata? Thanks and best regards, Averell -- Sent from: http://apache-flink

Re: Identifying missing events in keyed streams

Thank you Fabian. Tried (2), and it's working well. I found one more benefit of (2) over (3) is that it allow me to easily raise multiple levels of alarms for each keyed stream (i.e: minor: missed 2 cycles, major: missed 5 cycles,...) Thanks for your help. Regards, Averell -- Sent from: http

Re: Identifying missing events in keyed streams

to the minimum level. Could you please explain why (2) is better? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

t=2018-09-20/part-7-5: org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool/ 4. Where is the temporary folder that you store the parquet file before uploading to S3? Thanks a lot for your help. Best rega

Re: Error restoring from savepoint while there's no modification to the job

Hi Kostas, No, the same code was used. I (1) started the job, (2) created a savepoint, (3) cancelled the job, (4) restored the job with the same command as in (1) with the addition "-s ". Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

ing as keys are of types either String or Long. For this, I don't have to define equals and hashcode method, do I? Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Error restoring from savepoint while there's no modification to the job

multiple times), and had not been moved. Is there any kind of improper user code can cause such error? Thanks for your support. Best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Error restoring from savepoint while there's no modification to the job

please help give a look? Thanks and best regards, Averell taskmanager.gz <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/taskmanager.gz> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 606ad5239f5e23cedb85d3e75b

Re: Streaming to Parquet Files in HDFS

ot; >From Flink GUI, all checkpoints were shown as completed successfully. How could I debug further? Thanks a lot for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Streaming to Parquet Files in HDFS

closing in-progress part file for bucket id=dt=2018-09-22 on checkpoint. 2018-10-06 14:40:18.254 [Sink: Unnamed (1/1)] DEBUG o.a.flink.streaming.api.functions.sink.filesystem.Buckets - Subtask 0 checkpointing: BucketState for bucketId=dt=2018-09-22 and bucketPath=s3a://assn-averell/Test/output/dt

Re: Utilising EMR's master node

Hi Gary, Thanks for the information. I didn't know that -yn is obsolete :( I am using Flink 1.6. Not sure whether that's a bug when I tried to set -yn explicitly, but I started only 1 cluster. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4

"unable to establish the security context" with shaded Hadoop S3

stacktrace below. The shading of hadoop jars started from this ticket FLINK-10366 <https://issues.apache.org/jira/browse/FLINK-10366> . Googling the error didn't help. Could someone please help me? Thanks and best regards, Averell /Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no H

Re: Streaming to Parquet Files in HDFS

to HADOOP configurations? Thanks and best regards, Averell java.lang.Exception: unable to establish the security context at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:73) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1118) Caused

Re: Streaming to Parquet Files in HDFS

What a great news. Thanks for that, Kostas. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

  1   2   >