Re: Rolling sink parquet/Avro output

Thank for the mail Bruno !! On Wed, Jan 18, 2017 at 1:10 AM, Bruno Aranda wrote: > Sorry, something went wrong with the code for the Writer. Here it is again: > > import org.apache.avro.Schema > import org.apache.flink.streaming.connectors.fs.Writer > import

Kryo Deserializer

Hello, Having an issue with nested protobuf deserialization, event tried with register the class with Kryo like beloe but seems like no help , one of the options left for me is to write a custom serializer or convert the byte array to a Dictionary object . *val clazz =

Rolling sink parquet/Avro output

Hi There , Does any have Rolling sink parquet/Avro writer reference ??. I'm seeing some issue given stream is handle at rolling sink and I don't see much option override or even open at subclass . I could resolve the same with a custom rolling sink writer, just wondering if any one has done

Custom writer with Rollingsink

Hello , I have to create a custom Parquet writer with rolling sink , I'm seeing error like this , I'm expecting every partition should write in a new file ?? Any tips ? --- 18:12:12.551 [flink-akka.actor.default-dispatcher-5] DEBUG akka.event.EventStream - shutting down:

Re: Custom writer with Rollingsink

aybe you need to check the code to generate filenames. > > Best, > Fabian > > 2017-01-11 3:13 GMT+01:00 Biswajit Das <biswajit...@gmail.com>: > >> Hello , >> >> I have to create a custom Parquet writer with rolling sink , I'm seeing >>

Fwd: Flink -mesos-app master hang

message -- From: Eron Wright <ewri...@live.com> Date: Wed, Aug 2, 2017 at 10:51 AM -- *From:* Biswajit Das <biswajit...@gmail.com> *Sent:* Wednesday, August 2, 2017 10:19:45 AM *To:* Eron Wright *Subject:* Re: Flink -mesos-app master hang Hi Eron ,

Automate Job submission with container

Hi There, A few weeks back I have posted here regarding flink docker container running on mesos. I'm able to run the same successfully; I'm heading towards full CI/CD deployment with the marathon, most of our deployment is automated via the marathon, and sometimes even I trigger metrics based

Re: Flink -mesos-app master hang

problem. > > Cheers, > Till > > On Wed, Aug 2, 2017 at 8:48 PM, Biswajit Das <biswajit...@gmail.com> > wrote: > >> Hi There, >> >> I have posted this here in the group a few days back and after that I >> have been exchanging email with Eron, thanks to Eron

Flink -mesos-app master hang

Hi All, I'm trying to run a flink docker from the marathon with mesos app master; I could see it goes on a continuous loop and failed to launch the task manger. If I go to mesos master UI I could see job manager web UI with task manager zero . I have pretty much checked every possible log

custom writer fail to recover

Hi There , I'm using custom writer with hourly Rolling Bucket sink . I'm seeing two issue first one if write the same file on s3 all the files gets committed , however when I write the same on HDFS I see its remains on .pending state , could be related to second problem below Second issue : My

Re: custom writer fail to recover

ppened in the ParquetWriter, > but then we are looking at the wrong stack trace. > > As for the pending files, different DFS implementations could have > different points where flush() is called. I think your implementation also > properly forward to writer.flush? > > Best, > S

Maven release

Hi Team , Is there any instruction any where like how to publish release , I have been trying to publish release to my own private nexus repository but some how it seems always trying to upload *https://repository.apache.org/service/local/staging/deploy/

state.checkpoints.dir

Hello , Is there any hack to supply *state.checkpoints.*dir as argument or JVM parameter when running locally . I can change the source *CheckpointCoordinator* and make it work , trying to find if there is any shortcuts ?? Thank you ~ Biswajit

Re: state.checkpoints.dir

. > > On Mon, Jan 22, 2018 at 12:53 PM Biswajit Das <biswajit...@gmail.com> > wrote: > >> Hello , >> >> Is there any hack to supply *state.checkpoints.*dir as argument or JVM >> parameter when running locally . I can change the source >&g

Re: Streaming to Parquet Files in HDFS

Nice to see this finally! On Mon, Oct 1, 2018 at 1:53 AM Fabian Hueske wrote: > Hi Bill, > > Flink 1.6.0 supports writing Avro records as Parquet files to HDFS via the > previously mentioned StreamingFileSink [1], [2]. > > Best, Fabian > > [1] https://issues.apache.org/jira/browse/FLINK-9753 >