Re: CSV StreamingFileSink

2020-02-18 Thread Austin Cawley-Edwards
-02-19/ large-file-3.txt Or is there another way to accomplish this? Thanks! Austin On Tue, Feb 18, 2020 at 5:33 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey all, > > Has anyone had success using the StreamingFileSink[1] to write CSV files? > And

CSV StreamingFileSink

2020-02-18 Thread Austin Cawley-Edwards
Hey all, Has anyone had success using the StreamingFileSink[1] to write CSV files? And if so, what about compressed (Gzipped, ideally) files/ which libraries did you use? Best, Austin [1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html

Re: Custom label for Prometheus Exporter

2020-01-22 Thread Austin Cawley-Edwards
Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Anaray, > > Have you checked out the “scope” configuration?[1] > > Best, > Austin > > > [1]: > > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#scope > > On Wed, J

Re: Custom label for Prometheus Exporter

2020-01-22 Thread Austin Cawley-Edwards
Hey Anaray, Have you checked out the “scope” configuration?[1] Best, Austin [1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#scope On Wed, Jan 22, 2020 at 4:09 PM anaray wrote: > Hi flink team, > > Is there a way to add a custom label to flink metrics when

Flink x GATE Integration

2019-12-17 Thread Austin Cawley-Edwards
to GATE, but trying to explore the options out there for good text processing within Flink. Best, Austin Cawley-Edwards [1]: https://gate.ac.uk [2]: https://groups.io/g/gate-users/topic/usage_with_data_processing/68761558?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,68761558

Flink 1.8.0 S3 Checkpoints fail with java.security.ProviderException: Could not initialize NSS

2019-10-10 Thread Austin Cawley-Edwards
Hi there, I'm getting the following error message on a Flink 1.8 cluster deployed on Kubernetes. I've already confirmed that the pod has access to S3 and write permissions to the bucket, but I can't understand what the SSL issue is and if it is related to S3 or not. I have tried both with the

Re: Recommended approach to debug this

2019-09-22 Thread Austin Cawley-Edwards
Have you reached out to the FlinkK8sOperator team on Slack? They’re usually pretty active on there. Here’s the link: https://join.slack.com/t/flinkk8soperator/shared_invite/enQtNzIxMjc5NDYxODkxLTEwMThmN2I0M2QwYjM3ZDljYTFhMGRiNDUzM2FjZGYzNTRjYWNmYTE1NzNlNWM2YWM5NzNiNGFhMTkxZjA4OGU Best, Austin

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

2019-02-28 Thread Austin Cawley-Edwards
t; [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink > > > On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Thanks Gary

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

2019-02-27 Thread Austin Cawley-Edwards
://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths > >

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

2019-02-27 Thread Austin Cawley-Edwards
${main.class} Thanks again and let me know if there is more information I can provide. Austin On Tue, Feb 26, 2019 at 10:59 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi, >

okio and okhttp not shaded in the Flink Uber Jar on EMR

2019-02-26 Thread Austin Cawley-Edwards
e$1.class okio/RealBufferedSource.class okio/Segment.class okio/SegmentPool.class okio/SegmentedByteString.class okio/Sink.class okio/Source.class okio/Timeout$1.class okio/Timeout.class okio/Util.class Thank you, Austin Cawley-Edwards

Re: Over-requesting Containers on YARN

2018-12-04 Thread Austin Cawley-Edwards
d here <https://mail-archives.apache.org/mod_mbox/flink-user/201811.mbox/%3cc7e61cb0-8799-403a-861b-88d2f3eb2...@bytedance.com%3E>, and though it lasted longer, the checkpoint still failed. Thanks, Austin On Tue, Dec 4, 2018 at 12:24 PM Austin Cawley-Edwards < austin.caw...@gmail

Re: Over-requesting Containers on YARN

2018-12-04 Thread Austin Cawley-Edwards
, 2018 at 11:24 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi all, > > We have a Flink 1.6 streaming application running on Amazon EMR, with a > YARN session configured with 20GB for the Task Manager, 2GB for the Job > Manager, and 4 slots (number of vCPUs), i

Over-requesting Containers on YARN

2018-12-04 Thread Austin Cawley-Edwards
application on EMR with YARN? Thank you, Austin Cawley-Edwards

Re: Flink CEP Watermark Exception

2018-11-06 Thread Austin Cawley-Edwards
regarding CEP you've linked concern very old Flink > version (1.0.3), CEP library was heavily reworked since then and I would > not look for any similarities in those cases. > > Best, > > Dawid > On 01/11/2018 14:24, Austin Cawley-Edwards wrote: > > Hi Dawid, > &g

Re: Flink CEP Watermark Exception

2018-11-01 Thread Austin Cawley-Edwards
ot;after a checkpoint > fails", what is the reason why checkpoint fails? Would it be possible for > you to prepare some reproducible example for that problem? Finally, I would > also recommend trying out Flink 1.6.x, as we reworked the underlying > structure for CEP - SharedBuff

Re: Flink CEP Watermark Exception

2018-10-30 Thread Austin Cawley-Edwards
Following up, we are using Flink 1.5.0 and Flink-CEP 2.11. Thanks, Austin On Tue, Oct 30, 2018 at 3:58 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi there, > > > We have a streaming application that uses CEP processing but are getting this > error fa

Flink CEP Watermark Exception

2018-10-30 Thread Austin Cawley-Edwards
Hi there, We have a streaming application that uses CEP processing but are getting this error fairly frequently after a checkpoint fails, though not sure if it is related. We have implemented both `hashCode` and `equals()` using `Objects.hash(...properties)` and basic equality, respectively.

Re: Accumulating a batch

2018-10-26 Thread Austin Cawley-Edwards
> [2] > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/udfs.html#aggregation-functions > > On Fri, Oct 26, 2018 at 5:08 AM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hi there, >> >> Is it possible to use an Aggregatio

Accumulating a batch

2018-10-25 Thread Austin Cawley-Edwards
Hi there, Is it possible to use an AggregationFunction to accumulate n values in a buffer until a threshold is met, then stream down all records in the batch? Thank you! Austin Cawley-Edwards

Re: Triggering Savepoints with the Monitoring API

2018-09-11 Thread Austin Cawley-Edwards
these two jobs will be subject > to certain restrictions. For more information, please refer to Official > documentation.[1] > > Thanks, vino. > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/upgrading.html#application-state-compatibility > > Aus

Triggering Savepoints with the Monitoring API

2018-09-10 Thread Austin Cawley-Edwards
Hi there, I would just like a quick sanity-check: it is possible to start a job with a savepoint from another job, and have the new job save to a new checkpoint directory without overwriting the original checkpoints, correct? Thank you so much! Austin

Re: Setting Flink Monitoring API Port on YARN Cluster

2018-09-07 Thread Austin Cawley-Edwards
> > [4] > https://hadoop.apache.org/docs/r2.8.3/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getRpcPort() > > On Fri, Sep 7, 2018 at 12:33 AM, Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hi everyone, >> >> I'm

Setting Flink Monitoring API Port on YARN Cluster

2018-09-06 Thread Austin Cawley-Edwards
Hi everyone, I'm running a YARN session on a cluster with one master and one core and would like to use the Monitoring API programmatically to submit jobs. I have found that the configuration variables are read but ignored when starting the session - it seems to choose a random port each run.

<    1   2