uild working.
>
> Are you suggesting using docker for production instead? I found some
> license-related challenges with docker and had to see what would be the
> cost implications.
>
> ~JAK
>
> On Mon, Feb 20, 2023 at 7:37 AM Cliff Resnick wrote:
>
>> The answer to
The answer to all three of your questions is yes. We build docker
containers from the Apache source and use those in production.
On Thu, Feb 16, 2023, 4:16 AM Arun J wrote:
> Team,
>
> Looking to migrate from CDH 6.3 to the OpenSource stack. Managed to make
> Impala 4.2 work with the help of
[
https://issues.apache.org/jira/browse/FLINK-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679874#comment-17679874
]
Cliff Resnick edited comment on FLINK-25672 at 1/23/23 4:13 PM:
I
[
https://issues.apache.org/jira/browse/FLINK-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679874#comment-17679874
]
Cliff Resnick commented on FLINK-25672:
---
I imagine it may require a breaking change from what I
[
https://issues.apache.org/jira/browse/FLINK-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679868#comment-17679868
]
Cliff Resnick commented on FLINK-25672:
---
This issue has turned into a real problem for us with our
We think Embedded Statefun is a nicer fit than Datastream for some problem
domains, but one thing we miss is support for custom metrics/counters. Is
there a way to access the Flink support? It looks like if we want custom
metrics we'll need to roll our own.
.
On Sat, Apr 25, 2020, 9:10 PM Cliff Resnick wrote:
> That's pretty much it. Every now and then we get notice an instance is
> scheduled for retirement, or maybe just goes flakey on its own, and we swap
> it out. 3x replication has been plenty for us, and though we regularly back
> up
nted stateless Impala Spot Fleet clusters for HLL and other
>> compute oriented queries. "
>>
>> I cannot really find anything on the web that would compare Impala/Kudu
>> to Snowflake and Redshift. Everything I see is about Snowflake, Redshift
>> and BigQ
I'm running a massive file sifting by timestamp DataSteam job from s3.
The basic job is:
FileMonitor -> ContinuousFileReader -> MultipleFileOutputSink
The MultipleFileOutputSink sifts data based on timestamp to date-hour
directories
It's a lot of data, so I'm using high parallelism, but I want
We have an event-time pipeline that uses a ProcessFunction to accept events
with an allowed lateness of a number of days. We a
BoundedOutOfOrdernessTimestampExtractor and our event stream has a long
tail that occasionally exceeds our allowed lateness, in which case we drop
the events.
The logic
the scan
> perf (including if the runtime filters pushed into the scans were filtering
> most data before joins).
>
> On Mon, Mar 16, 2020 at 2:58 PM Boris Tyukin
> wrote:
>
>> appreciate your thoughts, Cliff
>>
>> On Mon, Mar 16, 2020 at 11:18 AM Cliff Resni
ke Redshift and Snowflake. It is getting harder to explain
>>>>> why :)
>>>>>
>>>>> My biggest gripe with Kudu besides known limitations is the management
>>>>> of partitions and compaction process. For our largest tables, we just max
>
This is a good conversation but I don't think the comparison with Snowflake
is a fair one, at least from an older version of Snowflake (In my last job,
about 5 years ago, I pretty much single-handedly scale tested Snowflake in
exchange for a sweetheart pricing deal) . Though Snowflake is closed
I know from experience that Flink's shaded S3A FileSystem does not
reference core-site.xml, though I don't remember offhand what file (s) it
does reference. However since it's shaded, maybe this could be fixed by
building a Flink FS referencing 3.3.0? Last I checked I think it referenced
3.1.0.
e *remote *metastore host URIs?
> The logs tell that your impalad doesn't find a hive-site.xml, and the
> weird errors could be the result of misconfiguration regarding catalog
> and statestore.
>
> HTH!
>
> On Tue, Jan 14, 2020 at 5:41 PM Cliff Resnick wrote:
> >
&g
l hive metastore db on the impalad instance, not just
> the catalogd instance.' - this sounds weird and is unexpected, and
> multiple metastores can easily lead to issues further down the line.
> What was the end of the log when the impalad failed to start?
>
> On Tue, Jan 14,
I just built Impala 3.3 from source with Kudu 1.11. Unlike previous
versions, in 3.3 impalad would not start unless installed a local hive
metastore db on the impalad instance, not just the catalogd instance. I'm
now getting a strange "IllegalArgumentException:null" error when creating
kudu tables
[
https://issues.apache.org/jira/browse/IMPALA-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999373#comment-16999373
]
Cliff Resnick commented on IMPALA-5323:
---
We store HLL intermediates in Kudu, and UTF-8 encoding
ely, there doesn't appear to have been
> any progress on IMPALA-5323, which would be the clearest path forward.
> Maybe you could update that ticket with your use case and hopefully
> get the attention of some Impala developers?
>
> On Mon, Dec 16, 2019 at 10:16 AM Cliff Resnick wrote:
>
We use a large Impala/Kudu cluster for our analytic reporting. HDFS is not
on the critical path for this, and after experimenting with a cluster
configuration without it, we simply added a co-located HDFS cluster. It
turns out we use HDFS for our dimension staging, and I'm guessing Impala
uses it
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856993#comment-16856993
]
Cliff Resnick edited comment on FLINK-11947 at 6/5/19 7:49 PM:
---
[~klion26
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856993#comment-16856993
]
Cliff Resnick commented on FLINK-11947:
---
[~klion26] confirmed working! Sorry it took me so lone
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854459#comment-16854459
]
Cliff Resnick commented on FLINK-11947:
---
I was not able to get to it on Friday, but will give
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851810#comment-16851810
]
Cliff Resnick commented on FLINK-11947:
---
[~klion26] great! I'll try it today.
> Support MapSt
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850743#comment-16850743
]
Cliff Resnick commented on FLINK-11947:
---
Thanks [~klion26]. I look forward to testing your fix
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847669#comment-16847669
]
Cliff Resnick commented on FLINK-11947:
---
Does the deescalation from Blocker mean
[
https://issues.apache.org/jira/browse/FLINK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831932#comment-16831932
]
Cliff Resnick edited comment on FLINK-12334 at 5/2/19 8:28 PM:
---
Hi
[
https://issues.apache.org/jira/browse/FLINK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831932#comment-16831932
]
Cliff Resnick edited comment on FLINK-12334 at 5/2/19 8:27 PM:
---
Hi
[
https://issues.apache.org/jira/browse/FLINK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831932#comment-16831932
]
Cliff Resnick edited comment on FLINK-12334 at 5/2/19 8:27 PM:
---
Hi
[
https://issues.apache.org/jira/browse/FLINK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831932#comment-16831932
]
Cliff Resnick edited comment on FLINK-12334 at 5/2/19 8:26 PM:
---
Hi
[
https://issues.apache.org/jira/browse/FLINK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831932#comment-16831932
]
Cliff Resnick commented on FLINK-12334:
---
Hi [~fan_li_ya]
We have a custom StreamOperator
Cliff Resnick created FLINK-12334:
-
Summary: change to MockStreamTask breaks
OneInputStreamOperatorTestHarness
Key: FLINK-12334
URL: https://issues.apache.org/jira/browse/FLINK-12334
Project: Flink
Cliff Resnick created FLINK-12334:
-
Summary: change to MockStreamTask breaks
OneInputStreamOperatorTestHarness
Key: FLINK-12334
URL: https://issues.apache.org/jira/browse/FLINK-12334
Project: Flink
till bump into the same issue.
>
> Best,
> Gordon
>
> On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick wrote:
>
>> Hi Gordon,
>>
>> I noticed there has been no movement on this issue and I'm wondering if I
>> can find some way to work around this.
>> M
s definitely possible.
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-11947
>
> On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick wrote:
>
>> After trying out state migration in 1.8 rc2 I ran into this hard stop
>> below. The comment do
[
https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795072#comment-16795072
]
Cliff Resnick commented on FLINK-11947:
---
Thanks for the explanation, that makes sense. However
After trying out state migration in 1.8 rc2 I ran into this hard stop
below. The comment does not give an indication why rocksdb map state cannot
be migrated, and I'm wondering what the status is, since we do need this
functionality and would like to know if this is a long-term blocker or not.
I need to process some Parquet data from S3 as a unioned input in my
DataStream pipeline. From what I know, this requires using the
hadoop AvroParquetInputFormat. The problem I'm running into is that also
requires using un-shaded hadoop classes that conflict with the Flink shaded
hadoop3
scala-2.11 profile)
>
> On 02.01.2019 17:48, Cliff Resnick wrote:
> > The build fails at flink-connector-kafka-0.9 because _2.12 libraries
> > apparently do not exist for kafka < 0.10. Any help appreciated!
> >
> >
> > -Cliff
>
>
>
The build fails at flink-connector-kafka-0.9 because _2.12 libraries
apparently do not exist for kafka < 0.10. Any help appreciated!
-Cliff
We're doing some testing storing Hyperloglog synopsis in Kudu. It works
well in spark, but the hope is to also query through Impala with a UDF.
Spark would remain as the writer, with Impala read-only. To work with
Impala I'm wondering if it's best to define the HLL data as Kudu string
type with
We're doing some testing storing Hyperloglog synopsis in Kudu. It works
well in spark, but the hope is to also query through Impala with a UDF.
Spark would remain as the writer, with Impala read-only. To work with
Impala I'm wondering if it's best to define the HLL data as Kudu string
type with
icationId ` you should see the problem why the TMs
> don't start up.
>
> Cheers,
> Till
>
> On Fri, Nov 9, 2018 at 8:32 PM Cliff Resnick wrote:
>
>> Hi Till,
>>
>> Here are Job Manager logs, same job in both 1.6.0 and 1.6.2 at DEBUG
>> level. I saw
+1!
On Fri, Nov 9, 2018 at 1:34 PM Gary Yao wrote:
> Hi,
>
> We only propagate the exception message but not the complete stacktrace
> [1].
> Can you create a ticket for that?
>
> Best,
> Gary
>
> [1]
>
I'm running a YARN cluster of 8 * 4 core instances = 32 cores, with a
configuration of 3 slots per TM. The cluster is dedicated to a single job
that runs at full capacity in "FLIP6" mode. So in this cluster, the
parallelism is 21 (7 TMs * 3, one container dedicated for Job Manager).
When I run
[
https://issues.apache.org/jira/browse/FLINK-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675184#comment-16675184
]
Cliff Resnick commented on FLINK-10671:
---
[~gjy] yes, I think when I created the issue I couldn't
Cliff Resnick created FLINK-10671:
-
Summary: rest monitoring api Savepoint status call fails if
akka.ask.timeout < checkpoint duration
Key: FLINK-10671
URL: https://issues.apache.org/jira/browse/FLINK-10
Cliff Resnick created FLINK-10671:
-
Summary: rest monitoring api Savepoint status call fails if
akka.ask.timeout < checkpoint duration
Key: FLINK-10671
URL: https://issues.apache.org/jira/browse/FLINK-10
, Aug 21, 2018 at 8:02 AM Cliff Resnick wrote:
> Hi Aljoscha,
>
> We need flink-shaded-hadoop2-uber.jar because there is no hadoop distro on
> the instance the Flink session/jobs is managed from and the process that
> launches Flink is not a java process, but execs a process that ca
5, vino yang wrote:
>
> Hi Cliff,
>
> If so, you can explicitly exclude Avro's dependencies from related
> dependencies (using ) and then directly introduce dependencies on
> the Avro version you need.
>
> Thanks, vino.
>
> Cliff Resnick 于2018年8月21日周二 上午5:13写道
the Avro
dependency to the user jar. However, since I'm using YARN, I'm required to
have flink-shaded-hadoop2-uber.jar loaded from lib -- and that has
avro bundled un-shaded. So I'm back to the start problem...
Any advice is welcome!
-Cliff
On Mon, Aug 20, 2018 at 1:42 PM Cliff Resnick wrote:
>
eratorStateBackend.deserializeStateValues(DefaultOperatorStateBackend.java:584)
>> at
>> org.apache.flink.runtime.state.DefaultOperatorStateBackend.restore(DefaultOperatorStateBackend.java:399)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.createOperatorStateBackend(
, 2018 at 8:43 AM Cliff Resnick wrote:
> Hi Vino,
>
> Thanks for the explanation, but the job only ever uses the Avro (1.8.2)
> pulled in by flink-formats/avro, so it's not a class version conflict
> there.
>
> I'm using default child-first loading. It might be a further tr
documentation.[2]
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/config.html
> [2]:
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>
> Thanks, vino.
>
> Cliff Resnick 于2018年8月20日周一 上午10:40写道：
>
>&
Our Flink/YARN pipeline has been reading Avro from Kafka for a while now.
We just introduced a source of Avro OCF (Object Container Files) read from
S3. The Kafka Avro continued to decode without incident, but the OCF files
failed 100% with anomalous parse errors in the decoding phase after the
stMetadataJoin.broadcast(metadata) for the* late-join*.
> Is this intended, or just a copy error?
>
>
> On 03.07.2018 04:16, Cliff Resnick wrote:
>
> Our topology has a metadata source that we push via Broadcast. Because
> this metadata source is critical, but sometimes late, we a
Our topology has a metadata source that we push via Broadcast. Because this
metadata source is critical, but sometimes late, we added a buffering
mechanism via a SideOutput. We call the initial look-up from Broadcast
"join" and the secondary, state-backed buffered lookup, "late-join"
Today I
Cliff Resnick created FLINK-9339:
Summary: Accumulators are not UI accessible running in FLIP-6 mode
Key: FLINK-9339
URL: https://issues.apache.org/jira/browse/FLINK-9339
Project: Flink
Cliff Resnick created FLINK-9339:
Summary: Accumulators are not UI accessible running in FLIP-6 mode
Key: FLINK-9339
URL: https://issues.apache.org/jira/browse/FLINK-9339
Project: Flink
dcast tables better? Perhaps Impala can cache
>>> small tables locally when doing joins.
>>>
>>> - Dan
>>>
>>> On Fri, Mar 16, 2018 at 10:55 AM, Clifford Resnick <
>>> cresn...@mediamath.com> wrote:
>>>
>>>> The problem is,
[
https://issues.apache.org/jira/browse/FLINK-8616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cliff Resnick updated FLINK-8616:
-
Summary: Missing null check in
OperatorChain.CopyingChainingOutput#pushToOperator masks
[
https://issues.apache.org/jira/browse/FLINK-8616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cliff Resnick updated FLINK-8616:
-
Summary: Missing null check in
OperatorChain.copyingChainOutput#pushToOperator masks
Cliff Resnick created FLINK-8616:
Summary: Missing null check in OperatorChain#pushToOperator masks
ClassCastException
Key: FLINK-8616
URL: https://issues.apache.org/jira/browse/FLINK-8616
Project
Cliff Resnick created FLINK-8616:
Summary: Missing null check in OperatorChain#pushToOperator masks
ClassCastException
Key: FLINK-8616
URL: https://issues.apache.org/jira/browse/FLINK-8616
Project
I've seen a similar issue while running successive Flink SQL batches on
1.4. In my case, the Job Manager would fail with the log output about
unreachability (with an additional statement about something going
"horribly wrong"). Under workload pressure, I reverted to 1.3.2 where
everything works
I've been trying out the Table API for some ETL using a two-stage job of
CsvTableSink (DataSet) -> CsvInputFormat (Stream). I ran into an issue
where the first stage produces output with trailing null values (valid),
which causes a parse error in the second stage.
Looking at
atch, or a scan batch with injected nullable-int columns, we will
> return NONE to the downstream operators directly, which will avoid the
> unintended consequence.
>
> I will probably wrap up that work in a few days, and submit a PR for
> review.
>
>
>
> On Mon, Jul 24, 2017 a
batch. Nullable int will be injected in downstream operator.
>
> 1.
> https://github.com/apache/drill/blob/master/contrib/
> storage-kudu/src/main/java/org/apache/drill/exec/store/
> kudu/KuduRecordReader.java#L149-L163
>
>
> On Mon, Jul 24, 2017 at 1:35 PM, Cliff Re
at 2:46 PM, Cliff Resnick <cre...@gmail.com> wrote:
> Jinfeng,
>
> Thanks, that confirms my thoughts as well. If I query using full range
> bounds and all hash keys, then Kudu prunes to the exact tablets and there
> is no error. I'll watch that jira expectantly becau
rowse/DRILL-5546
>
>
>
> On Mon, Jul 24, 2017 at 10:56 AM, Cliff Resnick <cre...@gmail.com> wrote:
>
> > I spent some time over the weekend altering Drill's storage-kudu to use
> > Kudu's predicate pushdown api. Everything worked great as long as I
> > perform
I spent some time over the weekend altering Drill's storage-kudu to use
Kudu's predicate pushdown api. Everything worked great as long as I
performed flat filtered selects (eg. SELECT .. FROM .. WHERE ..") but
whenever I tested aggregate queries, they would succeed sometimes, then
fail other times
Hi Gautam,
Though I did get the filter pushdown to kudu working I unfortunately
encountered sporadic Drill errors when performing aggregate queries. The
most common error was:
java.lang.IllegalStateException: Failure while reading vector. Expected
vector class of
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064079#comment-16064079
]
Cliff Resnick commented on FLINK-6964:
--
[~srichter] Looks good from this end, all tests passed
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063768#comment-16063768
]
Cliff Resnick commented on FLINK-6964:
--
[~srichter] So far, looks good! I need to leave early today
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061270#comment-16061270
]
Cliff Resnick commented on FLINK-6964:
--
looks likes it's still trying to register a Placeholder
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061257#comment-16061257
]
Cliff Resnick commented on FLINK-6964:
--
I ran with your newer precondition. It actually succeeded
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061216#comment-16061216
]
Cliff Resnick commented on FLINK-6964:
--
ok, will try that. meanwhile here is a run (and hang) from
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061178#comment-16061178
]
Cliff Resnick commented on FLINK-6964:
--
ok I'll wait on your push
> Fix recovery for incremen
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061163#comment-16061163
]
Cliff Resnick commented on FLINK-6964:
--
Ha! I just started running. ok, will merge and rebuild
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061092#comment-16061092
]
Cliff Resnick commented on FLINK-6964:
--
I'll merge and rerun. This is what I have for log scope
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061088#comment-16061088
]
Cliff Resnick commented on FLINK-6964:
--
[~srichter] By hanging I mean that the checkpoint, though
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060027#comment-16060027
]
Cliff Resnick commented on FLINK-6964:
--
[~srichter] I tried your fix. After resuming from
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058158#comment-16058158
]
Cliff Resnick commented on FLINK-6633:
--
Thanks [~srichter], I'll give this a try tomorrow morning EST
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056537#comment-16056537
]
Cliff Resnick commented on FLINK-6633:
--
Stefan, if you need me to unpack things further please feel
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056527#comment-16056527
]
Cliff Resnick commented on FLINK-6633:
--
Ok, new gist is here:
https://gist.github.com/cresny
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056342#comment-16056342
]
Cliff Resnick commented on FLINK-6633:
--
full log here: https://gist.github.com/cresny
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056334#comment-16056334
]
Cliff Resnick commented on FLINK-6633:
--
{noformat}
2017-06-20 18:44:39.376 [ip-10-150-96-228] INFO
[
https://issues.apache.org/jira/browse/FLINK-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056332#comment-16056332
]
Cliff Resnick commented on FLINK-6633:
--
The issue that [~gyfora] mentioned still exists in current
When a job cancel-with-savepoint finishes a successful Savepoint, the
preceding last successful Checkpoint is removed. Is this the intended
behavior? I thought that checkpoints and savepoints were separate entities
and, as such, savepoints should not infringe on checkpoints. This is
actually an
Cliff Resnick created FLINK-5646:
Summary: REST api documentation missing details on jar upload
Key: FLINK-5646
URL: https://issues.apache.org/jira/browse/FLINK-5646
Project: Flink
Issue
; Can you also please open a jira for fixing the documentation?
>
> - Sachin
>
>
> On Jan 25, 2017 06:55, "Cliff Resnick" <cre...@gmail.com> wrote:
>
>> The 1.2 release documentation (https://ci.apache.org/project
>> s/flink/flink-docs-release-1.2/monitoring
Cliff Resnick created FLINK-5646:
Summary: REST api documentation missing details on jar upload
Key: FLINK-5646
URL: https://issues.apache.org/jira/browse/FLINK-5646
Project: Flink
Issue
The 1.2 release documentation (https://ci.apache.org/
projects/flink/flink-docs-release-1.2/monitoring/rest_api.html) states "It
is possible to upload, run, and list Flink programs via the REST APIs and
web frontend". However there is no documentation about uploading a jar via
REST api. Does this
for a bit):
>
> - There is some much enhanced Checkpoint Monitoring almost ready to be
> merged. That should help in fining out where the barriers get delayed.
>
> - Finally, we are experimenting with some other checkpoint alignment
> variants (alternatives to the BarrierBuffer). We can p
[
https://issues.apache.org/jira/browse/FLINK-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15773084#comment-15773084
]
Cliff Resnick commented on FLINK-4228:
--
My last pull request is good to go so I guess it's up to you
We are running a DataStream pipeline using Exactly Once/Event Time
semantics on 1.2-SNAPSHOT. The pipeline sources from S3 using the
ContinuousFileReaderOperator. We use a custom version of the
ContinuousFileMonitoringFunction since our source directory changes over
time. The pipeline transforms
[
https://issues.apache.org/jira/browse/FLINK-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772906#comment-15772906
]
Cliff Resnick commented on FLINK-4228:
--
The issue now is exclusive to running on YARN with s3a
or how long did you look at iotop. It could be that the IO access happens
> in bursts, depending on how data is cached.
>
> I'll also add Stefan Richter to the conversation, he has maybe some more
> ideas what we can do here.
>
>
> On Mon, Dec 5, 2016 at 6:19 PM, Cliff Res
ou could look
> into tuning the RocksDB settings so that it uses more memory for caching.
>
> Regards,
> Robert
>
>
> On Fri, Dec 2, 2016 at 11:34 PM, Cliff Resnick <cre...@gmail.com> wrote:
>
>> In tests comparing RocksDb to fs state backend we observe mu
In tests comparing RocksDb to fs state backend we observe much lower
throughput, around 10x slower. While the lowered throughput is expected,
what's perplexing is that machine load is also very low with RocksDb,
typically falling to < 25% CPU and negligible IO wait (around 0.1%). Our
test
1 - 100 of 121 matches
Mail list logo
