Hey Arran,
It seems that the preferred way, even in the Java API is to use a DDL
statement:
https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/api/bridge/java/StreamTableEnvironment.java#L602-L639
Hope this helps!
Best,
Hey Sandeep,
(Maybe this thread is also relevant:
https://lists.apache.org/thread.html/7d56267d4c2344ccb5a774896682d0a3efb38c1c215ef3500c3569a2%40%3Cuser.flink.apache.org%3E
)
> My question is how do I initialise the pipeline for the first set of
records in the database? i.e. those that are not
Hi,
has the "state.savepoints.dir" configuration key the same value as
"state.checkpoints.dir"?
If not, can you post your configuration keys, and the invocation how you
trigger a savepoint?
Have you checked the logs? Maybe there's an error message?
On Thu, Mar 25, 2021 at 7:17 PM Robert Cullen
Hey Sandeep,
here's a project I've recently worked on, that deploys Flink on Minikube:
https://github.com/rmetzger/flink-reactive-mode-k8s-demo
The project is pretty big, but I guess you can pick the bits related to the
Flink deployment on minikube.
On Thu, Mar 25, 2021 at 7:48 PM Sandeep
Hey,
are you sure the class is in the lib/ folder of all machines / instances,
and you've restarted Flink after adding the files to lib/ ?
On Mon, Mar 15, 2021 at 3:42 PM Robert Cullen wrote:
> Shuiqiang,
>
> I added the flink-connector-kafka_2-12 jar to the /opt/flink/lib directory
>
> When
Hey Matthias,
are you sure you can connect to 127.0.1.1, since everything between
127.0.0.1 and 127.255.255.255 is bound to the loopback device?:
https://serverfault.com/a/363098
On Mon, Mar 15, 2021 at 11:13 AM Matthias Seiler <
matthias.sei...@campus.tu-berlin.de> wrote:
> Hi Arvid,
>
> I
Since this error is happening in your IDE, I would recommend using the
IntelliJ debugger to follow the filesystem initialization process and see
where it fails to pick up the credentials.
On Fri, Mar 12, 2021 at 11:11 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:
> Same error.
rly stages of just assessing what kind of deployment
> model we'd like to use, it's hard to say what will work best for us. We
> just want to see if reactive mode will be available in the future so that
> we can leverage it when we have more data.
>
> Thanks,
> Sonam
>
&
Hey Sonam,
I'm very happy to hear that you are interested in reactive mode. Your
understanding of the limitations for 1.13 is correct. Note that you can
deploy standalone Flink on Kubernetes [1]. I'm actually currently preparing
a demo for this [2].
We are certainly aware that support for active
Hey Rion,
you don't need special access to Flink's Jira: Any JIra user is assignable
to tickets, but only committers can assign people.
For low hanging fruits, we have a "starter" label to tag those tickets. I
also recommend keeping an eye on Jira tickets about topics you are
experienced with /
Hey,
Could it be this problem:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-S3-Permissions-does-StreamingFileSink-need-td31426.html
?
is this problem reproducible, or just happening every now and then (not
saying that this makes it less worse). Have you tried the
Hey John,
I haven't worked with the flink-cdc-connectors [1] myself. But if you take
the MySQL one as a template, it shouldn't be too difficult to adjust it to
MS SQL server. If you are doing that work, it would be nice if you would
contribute it back to the repo ;)
I don't think that you need
Thanks for reaching out to the Flink ML.
It reports getMetricStoreProgramHelper as a non-serializable field, even
though it looks a lot like a method. The only recommendation I have for you
is carefully reading the full error message + stack trace.
Your approach of using tagging fields as
Flink supports Hadoop's FileSystem abstraction, which has an implementation
for FTP:
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/ftp/FTPFileSystem.html
On Tue, Feb 2, 2021 at 3:43 AM 1095193...@qq.com <1095193...@qq.com> wrote:
> Hi
> I have investigate the relevant document
By default, a checkpoint times out after 10 minutes. This means if not all
operators are able to confirm the checkpoint, it will be cancelled.
If you have an operator that is blocking for more than 10 minutes on a
single record (because this record contains millions of elements that are
written
I'm not sure if this is related, but you are mixing scala 2.11 and 2.12
dependencies (and mentioning scala 2.1.1 dependencies).
On Tue, Feb 2, 2021 at 8:32 AM Eleanore Jin wrote:
> Hi experts,
> I am trying to experiment how to use Hive to store metadata along using
> Flink SQL. I am running
performance degradates greatly. I Moved the jobs to SSD disks and the
> performance has been better.
>
> Best regards!
>
> On Tue, 2 Feb 2021 at 20:22, Robert Metzger wrote:
> >
> > Hey Yordan,
> >
> > have you checked the log files from the processes in that cl
int; Standalone Cluster;
>
>
>
> Robert Metzger 于2021年2月5日周五 下午6:52写道：
>
>> Are you using unaligned checkpoints? (there's a known bug in 1.12.0 which
>> can lead to corrupted data when using UC)
>> Can you tell us a little bit about your environment? (How are you
>
I don't know what your dependency issue is (post it here if you want
help!), but I generally recommend using mvn dependency:tree to debug
version clashes (and then pin or exclude versions)
On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí wrote:
> The root of the previous error seemed to be the
Hey,
the code and exception are not included in your message. Did you try to
send them as images (screenshots)?
I recommend sending code and exceptions as text for better searchability.
On Wed, Feb 3, 2021 at 12:58 PM cristi.cioriia wrote:
> Hey guys,
>
> I'm pretty new to Flink, I hope I could
Answers inline:
On Wed, Feb 3, 2021 at 3:55 PM Marco Villalobos
wrote:
> Hi Gorden,
>
> Thank you very much for the detailed response.
>
> I considered using the state-state processor API, however, our enrichment
> requirements make the state-processor API a bit inconvenient.
> 1. if an element
Are you using unaligned checkpoints? (there's a known bug in 1.12.0 which
can lead to corrupted data when using UC)
Can you tell us a little bit about your environment? (How are you deploying
Flink, which state backend are you using, what kind of job (I guess
DataStream API))
Somehow the process
Another strategy to resolve such issues is by explicitly excluding the
conflicting dependency from one of the transitive dependencies.
Besides that, I don't think there's a nicer solution here.
On Thu, Feb 4, 2021 at 6:26 PM Jan Oelschlegel <
oelschle...@integration-factory.de> wrote:
> I
Hey Yordan,
have you checked the log files from the processes in that cluster?
The JobManager log should give you hints about issues with the coordination
/ scheduling of the job. Could it be something unexpected, like your job
could not start, because there were not enough TaskManagers
Hi all,
during a security maintenance PR [1], Chesnay noticed that the
flink-swift-fs-hadoop module is lacking test coverage [2].
Also, there hasn't been any substantial change since 2018, when it was
introduced.
On the user@ ML, I could not find any proof of significant use of the
module (no one
Hi all,
during a security maintenance PR [1], Chesnay noticed that the
flink-swift-fs-hadoop module is lacking test coverage [2].
Also, there hasn't been any substantial change since 2018, when it was
introduced.
On the user@ ML, I could not find any proof of significant use of the
module (no one
itially submitted from the client app
> logs and when the JobManager logs it as being received – we’re submitting a
> large number of jobs as a part of this application. Is it possible that
> it’s busy processing other jobs?
>
>
>
> *// *ah
>
>
>
> *From:* Robert Metz
Since our CI system is able to build Flink, I believe it's a local issue.
Are you sure that the build is failing when you build Flink from the root
directory (not calling maven from within a maven module?)
On Tue, Jan 19, 2021 at 11:19 AM Smile@LETTers
wrote:
> Hi,
> I got an error when tried
Thanks a lot for your message.
Why is there a difference of 5 minutes between the timestamp of the job
submission from the client to the timestamp on the JobManager where the
submission is received?
Is there any service / custom logic involved in the job submission? (e.g. a
proxy in between, that
Hey David,
this is a good catch! I've filed a JIRA ticket to address this in the docs
more prominently: https://issues.apache.org/jira/browse/FLINK-21073
Thanks a lot for reporting this issue!
On Thu, Jan 21, 2021 at 9:24 AM David Haglund
wrote:
> A colleague of mine found some hint under
CVE-2020-17518: Apache Flink directory traversal attack: remote file
writing through the REST API
Vendor:
The Apache Software Foundation
Versions Affected:
1.5.1 to 1.11.2
Description:
Flink 1.5.1 introduced a REST handler that allows you to write an uploaded
file to an arbitrary location on
CVE-2020-17519: Apache Flink directory traversal attack: reading remote
files through the REST API
Vendor:
The Apache Software Foundation
Versions Affected:
1.11.0, 1.11.1, 1.11.2
Description:
A change introduced in Apache Flink 1.11.0 (and released in 1.11.1 and
1.11.2 as well) allows
illed may be a sign of backpressure? Currently one of our operators takes
>> a tremendous amount of time to align during a checkpoint. Could increasing
>> direct memory help checkpointing by improving I/O performance across the
>> whole plan (assuming I/O is at least part of the bo
I guess you are seeing a different error now, because you are submitting
the job, and stopping it right away.
Can you produce new logs, where you wait until at least one Checkpoint
successfully completed before you stop?
>From the exception it seems that the job has not successfully been
Hey,
maybe your event time time-stamps are wrong, leading to an obscure year
(1705471 instead of 2020). Flink send's Long.MAX_Value as the final
watermark.
On Sat, Dec 12, 2020 at 2:29 PM Appleyuchi wrote:
>
>
> I'm trying flatAggregate, the whole code is bug free and as follows:
>
>
Hey Rex,
If I'm reading the Flink code correctly, then RocksDB will allocate it's
storage across all configured tmp directories. Flink is respecting the
io.tmp.dirs configuration property for that.
it seems that you are using Flink on YARN, where Flink is respecting the
tmp directory configs from
Hi,
the logs from the client are not helpful for debugging this particular
issue.
With kubectl get pods, you can get the TaskManger pod names, with kubectl
logs you can get the logs.
The JobManager log would also be nice to have.
On Mon, Dec 14, 2020 at 3:29 PM Folani wrote:
> Hi Piotrek,
>
>
If a TaskManager fails, the data stored on it will be lost and needs to be
recomputed. So even with the batch mode configured, more tasks might need a
restart.
To mitigate that, the Flink developers need to implement support for
external shuffle services.
On Wed, Dec 16, 2020 at 9:10 AM Robert
With region failover strategy, all connected subtasks will fail.
If you are using the DataSet API with env.getConfig().setExecutionMode(
ExecutionMode.BATCH);, you should get the desired behavior.
On Mon, Dec 14, 2020 at 5:24 PM Stanislav Borissov
wrote:
> Hi,
>
> I'm running a simple,
Hey Azeem,
I haven't tried this myself, but from the code / documentation, this could
work:
Flink ships with ZK 3.4 by default. You need to remove the ZK3.4 jar file
from the lib/ folder and add the ZK3.5 file from opt/ to lib/.
According to this guide, you could try passing the SSL
What you can also do is rely on Flink's backpressure mechanism: If the map
operator that validates the messages detects that the external system is
down, it blocks until the system is up again.
This effectively causes the whole streaming job to pause: the Kafka source
won't read new messages.
On
Hey Rex,
the direct memory is used for IO. There is no concept of direct memory
being "full". The only thing that can happen is that you have something in
place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
process, and you run out of your memory allowance. The direct memory
Hey Avi,
Maybe providing secret/access key + session token doesn't work, and you
need to provide either one of them?
https://docs.aws.amazon.com/credref/latest/refdocs/setting-global-aws_session_token.html
I'll also ping some AWS contributors active in Flink to take a look at this.
Best,
Robert
The Apache Flink community is very happy to announce the release of Apache
Flink 1.12.0, which is the latest major release.
Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.
The release is
Hey Tim,
delayed messages are stored in Flink's state while they are waiting to be
sent again. Thus they are not blocking any checkpoints (and thus the
persisting of Kafka offsets).
If you are restoring from a checkpoint (or savepoint), the pending delayed
messages will be reloaded into Flink's
resources
> for our orchestration?
>
> Also, good point on recovery. I'll spend some time looking into this.
>
> Thanks
>
>
> On Wed, Nov 11, 2020 at 11:53 PM Robert Metzger
> wrote:
>
>> Hey Rex,
>>
>> the second approach (spinning up a standby job and th
Hi,
from my experience serialization contributes a lot to the maximum
achievable throughput. I can strongly recommend checking out this blog
post, which has a lot of details on the topic:
https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
On Tue, Nov 10, 2020 at 9:46
Hey Tim,
what Is your Flink job doing? Is it restarting from time to time?
Is the JobManager crashing, or the TaskManager?
On Tue, Nov 10, 2020 at 6:01 PM Matthias Pohl
wrote:
> Hi Tim,
> I'm not aware of any memory-related issues being related to the deployment
> mode used. Have you checked
Hey Josh,
As far as I understand the code CompletedCheckpoint.discard(), Flink is
removing all the files in StateUtil.bestEffortDiscardAllStateObjects, then
deleting the directory.
Which files are left over in your case?
Do you see any exceptions on the TaskManagers?
Best,
Robert
On Wed, Nov
Hi Jiahui,
using the yarn.container-start-command-template is indeed a good idea.
I was also wondering whether the Flink YARN client that submits the Flink
cluster to YARN has knowledge of the host where the ApplicationMaster gets
deployed to. But that doesn't seem to be the case.
On Wed, Nov
Hey Rex,
the second approach (spinning up a standby job and then doing a handover)
sounds more promising to implement, without rewriting half of the Flink
codebase ;)
What you need is a tool that orchestrates creating a savepoint, starting a
second job from the savepoint and then communicating
Thanks a lot for posting a question to the user@ mailing list.
Note that the language of this list is English. For Chinese language
support, reach out to user...@flink.apache.org.
On Thu, Nov 12, 2020 at 5:53 AM 键 <1941890...@qq.com> wrote:
> batch模式broadcast hash join为什么会有数据丢失
>
Hi Vignesh,
I'm adding Aljoscha to the thread, he might have an idea how to solve this
with the existing Flink APIs (the closest idea I had was the N-ary stream
operator, but I guess that doesn't support backpressuring individual
upstream operators -- side inputs would be needed for that?)
The
Hi Alexandru,
1. You can either create a Flink cluster per job (preferred), or use one
big cluster to run all your jobs. This depends a bit on the resource
manager you are using, and the workloads you are planning to process. If
you are using Kubernetes, it makes sense to deploy each job
Hi Kevin,
thanks a lot for posting this problem.
I'm adding Jark to the thread, he or another committer working on Flink SQL
can maybe provide some insights.
On Tue, Nov 3, 2020 at 4:58 PM Kevin Kwon wrote:
> Looks like the event time that I've specified in the consumer is not being
>
Hi Noah,
sadly there's no generic guide on how to approach Flink logs.
What exactly do you mean by "the job hangs"?
Did you verify via the metrics that it is not making any progress anymore
at all? If so, are all operators affected, or just some?
If your Flink cluster really is stuck, and you
10.0.55.50:10200
>>>> Exception in thread "main" java.lang.IllegalArgumentException: App
>>>> admin client class name not specified for type Apache Flink
>>>> at
>>>> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClien
PM Claude M wrote:
> Thanks for your reply Robert. Please see attached log from the job
> manager, the last line is the only thing I see different from a pod that
> starts up successfully.
>
> On Tue, Nov 3, 2020 at 10:41 AM Robert Metzger
> wrote:
>
>> Hi Claude,
&g
kpointTimeout(6);
> - env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
> 3- Based on above it's possible that the sink takes longer than 60seconds
> sometimes...
> - Looking at adjusting timeouts.
> - Looking at reducing the load of the sink and reduce how long it
> takes in g
Hi Patrick,
The upcoming Flink 1.12 release will update the version to 5.4.2 at least:
https://github.com/apache/flink/pull/12919/files
This is closer to what you need, but still not there :(
What you can try is compile your own version of
flink-avro-confluent-registry, where you pass
Hi,
I agree that our docs are not mentioning this anywhere. I would have
expected it on this page:
https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html.
I filed a ticket to address this:
https://issues.apache.org/jira/browse/FLINK-19955
The only thing Flink officially
How did you configure the Kafka source as at least once? Afaik the source
is always exactly-once (as long as there aren't any restarts).
Are you seeing the duplicates in the context of restarts of the Flink job?
On Tue, Nov 3, 2020 at 1:54 AM John Smith wrote:
> Sorry, got confused with your
Hi Claude,
I agree that you should be able to restart individual pods with a changed
memory configuration. Can you share the full Jobmanager log of the failed
restart attempt?
I don't think that the log statement you've posted explains a start failure.
Regards,
Robert
On Tue, Nov 3, 2020 at
Hi,
from the messages you've sent on the user@ mailing list in the recent
weeks, I see that you are in the process of learning Flink. The Flink
community won't be able to provide you with full, runnable examples for
every method Flink provides.
Rather, we have a few running examples, and
FYI: For the sake of completeness, I have added some reasoning to all the
JIRA tickets why we are not backporting fixes to the 1.11-line of Flink.
On Mon, Oct 26, 2020 at 4:51 PM Robert Metzger wrote:
> Hey Suchithra,
> thanks a lot for this report. I'm in the process of closi
versions)
>
> This means that we need to duplicate the configuration in the
> flink-conf.yaml for each job
> instead of having a common configmap
>
> Thanks,
> Shachar
>
> On 2020/10/27 08:48:17, Robert Metzger wrote:
> > Hi Shachar,
> >
> > Why do you
Hi Shachar,
Why do you want to use the core-site.xml to configure the file system?
Since we are adding the file systems as plugins, their initialization is
customized. It might be the case that we are intentionally ignoring xml
configurations from the classpath.
You can configure the filesystem
Hi Flavio,
can you share the full stacktrace you are seeing? I'm wondering if the
error happens on the client or server side (among other questions I have).
On Mon, Oct 26, 2020 at 5:58 PM Flavio Pompermaier
wrote:
> Hi to all,
> I was trying to use the RestClusterClient to submit my job to the
Hi Rex,
1. You can also use the Flink UI for retrieving logs. That usually works
quite fast (unless your logs are huge).
2. These are the correct configuration files for setting the log level. Are
you running on a vanilla EMR cluster, or are there modifications? The
"problem" is that Flink on
Hi Vijayendra,
can you post or upload the entire logs, so that we can see the Classpath
logged on startup, as well as the effective configuration parameters?
On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav
wrote:
> Hi Chesnay,
>
> Another log message:
>
> 2020-10-26 23:33:08,516 WARN
>
Hey Diwakar,
how are you deploying Flink on EMR? Are you using YARN?
If so, you could also use log aggregation to see all the logs at once (from
both JobManager and TaskManagers). (yarn logs -applicationId )
Could you post (or upload somewhere) all logs you have of one run? It is
much easier for
Hey Averell,
to clarify: You should be able to migrate using a savepoint from 1.10 to
1.11. Restoring from the state stored in Zookeeper (for HA) with a newer
Flink version won't work.
On Mon, Oct 26, 2020 at 5:05 PM Robert Metzger wrote:
> Hey Averell,
>
> you should be able t
Hey Kevin,
setting the uid is not needed for exactly-once guarantees. It is used if
you want to restore the operator state manually using a savepoint.
This blog blog post (there are probably a lot more explaining this) could
be helpful to understand how the checkpointing ensures exactly once
Hey Averell,
you should be able to migrate savepoints from Flink 1.10 to 1.11.
Is there a simple way for me to reproduce this issue locally? This seems to
be a rare, but probably valid issue. Are you using any special operators?
(like the new source API?)
Best,
Robert
On Wed, Oct 21, 2020 at
Hey Suchithra,
thanks a lot for this report. I'm in the process of closing all the tickets
Till has created (by pushing version upgrades to Flink).
The fixes will be released with the upcoming Flink 1.12 release.
I have decided against backporting the fixes to the 1.11 line of Flink,
because they
Hey Piyush,
thanks a lot for raising this concern. I believe we should keep Mesos in
Flink then in the foreseeable future.
Your offer to help is much appreciated. We'll let you know once there is
something.
On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang wrote:
> Thanks Kostas. If there's items
Hi all,
I wanted to discuss if it makes sense to remove support for Mesos in Flink.
It seems that nobody is actively maintaining that component (except for
necessary refactorings because of interfaces we are changing), and there
are almost no users reporting issues or asking for features.
The
interface. I got blocked using the Table
>> SQL interface and figured I'd try the SQL Client to see if I could get
>> unblocked.
>>
>>
>> On Fri, Sep 11, 2020 at 11:18 AM Robert Metzger
>> wrote:
>>
>>> Hi Dan,
>>>
>>> the nota
Hi Averell,
as far as I know these tmp files should be removed when the Flink job is
recovering. So you should have these files around only for the latest
incomplete checkpoint while recovery has not completed yet.
On Tue, Sep 1, 2020 at 2:56 AM Averell wrote:
> Hello Robert, Arvid,
>
> As I am
Hi Dan,
the notation of "flink-jobmanager/10.98.253.58:8081" is not a problem. It
is how java.net.InetAddress stringifies a resolved address (with both
hostname and IP).
How did you configure the SQL client to work with a Kubernetes Session?
Afaik this is not a documented, tested and officially
Hi Vijay,
Can you post the error you are referring to?
Did you properly set up an s3 plugin (
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/) ?
On Fri, Sep 11, 2020 at 8:42 AM Vijay Balakrishnan
wrote:
> Hi,
>
> I want to *get data from S3 and process and send to
Hi Prashant,
My initial suspicion is that this is a problem in the UI or with the
network connection from the browser to the Flink REST endpoints.
Since you can access the metrics with "curl", Flink seems to do
everything all right.
The first URL you posted is for the watermarks (it ends with
Hi Piper,
I personally like looking at the system load (if Flink is the only major
process on the system). It nicely captures the "stress" Flink puts on the
system (this would be the "System.CPU.Load5min class of metrics") (there
are a lot of articles about understanding linux load averages)
I
Hi Marek,
what you are describing is a known problem in Flink. There are some
thoughts on how to address this in
https://issues.apache.org/jira/browse/FLINK-11499 and
https://issues.apache.org/jira/browse/FLINK-17505
Maybe some ideas there help you already for your current problem (use long
Hi Ken,
Some random ideas that pop up in my head:
- make sure you use data types that are efficient to serialize, and cheap
to compare (ideally use primitive types in TupleN or POJOs)
- Maybe try the TableAPI batch support (if you have time to experiment).
- optimize memory usage on the
Hi Pierre,
It seems that the community is working on providing a fix with the next
1.11 bugfix release (and for 1.12). You can follow the status of the ticket
here: https://issues.apache.org/jira/browse/FLINK-18934
Best,
Robert
On Thu, Sep 10, 2020 at 11:00 AM Pierre Bedoucha
wrote:
> Hi and
Hi Sunitha,
(Note: You've emailed both the dev@ and user@ mailing list. Please only use
the user@ mailing list for questions on how to use Flink. I'm moving the
dev@ list to bcc)
Flink does not have facilities for scheduling batch jobs, and there are no
plans to add such a feature (this is not
Thanks a lot for reporting this problem here Cristian!
I am not super familiar with the involved components, but the behavior you
are describing doesn't sound right to me.
Which entrypoint are you using? This is logged at the beginning, like this:
"2020-09-08 14:45:32,807 INFO
s what is wrong there.
>
> I think there are actually two issues - the first one is the HBase
> InputFormat does not close a connection in close().
> Another is DataSourceNode not calling the close() method.
>
> Cheers,
> Mark
>
> ‐‐‐ Original Message ‐‐‐
> On Thur
Hi Prasanna,
(General remark: For such questions, please send the email only to
user@flink.apache.org. There's no need to email to dev@ as well.)
I don't think Flink can do much if the library you are using isn't throwing
exceptions. Maybe the library has other means of error reporting (a
Hi Vishwas,
Your scenario sounds like RocksDB would actually be recommended. I would
always suggest to start with RocksDB, unless your state is really small
compared to the available memory, or you need to optimize for performance.
But maybe your job is running fine with RocksDB (performance
I don't think these error messages give us a hint why you can't see the
metrics (because they are about registering metrics, not reporting them)
Are you sure you are using the right configuration parameters for Flink
1.10? That all required JARs are in the lib/ folder (on all machines) and
that
Hi,
I guess you've loaded the S3 filesystem using the s3 FS plugin.
You need to put the right jar file containing the SAX2 driver class into
the plugin directory where you've also put the S3 filesystem plugin.
You can probably find out the name of the right sax2 jar file from your
local setup
Hi,
Check out the KafkaDeserializationSchema (
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/kafka.html#the-deserializationschema)
which allows you to deserialize the key and value bytes coming from Kafka.
Best,
Robert
On Thu, Aug 27, 2020 at 1:56 PM Manas Kale
Congratulations Dian!
On Thu, Aug 27, 2020 at 3:09 PM Congxian Qiu wrote:
> Congratulations Dian
> Best,
> Congxian
>
>
> Xintong Song 于2020年8月27日周四 下午7:50写道：
>
> > Congratulations Dian~!
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Aug 27, 2020 at 7:42 PM Jark Wu wrote:
>
Congratulations Dian!
On Thu, Aug 27, 2020 at 3:09 PM Congxian Qiu wrote:
> Congratulations Dian
> Best,
> Congxian
>
>
> Xintong Song 于2020年8月27日周四 下午7:50写道：
>
> > Congratulations Dian~!
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Aug 27, 2020 at 7:42 PM Jark Wu wrote:
>
Hi Mark,
Thanks a lot for your message and the good investigation! I believe you've
found a bug in Flink. I filed an issue for the problem:
https://issues.apache.org/jira/browse/FLINK-19064.
Would you be interested in opening a pull request to fix this?
Otherwise, I'm sure a committer will pick
Hi,
Also, can we increase parallel processing, beyond the number of
> kafka partitions that we have, without causing any overhead ?
Yes, the Kafka sources produce a tiny bit of overhead, but the potential
benefit of having downstream operators at a high parallelism might be much
bigger.
How
Hi Vijayendra,
I'm not sure if -yD is the right argument as you've posted it: It is meant
to be used for Flink configuration keys, not for JVM properties.
With the Flink configuration "env.java.opts", you should be able to pass
JVM properties.
This should work: -yD env.java.opts="-D
Hi Forideal,
When using RocksDB, we need to serialize the data (to store it on disk),
whereas when using the memory backend, the data (in this
case RedConcat.ConcatString instances) is on the heap, thus we won't run
into this issue.
Are you registering your custom types in the ExecutionConfig?
101 - 200 of 901 matches
Mail list logo
