Re: [Lightning-dev] Code for sub second runtime of piecewise linarization to quickly approximate the minimum convex cost flow problem (makes fast multi part payments with large amounts possible)

2022-03-20 Thread Stefan Richter
Good morning everyone, with regards to zerobasefee, I think that the argument that HTLCs are costly doesn't quite hold up because they are always free to an attacker as it stands. However, I fully agree with Zmn's opinion that it's not necessary to bang our head against any opposition to this

Re: [Lightning-dev] Route reliability<->fee trade-off control parameter

2021-11-22 Thread Stefan Richter
I also don't believe putting a choice of more or less seconds expectation in the UI makes for a great user experience. IMHO the goal should just be: give the user an estimate of fees necessary to succeed within a reasonable time. Maybe give them an option to optimize for fees only if they are

Re: [Lightning-dev] Route reliability<->fee trade-off control parameter

2021-11-15 Thread Stefan Richter
It seems to me there can be no such law unless P=NP. Which would also imply Bitcoin is worthless. Joost Jager schrieb am Mo., 15. Nov. 2021, 13:44: > One direction that I explored is to start with a statement by the user in > this form: > > "If there is a route with a success probability of

Re: [Lightning-dev] Route reliability<->fee trade-off control parameter

2021-11-15 Thread Stefan Richter
Actually, if you look into our paper, the theory tells us the following: 1) A weighted sum of different cost aspects is attractive because it remains convex if all the aspects are convex themselves. This cannot be said of other methods like the harmonic mean, which kind of forces our hand if we

Re: [Lightning-dev] Handling nonzerobasefee when using Pickhard-Richter algo variants

2021-08-30 Thread Stefan Richter
Hi Zmn! While you have some interesting thoughts about the implementation of min-cost flow based routing, I feel that there are at least two grave misunderstandings here: First, the problem with the base-fee or any similar constant is that they make the fee function concave at the point where

Re: [Lightning-dev] Fee Budgets: A Possible Path Towards Unified Cost Functions For Lightning Pathfinding Problems

2021-08-24 Thread Stefan Richter
Good Morning Zmn! If you'd like to understand the min-cost flow problem and algorithms better, I would really recommend the textbook we have been citing throughout the paper. The algorithm you have found has a few shortcomings. It'll only work for the linear min-cost flow problem, and it is

Re: [Lightning-dev] Fee Budgets: A Possible Path Towards Unified Cost Functions For Lightning Pathfinding Problems

2021-08-21 Thread Stefan Richter
Hi Zmn! That is some amazing lateral thinking you have been applying there. I'm quite certain I haven't understood everything fully, but it has been highly entertaining to read. Will have to give it a closer read when I get some time. As a first impression, here are some preliminary observations:

Re: [Lightning-dev] #zerobasefee

2021-08-16 Thread Stefan Richter
Hi Zmn et al., >I propose that the algorithm be modified >as such, that is, it *ignore* the fee scheme. We actually started out thinking like this in the event we couldn't find a proper way to handle fees, and the real world experiments we've done so far have only involved probability costs, no

Re: [Lightning-dev] #zerobasefee

2021-08-15 Thread Stefan Richter
Good morning Zmn! That is indeed precisely what we do. We usually quantize the min-cost flow into minimum shares of, say, 10kSat to 100kSat. This makes the algorithm run faster and loses very little precision. It also gives a simple way of dealing with (reasonable) min-htlc-size values. Cheers,

Re: [PATCH] firewire: Use bitwise instead of arithmetic operator for flags

2021-03-14 Thread Stefan Richter
on(card, TCODE_WRITE_QUADLET_REQUEST, > device->node_id, generation, device->max_speed, > - CSR_REGISTER_BASE + CSR_BROADCAST_CHANNEL, > + CSR_REGISTER_BASE | CSR_BROADCAST_CHANNEL, >

Re: [PATCH] firewire: prevent integer overflow on 32bit systems

2021-03-02 Thread Stefan Richter
c(sizeof(*e) + request->length, GFP_KERNEL); > if (e == NULL) > return -ENOMEM; There is already a length check for asynchronous stream requests. It happens in ioctl_send_stream_packet(). -- Stefan Richter -==--=-= --== ---=- http://arcgraph.de/sr/

Re: [DISCUSS] FLIP-151: Incremental snapshots for heap-based state backend

2020-11-09 Thread Stefan Richter
Hi, Very happy to see that the incremental checkpoint idea is finally becoming a reality for the heap backend! Overall the proposal looks pretty good to me. Just wanted to point out one possible improvement from what I can still remember from my ideas back then: I think you can avoid doing

Re: [PATCH v2] firewire-core: remove cast of function callback

2020-05-24 Thread Stefan Richter
urn -EINVAL; - - cb = iso_callback; break; case FW_ISO_CONTEXT_RECEIVE_MULTICHANNEL: - cb = (fw_iso_callback_t)iso_mc_callback; break; default: @@ -990,9 +984,15 @@ static int ioctl_create_iso_context(struct client *client, union ioctl_arg *arg) } context = fw_iso_context_create(client->device->card, a->type, - a->channel, a->speed, a->header_size, cb, client); + a->channel, a->speed, a->header_size, NULL, client); if (IS_ERR(context)) return PTR_ERR(context); + + if (a->type == FW_ISO_CONTEXT_RECEIVE_MULTICHANNEL) + context->callback.mc = iso_mc_callback; + else + context->callback.sc = iso_callback; + if (client->version < FW_CDEV_VERSION_AUTO_FLUSH_ISO_OVERFLOW) context->drop_overflow_headers = true; -- Stefan Richter -==--=-- -=-= ==--= http://arcgraph.de/sr/

Re: [Y2038] [PATCH 16/16] firewire: ohci: stop using get_seconds() for BUS_TIME

2019-11-13 Thread Stefan Richter
external clocks. > > If we instead need UTC but can live with clock jumps or overflows, > then we should use ktime_get_real_seconds() instead, retaining the > existing behavior. > > Reviewed-by: Clemens Ladisch > Cc: Stefan Richter > Link: https://lore.kernel.org/lkml/201807

[jira] [Created] (FLINK-13325) Add test case for FLINK-13249

2019-07-18 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-13325: -- Summary: Add test case for FLINK-13249 Key: FLINK-13325 URL: https://issues.apache.org/jira/browse/FLINK-13325 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13325) Add test case for FLINK-13249

2019-07-18 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-13325: -- Summary: Add test case for FLINK-13249 Key: FLINK-13325 URL: https://issues.apache.org/jira/browse/FLINK-13325 Project: Flink Issue Type: Sub-task

[jira] [Closed] (FLINK-13249) Distributed Jepsen test fails with blocked TaskExecutor

2019-07-18 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-13249. -- Resolution: Fixed Merged in: master: 23bd23b325 release-1.9: 3eff6387b5 > Distributed Jep

[jira] [Closed] (FLINK-13256) Periodical checkpointing is stopped after failovers

2019-07-18 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-13256. -- Resolution: Fixed Merged in: master: 1ec34249a0 release-1.9: b7bfafca14 > Periodi

[jira] [Commented] (FLINK-13249) Distributed Jepsen test fails with blocked TaskExecutor

2019-07-16 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885981#comment-16885981 ] Stefan Richter commented on FLINK-13249: My tendency is also more towards having a well

[jira] [Commented] (FLINK-13249) Distributed Jepsen test fails with blocked TaskExecutor

2019-07-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885059#comment-16885059 ] Stefan Richter commented on FLINK-13249: [~till.rohrmann] alright, I think then we have

[jira] [Commented] (FLINK-13249) Distributed Jepsen test fails with blocked TaskExecutor

2019-07-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885002#comment-16885002 ] Stefan Richter commented on FLINK-13249: [~gaoyunhaii] That is also how I understood your

[jira] [Commented] (FLINK-13249) Distributed Jepsen test fails with blocked TaskExecutor

2019-07-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884960#comment-16884960 ] Stefan Richter commented on FLINK-13249: [~gaoyunhaii] thanks for the input, I think what you

[jira] [Updated] (FLINK-12958) Integrate AsyncWaitOperator with mailbox

2019-07-12 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-12958: --- Labels: (was: pull-request-available) > Integrate AsyncWaitOperator with mail

[jira] [Created] (FLINK-13248) Enhance mailbox executor with yield-to-downstream functionality

2019-07-12 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-13248: -- Summary: Enhance mailbox executor with yield-to-downstream functionality Key: FLINK-13248 URL: https://issues.apache.org/jira/browse/FLINK-13248 Project: Flink

[jira] [Created] (FLINK-13248) Enhance mailbox executor with yield-to-downstream functionality

2019-07-12 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-13248: -- Summary: Enhance mailbox executor with yield-to-downstream functionality Key: FLINK-13248 URL: https://issues.apache.org/jira/browse/FLINK-13248 Project: Flink

[jira] [Assigned] (FLINK-12958) Integrate AsyncWaitOperator with mailbox

2019-07-12 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-12958: -- Assignee: (was: Stefan Richter) > Integrate AsyncWaitOperator with mail

[jira] [Closed] (FLINK-12804) Introduce mailbox-based ExecutorService

2019-07-12 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12804. -- Resolution: Implemented Merged in: master: 6cf98b671d > Introduce mailbox-ba

[jira] [Updated] (FLINK-12804) Introduce mailbox-based ExecutorService

2019-07-12 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-12804: --- Fix Version/s: (was: 1.9.0) 1.10.0 > Introduce mailbox-ba

[jira] [Commented] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently

2019-07-10 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881825#comment-16881825 ] Stefan Richter commented on FLINK-13072: I think it is surely not a bad idea to improve

[jira] [Closed] (FLINK-13063) AsyncWaitOperator shouldn't be releasing checkpointingLock

2019-07-09 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-13063. -- Resolution: Fixed Release Note: This changes the default chaining behavior

[jira] [Commented] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently

2019-07-09 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881149#comment-16881149 ] Stefan Richter commented on FLINK-13072: [~zicat] After reading my previous comment, do you

[jira] [Closed] (FLINK-12730) Combine BitSet implementations in flink-runtime

2019-07-09 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12730. -- Resolution: Implemented Merged in: master: 14c4b23 > Combine BitSet implementati

[jira] [Closed] (FLINK-12693) Store state per key-group in CopyOnWriteStateTable

2019-07-09 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12693. -- Resolution: Implemented Merged in: master: 8f47b38 > Store state per key-gr

[jira] [Assigned] (FLINK-13063) AsyncWaitOperator shouldn't be releasing checkpointingLock

2019-07-05 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-13063: -- Assignee: Stefan Richter (was: Piotr Nowojski) > AsyncWaitOperator should

[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently

2019-07-05 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879099#comment-16879099 ] Stefan Richter edited comment on FLINK-13072 at 7/5/19 9:21 AM: [~zicat

[jira] [Commented] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently

2019-07-05 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879099#comment-16879099 ] Stefan Richter commented on FLINK-13072: [~zicat] All operators in Flink are assumed

[jira] [Commented] (FLINK-7289) Memory allocation of RocksDB can be problematic in container environments

2019-06-27 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874318#comment-16874318 ] Stefan Richter commented on FLINK-7289: --- [~mikekap] I think that is a very good suggestion

[jira] [Closed] (FLINK-11662) Discarded checkpoint can cause Tasks to fail

2019-06-27 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-11662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-11662. -- Resolution: Fixed Fix Version/s: 1.9.0 Merged in: master: b760d55 > Discar

[jira] [Created] (FLINK-12958) Integrate AsyncWaitOperator with mailbox

2019-06-24 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12958: -- Summary: Integrate AsyncWaitOperator with mailbox Key: FLINK-12958 URL: https://issues.apache.org/jira/browse/FLINK-12958 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-12958) Integrate AsyncWaitOperator with mailbox

2019-06-24 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12958: -- Summary: Integrate AsyncWaitOperator with mailbox Key: FLINK-12958 URL: https://issues.apache.org/jira/browse/FLINK-12958 Project: Flink Issue Type: Sub

[jira] [Closed] (FLINK-12364) Introduce a CheckpointFailureManager to centralized manage checkpoint failure

2019-06-19 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12364. -- Resolution: Implemented Merged in: master: 8c57e5ae > Introduce a CheckpointFailureMana

Re: [DISCUSS] Change threading-model in StreamTask to a mailbox-based approach

2019-06-12 Thread Stefan Richter
As there have been no objections, we are moving forward with this effort. Progress is tracked here: https://issues.apache.org/jira/browse/FLINK-12477 <https://issues.apache.org/jira/browse/FLINK-12477> Best, Stefan > On 7. May 2019, at 17:39, Stefan Richter wrote: > > H

[jira] [Created] (FLINK-12804) Introduce mailbox-based ExecutorService

2019-06-11 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12804: -- Summary: Introduce mailbox-based ExecutorService Key: FLINK-12804 URL: https://issues.apache.org/jira/browse/FLINK-12804 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-12804) Introduce mailbox-based ExecutorService

2019-06-11 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12804: -- Summary: Introduce mailbox-based ExecutorService Key: FLINK-12804 URL: https://issues.apache.org/jira/browse/FLINK-12804 Project: Flink Issue Type: Sub

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

2019-06-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855013#comment-16855013 ] Stefan Richter commented on FLINK-12070: I have one more additional question: can you share

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Stefan Richter
Hi Yu, Sorry for the late reaction. As already discussed internally, I think this is a very good proposal and design that can help to improve a major limitation of the current state backend. I think that most discussion is happening in the design doc and I left my comments there. Looking

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Stefan Richter
Hi Yu, Sorry for the late reaction. As already discussed internally, I think this is a very good proposal and design that can help to improve a major limitation of the current state backend. I think that most discussion is happening in the design doc and I left my comments there. Looking

[jira] [Comment Edited] (FLINK-8871) Checkpoint cancellation is not propagated to stop checkpointing threads on the task manager

2019-05-25 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848118#comment-16848118 ] Stefan Richter edited comment on FLINK-8871 at 5/25/19 9:58 AM: After

[jira] [Commented] (FLINK-8871) Checkpoint cancellation is not propagated to stop checkpointing threads on the task manager

2019-05-25 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848118#comment-16848118 ] Stefan Richter commented on FLINK-8871: --- After this quiet intense exchange, I was thinking a bit

[jira] [Assigned] (FLINK-12482) Make checkpoint trigger/notifyComplete run via the mailbox queue

2019-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-12482: -- Assignee: Stefan Richter (was: vinoyang) > Make checkpoint trigger/notifyCompl

[jira] [Closed] (FLINK-12483) Support (legacy) SourceFunctions with mailbox

2019-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12483. -- Resolution: Implemented Fix Version/s: 1.9.0 Merged in: master ead91396 > Supp

[jira] [Closed] (FLINK-12480) Introduce mailbox to StreamTask main-loop

2019-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12480. -- Resolution: Implemented Merged in: master 022f6cce > Introduce mailbox to StreamTask m

[jira] [Closed] (FLINK-12478) Decompose monolithic run-loops in StreamTask implementations into step-wise calls

2019-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12478. -- Resolution: Implemented Merged in: master 022f6cce > Decompose monolithic run-lo

[jira] [Created] (FLINK-12591) StatsDReporterTest is unstable

2019-05-22 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12591: -- Summary: StatsDReporterTest is unstable Key: FLINK-12591 URL: https://issues.apache.org/jira/browse/FLINK-12591 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-12591) StatsDReporterTest is unstable

2019-05-22 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12591: -- Summary: StatsDReporterTest is unstable Key: FLINK-12591 URL: https://issues.apache.org/jira/browse/FLINK-12591 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-10855) CheckpointCoordinator does not delete checkpoint directory of late/failed checkpoints

2019-05-21 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844829#comment-16844829 ] Stefan Richter commented on FLINK-10855: [~yanghua] Just a heads-up that you pinged the wrong

[jira] [Comment Edited] (FLINK-6755) Allow triggering Checkpoints through command line client

2019-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842210#comment-16842210 ] Stefan Richter edited comment on FLINK-6755 at 5/17/19 2:30 PM: IMO

[jira] [Comment Edited] (FLINK-6755) Allow triggering Checkpoints through command line client

2019-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842210#comment-16842210 ] Stefan Richter edited comment on FLINK-6755 at 5/17/19 2:24 PM: IMO

[jira] [Comment Edited] (FLINK-6755) Allow triggering Checkpoints through command line client

2019-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842210#comment-16842210 ] Stefan Richter edited comment on FLINK-6755 at 5/17/19 2:25 PM: IMO

[jira] [Commented] (FLINK-6755) Allow triggering Checkpoints through command line client

2019-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842210#comment-16842210 ] Stefan Richter commented on FLINK-6755: --- IMO, this somehow leads to the more general question about

[jira] [Commented] (FLINK-10855) CheckpointCoordinator does not delete checkpoint directory of late/failed checkpoints

2019-05-16 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841428#comment-16841428 ] Stefan Richter commented on FLINK-10855: @vinoyang i guess i cannot. I was just testing

[jira] [Assigned] (FLINK-12483) Support (legacy) SourceFunctions with mailbox

2019-05-14 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-12483: -- Assignee: Stefan Richter (was: vinoyang) > Support (legacy) SourceFuncti

[jira] [Commented] (FLINK-12481) Make processing time timer trigger run via the mailbox

2019-05-13 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838391#comment-16838391 ] Stefan Richter commented on FLINK-12481: Hi [~aitozi] and thanks for offering some help. Can we

[jira] [Assigned] (FLINK-12480) Introduce mailbox to StreamTask main-loop

2019-05-10 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-12480: -- Assignee: Stefan Richter > Introduce mailbox to StreamTask main-l

[jira] [Created] (FLINK-12484) Drop unnecessary locking for code-paths that now use the mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12484: -- Summary: Drop unnecessary locking for code-paths that now use the mailbox Key: FLINK-12484 URL: https://issues.apache.org/jira/browse/FLINK-12484 Project: Flink

[jira] [Created] (FLINK-12484) Drop unnecessary locking for code-paths that now use the mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12484: -- Summary: Drop unnecessary locking for code-paths that now use the mailbox Key: FLINK-12484 URL: https://issues.apache.org/jira/browse/FLINK-12484 Project: Flink

[jira] [Created] (FLINK-12483) Support (legacy) SourceFunctions with mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12483: -- Summary: Support (legacy) SourceFunctions with mailbox Key: FLINK-12483 URL: https://issues.apache.org/jira/browse/FLINK-12483 Project: Flink Issue Type

[jira] [Created] (FLINK-12483) Support (legacy) SourceFunctions with mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12483: -- Summary: Support (legacy) SourceFunctions with mailbox Key: FLINK-12483 URL: https://issues.apache.org/jira/browse/FLINK-12483 Project: Flink Issue Type

[jira] [Updated] (FLINK-12477) Change threading-model in StreamTask to a mailbox-based approach

2019-05-10 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-12477: --- Description: This is the umbrella issue to change the threading-model in StreamTask

[jira] [Created] (FLINK-12482) Make checkpoint trigger/notifyComplete run via the mailbox queue

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12482: -- Summary: Make checkpoint trigger/notifyComplete run via the mailbox queue Key: FLINK-12482 URL: https://issues.apache.org/jira/browse/FLINK-12482 Project: Flink

[jira] [Created] (FLINK-12482) Make checkpoint trigger/notifyComplete run via the mailbox queue

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12482: -- Summary: Make checkpoint trigger/notifyComplete run via the mailbox queue Key: FLINK-12482 URL: https://issues.apache.org/jira/browse/FLINK-12482 Project: Flink

[jira] [Created] (FLINK-12481) Make processing time timer trigger run via the mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12481: -- Summary: Make processing time timer trigger run via the mailbox Key: FLINK-12481 URL: https://issues.apache.org/jira/browse/FLINK-12481 Project: Flink

[jira] [Created] (FLINK-12481) Make processing time timer trigger run via the mailbox

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12481: -- Summary: Make processing time timer trigger run via the mailbox Key: FLINK-12481 URL: https://issues.apache.org/jira/browse/FLINK-12481 Project: Flink

[jira] [Created] (FLINK-12480) Introduce mailbox to StreamTask main-loop

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12480: -- Summary: Introduce mailbox to StreamTask main-loop Key: FLINK-12480 URL: https://issues.apache.org/jira/browse/FLINK-12480 Project: Flink Issue Type

[jira] [Created] (FLINK-12480) Introduce mailbox to StreamTask main-loop

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12480: -- Summary: Introduce mailbox to StreamTask main-loop Key: FLINK-12480 URL: https://issues.apache.org/jira/browse/FLINK-12480 Project: Flink Issue Type

[jira] [Created] (FLINK-12478) Decompose monolithic run-loops in StreamTask implementations into step-wise calls

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12478: -- Summary: Decompose monolithic run-loops in StreamTask implementations into step-wise calls Key: FLINK-12478 URL: https://issues.apache.org/jira/browse/FLINK-12478

[jira] [Created] (FLINK-12479) Make InputProcessors and StreamIterationHead non-blocking

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12479: -- Summary: Make InputProcessors and StreamIterationHead non-blocking Key: FLINK-12479 URL: https://issues.apache.org/jira/browse/FLINK-12479 Project: Flink

[jira] [Created] (FLINK-12479) Make InputProcessors and StreamIterationHead non-blocking

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12479: -- Summary: Make InputProcessors and StreamIterationHead non-blocking Key: FLINK-12479 URL: https://issues.apache.org/jira/browse/FLINK-12479 Project: Flink

[jira] [Created] (FLINK-12478) Decompose monolithic run-loops in StreamTask implementations into step-wise calls

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12478: -- Summary: Decompose monolithic run-loops in StreamTask implementations into step-wise calls Key: FLINK-12478 URL: https://issues.apache.org/jira/browse/FLINK-12478

[jira] [Created] (FLINK-12477) Change threading-model in StreamTask to a mailbox-based approach

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12477: -- Summary: Change threading-model in StreamTask to a mailbox-based approach Key: FLINK-12477 URL: https://issues.apache.org/jira/browse/FLINK-12477 Project: Flink

[jira] [Created] (FLINK-12477) Change threading-model in StreamTask to a mailbox-based approach

2019-05-10 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-12477: -- Summary: Change threading-model in StreamTask to a mailbox-based approach Key: FLINK-12477 URL: https://issues.apache.org/jira/browse/FLINK-12477 Project: Flink

[DISCUSS] Change threading-model in StreamTask to a mailbox-based approach

2019-05-07 Thread Stefan Richter
Hi, we want to start a discussion about changing the threading-model in stream task to a mailbox-based approach: https://docs.google.com/document/d/1eDpsUKv2FqwZiS1Pm6gYO5eFHScBHfULKmH1-ZEWB4g/edit?usp=sharing

Re: RocksDB native checkpoint time

2019-05-03 Thread Stefan Richter
Hi, out of curiosity, does it happen with jobs that have a large number of states (column groups) or also for jobs with few column groups and just “big state”? Best, Stefan > On 3. May 2019, at 11:04, Piotr Nowojski wrote: > > Hi Gyula, > > Have you read our tuning guide? >

[jira] [Closed] (FLINK-12350) RocksDBStateBackendTest doesn't cover the incremental checkpoint code path

2019-04-29 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12350. -- Resolution: Fixed Fix Version/s: 1.8.1 Merged in: master: 9aeb4e5 release-1.8

[jira] [Closed] (FLINK-12350) RocksDBStateBackendTest doesn't cover the incremental checkpoint code path

2019-04-29 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12350. -- Resolution: Fixed Fix Version/s: 1.9.0 Merged in: master: 9aeb4e5

[jira] [Reopened] (FLINK-12350) RocksDBStateBackendTest doesn't cover the incremental checkpoint code path

2019-04-29 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reopened FLINK-12350: > RocksDBStateBackendTest doesn't cover the incremental checkpoint code p

[jira] [Closed] (FLINK-11167) Optimize RocksDBList#put for no empty input

2019-04-29 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-11167. -- Resolution: Implemented Fix Version/s: 1.9.0 Merged in: master: 03faeb9 > Optim

[jira] [Closed] (FLINK-10724) Refactor failure handling in check point coordinator

2019-04-29 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-10724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-10724. -- Resolution: Done Fix Version/s: 1.9.0 Merged in: master: c4b0e8f > Refactor fail

[jira] [Comment Edited] (FLINK-12296) Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task

2019-04-26 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826787#comment-16826787 ] Stefan Richter edited comment on FLINK-12296 at 4/26/19 9:00 AM

[jira] [Commented] (FLINK-12296) Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task

2019-04-26 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826787#comment-16826787 ] Stefan Richter commented on FLINK-12296: [~klion26] I was just taking a look into the code

Re: kafka partitions, data locality

2019-04-26 Thread Stefan Richter
Hi Sergey, The point why this I flagged as beta is actually less about stability but more about the fact that this is supposed to be more of a "power user" feature because bad things can happen if your data is not 100% correctly partitioned in the same way as Flink would partition it. This is

Re: kafka partitions, data locality

2019-04-26 Thread Stefan Richter
Hi Sergey, The point why this I flagged as beta is actually less about stability but more about the fact that this is supposed to be more of a "power user" feature because bad things can happen if your data is not 100% correctly partitioned in the same way as Flink would partition it. This is

[jira] [Commented] (FLINK-12296) Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task

2019-04-24 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825138#comment-16825138 ] Stefan Richter commented on FLINK-12296: [~klion26] In general I think it is a good idea

[jira] [Commented] (FLINK-12296) Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task

2019-04-24 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824945#comment-16824945 ] Stefan Richter commented on FLINK-12296: I guess this is one remaining relic from how backends

Re: Fast restart of a job with a large state

2019-04-18 Thread Stefan Richter
Hi, If rescaling is the problem, let me clarify that you can currently rescale from savepoints and all types of checkpoints (including incremental). If that was the only problem, then there is nothing to worry about - the documentation is only a bit conservative about this because we will not

[jira] [Closed] (FLINK-12212) Clarify documentation about async checkpointing

2019-04-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-12212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-12212. -- Resolution: Fixed Fix Version/s: 1.9.0 Merged in: master: 046d752c6e > Clar

[jira] [Closed] (FLINK-10712) RestartPipelinedRegionStrategy does not restore state

2019-04-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-10712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-10712. -- Resolution: Fixed Fix Version/s: 1.9.0 Merged in: master: c3a9293

[jira] [Commented] (FLINK-11955) Modify build to move filesystems from lib to plugins folder

2019-04-16 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-11955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818833#comment-16818833 ] Stefan Richter commented on FLINK-11955: Great, thanks! I would suggest to use option 1, because

[jira] [Resolved] (FLINK-11954) Provide unit test(s) for new plugin infrastructure

2019-04-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-11954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter resolved FLINK-11954. Resolution: Done Work is already done as part of FLINK-11953 > Provide unit tes

[jira] [Closed] (FLINK-11953) Introduce Plugin/Loading system and integrate it with FileSystem

2019-04-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-11953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-11953. -- Resolution: Implemented Fix Version/s: 1.9.0 Merged in: master: c6307b5990

<    1   2   3   4   5   6   7   8   9   10   >