True. It would be easier for the users to understand if JM and TM have
corresponding config options. Maybe we can have another FLIP addressing
this afterwards.
Thank you~
Xintong Song
On Mon, Sep 9, 2019 at 11:38 PM Stephan Ewen wrote:
> One thing that I just came across: Some of these
One thing that I just came across: Some of these options should also have a
corresponding value for the JobManager, like JVM overhead, metaspace,
direct memory.
On Fri, Sep 6, 2019 at 4:34 AM Xintong Song wrote:
> Thanks all for the votes.
> So far, we have
>
>- 4 binding +1 votes (Stephan,
Thanks all for the votes.
So far, we have
- 4 binding +1 votes (Stephan, Andrey, Till, Zhijiang)
- 2 un-binding +1 votes (Xintong, Yu)
- No -1 votes
There are more than 3 binding +1 votes and no -1 votes, and the voting time
has past. According to the new bylaws, I'm happily to announce
+1 (non-binding) from my side.
@Yu, thanks for the vote.
- The current FLIP document already mentioned the issue that Unsafe is not
supported in Java 12, in the section 'Unifying Explicit and Implicit Memory
Allocation'. It makes sense to me to emphasize this by adding a separate
limitation
+1 (non-binding)
Minor:
1. Is it worth a separate "Limitations" section to contain all relative
information like the Unsafe support issue in Java 12, just like many other
FLIPs?
2. About the config names, if we change them later in PR, does it mean we
will need to update the FLIP document? If so,
Let's not block on config key names, just go ahead and we figure this out
concurrently or on the PR later.
On Wed, Sep 4, 2019 at 3:48 PM Stephan Ewen wrote:
> Maybe to clear up confusion about my suggestion:
>
> I would vote to keep the name of the config parameter
>
Maybe to clear up confusion about my suggestion:
I would vote to keep the name of the config parameter
"taskmanager.memory.network" because it is the same key as currently (good
to not break things unless good reason) and there currently is no case or
even a concrete follow-up where we would
@till
> Just to clarify Xintong, you suggest that Task off-heap memory represents
> direct and native memory. Since we don't know how the user will allocate
> the memory we will add this value to -XX:MaxDirectMemorySize so that the
> process won't fail if the user allocates only direct memory and
Just to clarify Xintong, you suggest that Task off-heap memory represents
direct and native memory. Since we don't know how the user will allocate
the memory we will add this value to -XX:MaxDirectMemorySize so that the
process won't fail if the user allocates only direct memory and no native
@Zhijiang @Stephan
I agree with @Xintong for the scope of the shuffle memory.
but as @Zhijinag pointed out it is not easy to estimate real netty shuffle
memory consumption due to the overhead.
Everything that is pretty much O(1) comparing to the shuffle buffer size
can be accommodated in the
@Stephan
Not sure what do you mean by "just having this value". Are you suggesting
that having "taskmanager.memory.network" refers to "shuffle" and "other
network memory", or only "shuffle"?
I guess what you mean is only "shuffle"? Because currently
"taskmanager.network.memory" refers to shuffle
If we later split the network memory into "shuffle" and "other network
memory", I think it would make sense to split the option then.
In that case "taskmanager.memory.network" would probably refer to the total
network memory, which would most likely be what most users actually
configure.
My
Thanks for creating this FLIP and starting the vote Xintong.
+1 for the proposal from my side.
I agree with Stephan that we might wanna revisit some of the configuration
names.
If I understood it correctly, then Task Off-heap memory represents the
direct memory used by the user code, right? How
Thanks for starting the vote Xintong
Also +1 for the proposed FLIP-49.
@Stephan regarding namings: network vs shuffle.
My understanding so far was that the network memory is what we basically
give to Shuffle implementations and default netty implementation uses it in
particular mostly for
+1 to the proposal in general
A few things seems to be a bit put of sync with the latest discussions
though.
The section about JVM Parameters states that the
-XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle
Memory and JVM Overhead.
The way I understand the last discussion
Hi everyone,
I'm here to re-start the voting process for FLIP-49 [1], with respect to
consensus reached in this thread [2] regarding some new comments and
concerns.
This voting will be open for at least 72 hours. I'll try to close it Sep.
5, 14:00 UTC, unless there is an objection or not enough
Alright, then let's keep the discussion in the DISCUSS mailing thread, and
see whether we need to restart the vote.
Thank you~
Xintong Song
On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann wrote:
> I had a couple of comments concerning the implementation plan. I've posted
> them to the
I had a couple of comments concerning the implementation plan. I've posted
them to the original discussion thread. Depending on the outcome of this
discussion we might need to restart the vote.
Cheers,
Till
On Tue, Aug 27, 2019 at 11:14 AM Xintong Song wrote:
> Hi all,
>
> I would like to
18 matches
Mail list logo
