Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-138298479
Closing this to continue the discussion at #1093.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proj
Github user mxm closed the pull request at:
https://github.com/apache/flink/pull/290
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-138237563
This is subsumed by Stephan's new PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/290#discussion_r23950666
--- Diff:
flink-core/src/main/java/org/apache/flink/core/memory/DirectMemorySegment.java
---
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Softwar
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-70455214
Not entirely, all unit tests in MemorySegmentTest and MemoryManagerTest
are executed for both, the old heap and the new off-heap implementation.
Nevertheless, more
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-70432006
At a first glance, this is entirely missing test cases for the
DirectMemorySegment.
Considering that this is code that an kill the JVM with segfaults if buggy,
thi
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69921554
We could implement a DataInputDeserializer variant, which directly operates
on `MemorySegment` instances to circumvent this. I agree that this should not
be part of this PR.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69920591
@uce Yes, it will be copied from off-heap to heap first. There are also
other places like the `EventSerializer` where this is the case. I guess it
depends on the amount of dat
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69917694
Nice :-) Regarding the DataInputDeserializer: You are right that it is
currently not used, but I think with off-heap memory enabled the buffers will
always be copied to a heap
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69917146
I made some finally changes that were required for the current master.
Also, I ran all integration tests with direct memory allocation enabled. Any
objections for merging this
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69723062
@uce `DataInputDeserializer.setBuffer(ByteBuffer)` doesn't seem relevent
for this pull request. The check for the direct byte buffer is not necessary
since it is always called
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69582485
Ufuk and me just investigated the cause of the failed integration tests
with direct memory management enabled. Turns out, the compare method was
implemented incorrectly.
---
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69571778
I've just seen that there is a (pre this PR) check in
`DataInputDeserializer.setBuffer(ByteBuffer)` whether the given byte buffer is
a direct byte buffer or not. In case that
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69361811
@rmetzger I added some documentation for the config parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69345274
After running several integration tests, I discovered some issues with the
off-heap memory allocation. I'm not completely sure what is causing the issue
but I'm looking into i
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69341391
@StephanEwen Thanks for pointing that out. This should probably go into a
separate issue.
---
If your project is set up for it, you can reply to this email and have your
repl
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69321520
I just realized that the whole code of Flink actually runs only on
architectures that support unaligned memory accesses, i.e., where you can fetch
a `long` from an add
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/290#discussion_r22710693
--- Diff:
flink-core/src/main/java/org/apache/flink/core/memory/DirectMemorySegment.java
---
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Soft
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69320368
I actually like the idea that tests (when starting the mini cluster)
randomly select one of the two implementations.
We can add that after including this, thou
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/290#issuecomment-69320016
@uce To achieve optimal test coverage, we could modify all tests which use
the HeapMemorySegment to run with the DirectMemorySegment as well. Another
approach would be to rand
20 matches
Mail list logo
