then the only blocker is to guarantee we will never add back to any cache the extended buffer. This can just be a matter of reviewing our strategies but didn't get a chance to do it yet. Can surely have a look this week-end but fear i'll not be able before :(
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory <https://javaeefactory-rmannibucau.rhcloud.com> 2017-09-26 9:22 GMT+02:00 Mark Struberg <[email protected]>: > Hmm maybe, but in a later version, ok? > > Would love to start the release today, ok? > > LieGrue, > Strub > > > Am 26.09.2017 um 08:10 schrieb Romain Manni-Bucau <[email protected] > >: > > > > We already have it, what is important with this auto extend logic is a > > toggle flag to add back in the cache either the original ("small") buffer > > or the extended one ("big"). We dont have that support yet. But in all > > cases we should call release to let the strategy have a chance to cleanup > > things. > > > > Le 26 sept. 2017 08:01, "Mark Struberg" <[email protected]> a > > écrit : > > > >> Hmm, what about a keepMaxBuffers = n config? > >> > >> By default it would be e.g. 50. > >> Means if the queue is > n we do not return it. > >> By configure 0 as n we automatically have what you want, right? > >> > >> LieGrue, > >> strub > >> > >> > >>> Am 26.09.2017 um 06:59 schrieb Romain Manni-Bucau < > [email protected] > >>> : > >>> > >>> We should get a config about it and ensure we release() *any* buffer in > >>> case we go native (validate our lifecycle). This mean we should > >> distinguish > >>> between release (end of buffer usage) and going back to pool (or the > >>> opposite: newPoolableBuffer vs newVolatileBuffer). > >>> > >>> The test is not super good yet but a start, it misses some size > asserts. > >>> > >>> > >>> Le 25 sept. 2017 23:44, "Mark Struberg" <[email protected]> a > >>> écrit : > >>> > >>>> Not quite sure what this should tell us. > >>>> > >>>> Of course the exceeded buffer doesn't get returned! > >>>> We might probably return the original one and thus improve our logic. > >>>> But it's not strictly wrong the way it currently behaves. > >>>> It's now way faster under normal conditions and it consumes way less > mem > >>>> than before. > >>>> > >>>> LieGrue, > >>>> strub > >>>> > >>>>> Am 25.09.2017 um 08:22 schrieb [email protected]: > >>>>> > >>>>> Repository: johnzon > >>>>> Updated Branches: > >>>>> refs/heads/master cc24e8e1c -> 60f48cf65 > >>>>> > >>>>> > >>>>> adding a broken test to show why previous commit broke the buffer > >>>> strategies > >>>>> > >>>>> > >>>>> Project: http://git-wip-us.apache.org/repos/asf/johnzon/repo > >>>>> Commit: http://git-wip-us.apache.org/repos/asf/johnzon/commit/ > 60f48cf6 > >>>>> Tree: http://git-wip-us.apache.org/repos/asf/johnzon/tree/60f48cf6 > >>>>> Diff: http://git-wip-us.apache.org/repos/asf/johnzon/diff/60f48cf6 > >>>>> > >>>>> Branch: refs/heads/master > >>>>> Commit: 60f48cf65708e3a9bc9943b6d5cac0435772b6e9 > >>>>> Parents: cc24e8e > >>>>> Author: Romain Manni-Bucau <[email protected]> > >>>>> Authored: Mon Sep 25 08:22:26 2017 +0200 > >>>>> Committer: Romain Manni-Bucau <[email protected]> > >>>>> Committed: Mon Sep 25 08:22:26 2017 +0200 > >>>>> > >>>>> ------------------------------------------------------------ > ---------- > >>>>> .../apache/johnzon/core/BrokenDefaultTest.java | 101 > >>>> +++++++++++++++++++ > >>>>> 1 file changed, 101 insertions(+) > >>>>> ------------------------------------------------------------ > ---------- > >>>>> > >>>>> > >>>>> http://git-wip-us.apache.org/repos/asf/johnzon/blob/ > >>>> 60f48cf6/johnzon-core/src/test/java/org/apache/johnzon/ > >>>> core/BrokenDefaultTest.java > >>>>> ------------------------------------------------------------ > ---------- > >>>>> diff --git a/johnzon-core/src/test/java/org/apache/johnzon/core/ > >> BrokenDefaultTest.java > >>>> b/johnzon-core/src/test/java/org/apache/johnzon/core/ > >>>> BrokenDefaultTest.java > >>>>> new file mode 100644 > >>>>> index 0000000..ec63d9f > >>>>> --- /dev/null > >>>>> +++ b/johnzon-core/src/test/java/org/apache/johnzon/core/ > >>>> BrokenDefaultTest.java > >>>>> @@ -0,0 +1,101 @@ > >>>>> +/* > >>>>> + * Licensed to the Apache Software Foundation (ASF) under one or > more > >>>>> + * contributor license agreements. See the NOTICE file distributed > >> with > >>>>> + * this work for additional information regarding copyright > ownership. > >>>>> + * The ASF licenses this file to You under the Apache License, > Version > >>>> 2.0 > >>>>> + * (the "License"); you may not use this file except in compliance > >> with > >>>>> + * the License. You may obtain a copy of the License at > >>>>> + * > >>>>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>>>> + * > >>>>> + * Unless required by applicable law or agreed to in writing, > software > >>>>> + * distributed under the License is distributed on an "AS IS" BASIS, > >>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >>>> implied. > >>>>> + * See the License for the specific language governing permissions > and > >>>>> + * limitations under the License. > >>>>> + */ > >>>>> +package org.apache.johnzon.core; > >>>>> + > >>>>> +import static java.util.Collections.emptyMap; > >>>>> +import static org.junit.Assert.assertEquals; > >>>>> + > >>>>> +import java.io.ByteArrayInputStream; > >>>>> +import java.io.IOException; > >>>>> +import java.io.InputStream; > >>>>> +import java.lang.reflect.Field; > >>>>> +import java.nio.charset.StandardCharsets; > >>>>> +import java.util.Queue; > >>>>> + > >>>>> +import javax.json.Json; > >>>>> +import javax.json.stream.JsonParser; > >>>>> +import javax.json.stream.JsonParserFactory; > >>>>> + > >>>>> +import org.junit.Ignore; > >>>>> +import org.junit.Test; > >>>>> + > >>>>> +public class BrokenDefaultTest { > >>>>> + > >>>>> + @Test > >>>>> + @Ignore("buggy but pushing to share the use case") > >>>>> + public void run() throws NoSuchFieldException, > >>>> IllegalAccessException { // shouldnt fail by default > >>>>> + final JsonParserFactory factory = Json.createParserFactory( > >>>> emptyMap()); > >>>>> + final int length = 1024 * 1024; > >>>>> + assertEquals(0, get(Queue.class, get( > >>>>> + BufferStrategy.BufferProvider.class, factory, > >>>> "bufferProvider"), "queue").size()); > >>>>> + try (final JsonParser parser = factory.createParser( > >> newDynamicInput(length))) > >>>> { > >>>>> + int eventCount = 0; > >>>>> + while (parser.hasNext()) { > >>>>> + eventCount++; > >>>>> + final JsonParser.Event next = parser.next(); > >>>>> + if (eventCount == 2 && next == > >>>> JsonParser.Event.VALUE_STRING) { > >>>>> + assertEquals(length, > parser.getString().length()); > >>>>> + } > >>>>> + } > >>>>> + } > >>>>> + assertEquals(1, get(Queue.class, get( > >>>>> + BufferStrategy.BufferProvider.class, factory, > >>>> "bufferProvider"), "queue").size()); > >>>>> + } > >>>>> + > >>>>> + private <T> T get(final Class<T> returnType, final Object > >> instance, > >>>> final String field) > >>>>> + throws NoSuchFieldException, IllegalAccessException { > >>>>> + Class<?> current = instance.getClass(); > >>>>> + while (current != Object.class) { > >>>>> + try { > >>>>> + final Field declaredField = > current.getDeclaredField( > >>>> field); > >>>>> + if (!declaredField.isAccessible()) { > >>>>> + declaredField.setAccessible(true); > >>>>> + } > >>>>> + return returnType.cast(declaredField. > get(instance)); > >>>>> + } catch (final NoSuchFieldException nsfe) { > >>>>> + current = current.getSuperclass(); > >>>>> + } > >>>>> + } > >>>>> + throw new IllegalAccessError(instance + " field: " + field); > >>>>> + } > >>>>> + > >>>>> + private InputStream newDynamicInput(final int size) { > >>>>> + return new InputStream() { > >>>>> + > >>>>> + private InputStream before = new > >>>> ByteArrayInputStream("{\"key\":\"".getBytes(StandardCharsets.UTF_8)); > >>>>> + > >>>>> + private InputStream after = new > >> ByteArrayInputStream("\"}". > >>>> getBytes(StandardCharsets.UTF_8)); > >>>>> + > >>>>> + private int remaining = size; > >>>>> + > >>>>> + @Override > >>>>> + public int read() throws IOException { > >>>>> + { > >>>>> + final int val = before.read(); > >>>>> + if (val >= 0) { > >>>>> + return val; > >>>>> + } > >>>>> + } > >>>>> + if (remaining < 0) { > >>>>> + return after.read(); > >>>>> + } > >>>>> + remaining--; > >>>>> + return 'a'; > >>>>> + } > >>>>> + }; > >>>>> + } > >>>>> +} > >>>>> > >>>> > >>>> > >>>> . > >>>> > >> > >> > >
