subject:"File idempotent store problem"

Re: File idempotent store problem

2018-01-16 Thread Claus Ibsen

>> > throw ObjectHelper.wrapRuntimeCamelException(e);
>>> > } finally {
>>> > IOHelper.close(fos, "Trunking file idempotent repository",
>>> LOG);
>>> > }
>>> > }
>>> >
>>> > LRUCache:
>>> >
>>> > @Override
>>> > public Set keySet() {
>>> > return map.keySet();
>>> > }
>>> >
>>> > where previously it was:
>>> >
>>> > @Override
>>> > public Set keySet() {
>>> > return map.*ascendingKeySet*();
>>> > }
>>> >
>>> > Regards
>>> > Kris
>>> > --
>>> >
>>> > Pozdrawiam
>>> >
>>> > Krzysztof Hołdanowicz
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>
>>>
>>> --
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html
>>> To unsubscribe from Camel - Users, click here
>>> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=465428=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==>
>>> .
>>> NAML
>>> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>> --
>>
>> Pozdrawiam
>>
>> Krzysztof Hołdanowicz
>>
>
>
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: File idempotent store problem

2018-01-04 Thread Onder SEZGIN

  fos.write(key.getBytes());
> >> > fos.write(STORE_DELIMITER.getBytes());
> >> > }
> >> > } catch (IOException e) {
> >> > throw ObjectHelper.wrapRuntimeCamelException(e);
> >> >     } finally {
> >> > IOHelper.close(fos, "Trunking file idempotent repository",
> >> LOG);
> >> > }
> >> > }
> >> >
> >> > LRUCache:
> >> >
> >> > @Override
> >> > public Set keySet() {
> >> > return map.keySet();
> >> > }
> >> >
> >> > where previously it was:
> >> >
> >> > @Override
> >> > public Set keySet() {
> >> > return map.*ascendingKeySet*();
> >> > }
> >> >
> >> > Regards
> >> > Kris
> >> > --
> >> >
> >> > Pozdrawiam
> >> >
> >> > Krzysztof Hołdanowicz
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -
> >> http://davsclaus.com @davsclaus
> >> Camel in Action 2: https://www.manning.com/ibsen2
> >>
> >>
> >> --
> >> If you reply to this email, your message will be added to the discussion
> >> below:
> >>
> >>
> http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html
> >> To unsubscribe from Camel - Users, click here
> >> <
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=465428=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==
> >
> >> .
> >> NAML
> >> <
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >>
> > --
> >
> > Pozdrawiam
> >
> > Krzysztof Hołdanowicz
> >
>
>
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz
>

Re: File idempotent store problem

2018-01-03 Thread Krzysztof Hołdanowicz

Hi,

regarding CAMEL-12058 I don't know if you are aware of all consequences of
wrong order in the idempotent file store.
The wrong order in the file is not the problem itself as long as elemens
are added and eviceted on runtime, because caffeine provides an api for
ordering like:

   - @Override public Map<K, V> coldest(int limit)
   - @Override public Map<K, V> hottest(int limit)
   - @Override public Map<K, V> oldest(int limit),
   - @Override public Map<K, V> youngest(int limit) )

however the consequences of this appears after RESTART. The memory cache
does not contain the proper entries (in case of reaching the max limit
size) because it does not load elements from hottest to coldest but with
the file entries order hence some of the files are consumed multiple times.
It means that current implementation of file idempotent store is not usable
at all anymore. Ignoring the issue (CAMEL-12058) means that Camel library
does not provide any implementation of idempotent file store as the current
behaviour is completely wrong and causes consuming multiple times the same
file after reaching max size limit and after restarting application.


Regards
Kris

sob., 2 gru 2017 o 15:14 użytkownik Krzysztof Hołdanowicz <
holdanow...@gmail.com> napisał:

> I don't know if I understood you correctly.
> Instead of looping via cache.keySet() you mean looping via:
> Map.Entry<String, Object> entry : cache.entrySet() or cache.foreach((k, v)
> -> {...})?
>
> If yes what is the difference? Isn't it the same unordered collection?
> If Caffeince returns unordered collection how we can get ordered entries?
> Isn't it related with:
> https://github.com/ben-manes/caffeine/issues/86
>
> Shouldn't we use a kind of LinkedHashMap implementation?
>
> Regards
> Kris
>
> wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] <
> ml+s465427n5815878...@n5.nabble.com> napisał:
>
>> Hi
>>
>> Ah well spotted.
>> I think we should for loop via Map.Entry (or (k,v) via lambda) which I
>> think will be in the correct order.
>>
>> You are welcome to log a JIRA. And also work on unit test and patch.
>> http://camel.apache.org/contributing
>>
>> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz
>> <[hidden email] <http:///user/SendEmail.jtp?type=node=5815878=0>>
>> wrote:
>>
>> > Hi all,
>> >
>> > I recently noticed that there is wrong entry order in file using
>> > FileIdempotentRepository implementation.
>> > The effect is that instead of having order like:
>> >
>> > file1.txt.20171123
>> > file2.txt.20171123
>> > file1.txt.20171124
>> > file3.txt.20171125
>> > file2.txt.20171126
>> >
>> > we have:
>> >
>> > file1.txt.20171123
>> > file1.txt.20171124
>> > file2.txt.20171123
>> > file2.txt.20171126
>> > file3.txt.20171125
>> >
>> > where date extension represents order in which particular file was
>> consumed
>> > by the idempotent file consumer.
>> > As a consequence instead of initializing memory cache with newest
>> values,
>> > it is initialized (probably) based on hash function from truncStore
>> method
>> > and we consume same file more than once:
>> >
>> > protected void trunkStore() {
>> > LOG.info("Trunking idempotent filestore: {}", fileStore);
>> > FileOutputStream fos = null;
>> > try {
>> > fos = new FileOutputStream(fileStore);
>> > for (String key : *cache.keySet()*) {
>> > fos.write(key.getBytes());
>> > fos.write(STORE_DELIMITER.getBytes());
>> > }
>> > } catch (IOException e) {
>> > throw ObjectHelper.wrapRuntimeCamelException(e);
>> > } finally {
>> > IOHelper.close(fos, "Trunking file idempotent repository",
>> LOG);
>> > }
>> > }
>> >
>> > LRUCache:
>> >
>> > @Override
>> > public Set keySet() {
>> > return map.keySet();
>> > }
>> >
>> > where previously it was:
>> >
>> > @Override
>> > public Set keySet() {
>> > return map.*ascendingKeySet*();
>> > }
>> >
>> > Regards
>> > Kris
>> > --
>> >
>> > Pozdrawiam
>> >
>> > Krzysztof Hołdanowicz
>>
>>
>>
>> --
>> Claus Ibsen
>> -
>> h

Re: File idempotent store problem

2017-12-02 Thread Krzysztof Hołdanowicz

I don't know if I understood you correctly.
Instead of looping via cache.keySet() you mean looping via:
Map.Entry<String, Object> entry : cache.entrySet() or cache.foreach((k, v)
-> {...})?

If yes what is the difference? Isn't it the same unordered collection?
If Caffeince returns unordered collection how we can get ordered entries?
Isn't it related with:
https://github.com/ben-manes/caffeine/issues/86

Shouldn't we use a kind of LinkedHashMap implementation?

Regards
Kris

wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] <
ml+s465427n5815878...@n5.nabble.com> napisał:

> Hi
>
> Ah well spotted.
> I think we should for loop via Map.Entry (or (k,v) via lambda) which I
> think will be in the correct order.
>
> You are welcome to log a JIRA. And also work on unit test and patch.
> http://camel.apache.org/contributing
>
> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz
> <[hidden email] <http:///user/SendEmail.jtp?type=node=5815878=0>>
> wrote:
>
> > Hi all,
> >
> > I recently noticed that there is wrong entry order in file using
> > FileIdempotentRepository implementation.
> > The effect is that instead of having order like:
> >
> > file1.txt.20171123
> > file2.txt.20171123
> > file1.txt.20171124
> > file3.txt.20171125
> > file2.txt.20171126
> >
> > we have:
> >
> > file1.txt.20171123
> > file1.txt.20171124
> > file2.txt.20171123
> > file2.txt.20171126
> > file3.txt.20171125
> >
> > where date extension represents order in which particular file was
> consumed
> > by the idempotent file consumer.
> > As a consequence instead of initializing memory cache with newest
> values,
> > it is initialized (probably) based on hash function from truncStore
> method
> > and we consume same file more than once:
> >
> > protected void trunkStore() {
> > LOG.info("Trunking idempotent filestore: {}", fileStore);
> > FileOutputStream fos = null;
> > try {
> > fos = new FileOutputStream(fileStore);
> > for (String key : *cache.keySet()*) {
> > fos.write(key.getBytes());
> > fos.write(STORE_DELIMITER.getBytes());
> > }
> > } catch (IOException e) {
> > throw ObjectHelper.wrapRuntimeCamelException(e);
> > } finally {
> > IOHelper.close(fos, "Trunking file idempotent repository",
> LOG);
> > }
> > }
> >
> > LRUCache:
> >
> > @Override
> > public Set keySet() {
> > return map.keySet();
> > }
> >
> > where previously it was:
> >
> > @Override
> > public Set keySet() {
> > return map.*ascendingKeySet*();
> > }
> >
> > Regards
> > Kris
> > --
> >
> > Pozdrawiam
> >
> > Krzysztof Hołdanowicz
>
>
>
> --
> Claus Ibsen
> -
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=465428=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-- 

Pozdrawiam

Krzysztof Hołdanowicz

Re: File idempotent store problem

2017-11-28 Thread Claus Ibsen

Hi

Ah well spotted.
I think we should for loop via Map.Entry (or (k,v) via lambda) which I
think will be in the correct order.

You are welcome to log a JIRA. And also work on unit test and patch.
http://camel.apache.org/contributing

On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz
 wrote:
> Hi all,
>
> I recently noticed that there is wrong entry order in file using
> FileIdempotentRepository implementation.
> The effect is that instead of having order like:
>
> file1.txt.20171123
> file2.txt.20171123
> file1.txt.20171124
> file3.txt.20171125
> file2.txt.20171126
>
> we have:
>
> file1.txt.20171123
> file1.txt.20171124
> file2.txt.20171123
> file2.txt.20171126
> file3.txt.20171125
>
> where date extension represents order in which particular file was consumed
> by the idempotent file consumer.
> As a consequence instead of initializing memory cache with newest values,
> it is initialized (probably) based on hash function from truncStore method
> and we consume same file more than once:
>
> protected void trunkStore() {
> LOG.info("Trunking idempotent filestore: {}", fileStore);
> FileOutputStream fos = null;
> try {
> fos = new FileOutputStream(fileStore);
> for (String key : *cache.keySet()*) {
> fos.write(key.getBytes());
> fos.write(STORE_DELIMITER.getBytes());
> }
> } catch (IOException e) {
> throw ObjectHelper.wrapRuntimeCamelException(e);
> } finally {
> IOHelper.close(fos, "Trunking file idempotent repository", LOG);
> }
> }
>
> LRUCache:
>
> @Override
> public Set keySet() {
> return map.keySet();
> }
>
> where previously it was:
>
> @Override
> public Set keySet() {
> return map.*ascendingKeySet*();
> }
>
> Regards
> Kris
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

File idempotent store problem

2017-11-27 Thread Krzysztof Hołdanowicz

Hi all,

I recently noticed that there is wrong entry order in file using
FileIdempotentRepository implementation.
The effect is that instead of having order like:

file1.txt.20171123
file2.txt.20171123
file1.txt.20171124
file3.txt.20171125
file2.txt.20171126

we have:

file1.txt.20171123
file1.txt.20171124
file2.txt.20171123
file2.txt.20171126
file3.txt.20171125

where date extension represents order in which particular file was consumed
by the idempotent file consumer.
As a consequence instead of initializing memory cache with newest values,
it is initialized (probably) based on hash function from truncStore method
and we consume same file more than once:

protected void trunkStore() {
LOG.info("Trunking idempotent filestore: {}", fileStore);
FileOutputStream fos = null;
try {
fos = new FileOutputStream(fileStore);
for (String key : *cache.keySet()*) {
fos.write(key.getBytes());
fos.write(STORE_DELIMITER.getBytes());
}
} catch (IOException e) {
throw ObjectHelper.wrapRuntimeCamelException(e);
} finally {
IOHelper.close(fos, "Trunking file idempotent repository", LOG);
}
}

LRUCache:

@Override
public Set keySet() {
return map.keySet();
}

where previously it was:

@Override
public Set keySet() {
return map.*ascendingKeySet*();
}

Regards
Kris
-- 

Pozdrawiam

Krzysztof Hołdanowicz

Re: File idempotent store problem

Re: File idempotent store problem

Re: File idempotent store problem

Re: File idempotent store problem

Re: File idempotent store problem

File idempotent store problem

6 matches

Site Navigation

Mail list logo

Footer information