your document-as-byte-array is wrong--it only handles ascii, and very
inefficiently too.
Just say (.getBytes document-as-string "UTF-8") to get a utf-8 encoding of
the string as a byte array.
On Thu, Jul 9, 2015 at 10:41 AM, <gingersafflo...@gmail.com> wrote:
>
> > As for the escaped quotes, you may be using pr or prn to print, or maybe
> you are
> > using pr-str to produce the string representation. I can't be sure.
>
> At the moment I create the string like this:
>
> document-as-string (str "{\"transaction-id\" : \"" transaction-id
> "\", \"message\" : \"" message "\"}")
> document-as-byte-array (bytes (byte-array (map (comp byte int)
> document-as-string)))
>
> The crazy thing is that this sometimes works, but other times the quote
> marks appear in Redis as escaped quote marks. As near as I can tell, the
> important factor is the length of the string. A short string is likely to
> have its quote marks escaped. A long string does not have its quote marks
> escaped.
>
> My co-worker is working on the Java app. I am working on the Clojure app.
> We can both adjust out apps freely, just so long as we can get data to and
> from each other, in a manner that allows us to eventually cast the data to
> and from JSON.
>
> Any suggestions are welcome.
>
>
>
>
>
>
> On Wednesday, July 8, 2015 at 8:58:53 PM UTC-4, Francis Avila wrote:
>>
>> Who is saving these strings, and who is reading them? Do you have
>> complete control over both apps, or does one of them need to be aligned
>> with the other?
>>
>> If the Java app is the baseline, you need to know the exact details of
>> the format of the data it saves. Just knowing "it's JSON" is not enough,
>> because there may be other data types (e.g. dates) that don't have native
>> JSON representations. (The go-to JSON de/encoder for clojure is cheshire:
>> https://github.com/dakrone/cheshire )
>>
>> I took a quick look at Jedis and the easy, default way of using it is
>> with strings. It will encode strings to UTF-8 before sending to Redis and
>> decode from UTF-8 on read. You can set raw byte arrays too (which will not
>> be altered in any way before sending), but it's not clear to me how it can
>> read out raw byte arrays. (I'm sure there's a way, but it's not immediately
>> obvious.)
>>
>> As for the escaped quotes, you may be using pr or prn to print, or maybe
>> you are using pr-str to produce the string representation. I can't be sure.
>>
>> On Wednesday, July 8, 2015 at 5:31:27 PM UTC-5, gingers...@gmail.com
>> wrote:
>>>
>>> And I have another stupid question. Using the above code, I am sometimes
>>> getting strings in Redis that have escaped quotation marks, like this:
>>>
>>> " \"transaction-id\" : \" 1ec47c2e-21ee-427c-841c-80a0f89f55d7 \"
>>> \"debrief\" : \" Susan Hilly at Citi called to get a quotation for
>>> discounted weekly car rental for approximately 3 cars per week, or 150
>>> rentals annually. \" "
>>>
>>> Why is that happening?
>>>
>>>
>>>
>>> On Wednesday, July 8, 2015 at 5:38:20 PM UTC-4, gingers...@gmail.com
>>> wrote:
>>>>
>>>> Francis Avila,
>>>>
>>>> Thank you for your response. The Java app is using Jedis and the
>>>> Clojure app is using Carmine. I'm wondering if you can suggest what you
>>>> think would be the easiest way to allow these 2 apps to understand each
>>>> other's strings?
>>>>
>>>> You were correct about how unsafe the above code was. I tested it for
>>>> less than 15 minutes and ran into the fact that a \n newline made a mess of
>>>> everything.
>>>>
>>>>
>>>>
>>>> On Wednesday, July 8, 2015 at 5:15:35 PM UTC-4, Francis Avila wrote:
>>>>>
>>>>> You are running into Carmine's automatic nippy serialization.
>>>>> https://github.com/ptaoussanis/carmine#serialization
>>>>>
>>>>> Redis only stores byte arrays (what it calls "strings"). Carmine uses
>>>>> the nippy library (the meaning of "NPY" in your byte stream) to represent
>>>>> rich types compactly as bytes. https://github.com/ptaoussanis/nippy
>>>>>
>>>>> If you give Carmine a byte array to store, it will store it directly
>>>>> without nippy-encoding it. E.g. (.getBytes "{}" "UTF-8")
>>>>>
>>>>> BTW your document-as-string example is extremely unsafe: how will you
>>>>> reliably read this message out again? e.g. what if the 'debrief' string
>>>>> contains a single quote? Use a proper serialization format.
>>>>>
>>>>> So the key is to have both your Clojure and Java app store *bytes* in
>>>>> Redis using the same serialization. You can store anything you want
>>>>> (nippy,
>>>>> utf-8-encoded json, fressian, bson, utf-8 xml, utf-16 java strings,
>>>>> whatever) as long as it's bytes and it's read and written the same way in
>>>>> all your apps.
>>>>>
>>>>> The Redis library your Java app is using may have its own automatic
>>>>> de/serialization, too. You need to find out what it's doing and either
>>>>> work
>>>>> with this or turn it off, just like with Carmine.
>>>>>
>>>>> Nippy unfortunately does not have a Java API out of the box:
>>>>> https://github.com/ptaoussanis/nippy/issues/66
>>>>>
>>>>>
>>>>> On Wednesday, July 8, 2015 at 3:35:49 PM UTC-5, gingers...@gmail.com
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> I am not sure if this is a Clojure question or a Java question. I
>>>>>> don't know Java, so I could use whatever help folks can offer.
>>>>>>
>>>>>> "primitive string" here means what I can write when I am at the
>>>>>> terminal.
>>>>>>
>>>>>> We have 2 apps, one in Clojure, one in Java. They talk to each other
>>>>>> via Redis. I know the Java app can read stuff out of Redis, using our
>>>>>> "transaction-id", if I use the terminal and open up "redis-clj" and
>>>>>> write a
>>>>>> string directly from the terminal. But I have this Clojure code, which
>>>>>> depends on Peter Taoussanis's Carmine library:
>>>>>> (defn worker [document]
>>>>>> {:pre [(string? (:transaction-id document))]}
>>>>>> (let [transaction-id (:transaction-id document)
>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>> redis-connection {:pool {} :spec {:host "127.0.0.1" :port
>>>>>> 6379 }}]
>>>>>> (timbre/log :trace " message we will send to NLP "
>>>>>> document-as-string)
>>>>>> (carmine/wcar redis-connection (carmine/set transaction-id
>>>>>> document))
>>>>>> (loop [document-in-redis (carmine/wcar redis-connection
>>>>>> (carmine/get transaction-id))]
>>>>>>
>>>>>> (if-not (.contains (first document-in-redis) "processed")
>>>>>> (recur (carmine/wcar redis-connection (carmine/get
>>>>>> transaction-id)))
>>>>>> (do
>>>>>> (carmine/wcar redis-connection (carmine/del transaction-id))
>>>>>> document-in-redis)))))
>>>>>>
>>>>>> This line in particular, I have tried doing this several ways:
>>>>>>
>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>
>>>>>> In Redis, I expect to see:
>>>>>>
>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358', 'debrief'
>>>>>> : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig responded to our
>>>>>> proposal and said his company is read to move forward. The rate of $400
>>>>>> per
>>>>>> ton of shredded paper was acceptable to them, and they shred about 2 tons
>>>>>> of documents every month. $96,000 in potential revenue annually. I will
>>>>>> meet with him tomorrow and we will sign the contract.'}
>>>>>>
>>>>>> But if I then launch redis-cli, I see:
>>>>>>
>>>>>>
>>>>>> 127.0.0.1:6379> keys *
>>>>>> 1) "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>
>>>>>> 127.0.0.1:6379> get "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>> "\x00>NPY\b\x00\x00\x01\xfc\xf1\xfe\x1b\x00\x00\x00\nj\nip-addressi\x0e165.254.84.238j\x05tokeni$46b87d64-cff3-4b8b-895c-e089ac59544dj\x0bapi-versioni\x02v1j\x0etransaction-idi$42e574e7-3b80-424a-b9ff-01072f1e0358j\adebrief\r\x00\x00\x01YSmeek
>>>>>> Hallie of Withers, Smeg, Harrington and Norvig responded to our proposal
>>>>>> and said his company is rea-\x00\xf1\x06move forward. The
>>>>>> raty\x00\xf0\x0c$400 per ton of shredded pa\x16\x00\xf1\bwas acceptable
>>>>>> to
>>>>>> them,q\x00Bthey0\x00\x80 about 2E\x00\x10sF\x00\xf1Ldocuments every
>>>>>> month.
>>>>>> $96,000 in potential revenue annually. I will meet with him
>>>>>> tomorrow{\x00\"we#\x00@sign\x92\x00\xa0 contract."
>>>>>>
>>>>>>
>>>>>> I don't know what all of those extra characters are. The Java app is
>>>>>> not picking this item up, so I assume the Java app is not seeing this as
>>>>>> a
>>>>>> string. I expected this to look the same as if I had written this at the
>>>>>> terminal:
>>>>>>
>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358', 'debrief'
>>>>>> : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig responded to our
>>>>>> proposal and said his company is read to move forward. The rate of $400
>>>>>> per
>>>>>> ton of shredded paper was acceptable to them, and they shred about 2 tons
>>>>>> of documents every month. $96,000 in potential revenue annually. I will
>>>>>> meet with him tomorrow and we will sign the contract.'}
>>>>>>
>>>>>> I assume it is easy to get a string into a format that can be
>>>>>> understood by both a Clojure app and a Java app. I don't care what format
>>>>>> that is, but it needs to be consistent.
>>>>>>
>>>>>> Can anyone make suggestions about what I can do to make sure the
>>>>>> Clojure app and the Java app both write to Redis using a format that the
>>>>>> other will understand? In particular, both apps need to see the
>>>>>> "'transaction-id".
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/9swfBxAbT90/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
