Hi, I fixed the enum thing. It was a mismatch between int and long. D'oh! > Are there conventions for mapping more complex types such as Date for instance? No (at least I don't know any), so we should go as we need it.
> TTL Implemented. > Transactions: Ah, that's interesting. So it should be possible to link this with the JPA/JTA transaction APIs? Redis transactions work a bit different (was not aware of that, as I told you about transactions). Redis transactions are more batch than transactions. Redis transactions work very nice when writing, but there's a reading issue. Reads are queued until the transaction is submitted. So the read will come back with a result only if the transaction is confirmed with `EXEC`. I'm pretty sure, this can be a deal-breaker for using transactions in the JTA way. I submitted a PR https://github.com/hibernate/hibernate-ogm/pull/560 and started to work on the docs. Let's start a discussion based on the PR :-) Best regards, viele Grüße, Mark Gunnar Morling <gun...@hibernate.org> schrieb am Mi., 22. Juli 2015 um 10:41 Uhr: > 2015-07-21 14:14 GMT+02:00 Mark Paluch <mpal...@paluch.biz>: > >> Hi Gunnar, >> >> thanks for your helpful feedback. I will come up with a PR as soon as we >> agree on the strategies issue - well, perhaps in a couple days. I can and >> will support you and I can help you with support/maintenance for the Redis >> module in the future. >> > > That's great to hear. We are no Redis experts, so the help of one is > needed to make this fly :) > > >> I will have to ask you in return on some specific issues, such as why >> enum-ordinals cannot be mapped back. >> > > Sure, absolutely. I will take a look at the enum thingy. > > >> > * I saw you create a hash per entity type, using the ids as keys within >> this hash and a JSON document modelling the properties of a given entity >> instance. >> > >> > Have you considered to store JSON documents on the "top-level" instead? >> This would require to encode the table name into the key as well (similar >> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is >> that I am wondering whether it may be a source of contention and/or whether >> it may pose scaling issues if all entities of one type are stored within a >> single hash. >> >> Yes, I was thinking about that issue. I ended up with the hash per entity >> type in the first place. This is not a final thing, but rather a draft. >> Pulling the JSON strategy to the global level is a good idea and key >> patterns like "Order:123" are a recommended way. The top-level approach >> tackles also another issue: Redis can be operated as a cluster in terms of >> sharding. The sharding is key-based. A top-level key encourages a nice >> distribution amongst the cluster. Cluster support is not yet included in my >> code. I wanted to check with you guys first, whether a Redis module could >> be included in OGM. >> > > Yes, clustering was part of my concern with the original approach. The > top-level approach would help with that. > >> >> > Using JSON for the actual entity contents seems reasonable and a common >> approach in the Redis world from what I've read, so +1 for doing this from >> me. How would the "more native" approach look like you describe below? >> Would we have to serialise everything ourselves into String/byte[]? If so, >> I am not sure whether that's of much usefulness actually and I'd be >> inclined to go the JSON route exclusively. As you say, it also works nicely >> for hierarchical structures. >> >> My first idea on the more native approach was storing all basic data >> types as key-value pairs. I noticed, that embedded properties are >> transported with dot-notation, which comes handy. Converting the properties >> to String is also a common Redis style and interoperability between >> different systems/languages is encouraged hereby. >> > > Are there conventions for mapping more complex types such as Date for > instance? You'd need custom GridTypes to map everything into String (we > have String-mapping implementations for some types such as Date and Long, > but not for all of them). > > >> I'm just not sure, how to deal with external and embedded associations. >> Simulating associations by distributing the lists to Redis lists with >> referencing other hashes looks like an overkill to Redis. A lot of >> meta-data would have to be held. Feasible Yes, sensible No. The ROM >> framework (https://github.com/josiahcarlson/rom/) goes for this >> approach. >> >> So if omitting associations in the more native approach is not a >> deal-breaker, I'd vote to go this way as well. >> > > Not supporting associations at all in that mode seems like a heavy > limitation to me. I am not sure what you mean by "lot of meta-data would > have to be held"; Where, in the driver, or in the dialect? OGM engine > should give you all you need, I suggest to check out the Infinispan/Ehcache > backends and follow the same approach. > > >> I'm also fine if we start with just the JSON approach. >> > > +1 Let's start from there and let things evolve. > >> >> > * The id is used as key and also part of the JSON value. Could it be >> removed from the latter? >> Yes, this is due to the draft state >> >> > * Do you have any ideas how Redis-specific structures could be exposed >> which do not map exactly to entities, e.g. Lists stored directly via some >> key? >> >> Yes. Redis-backed Collections, Maps, Locks, Queues, Caching, and many >> more are possible. I'm not sure, whether you want to have that >> features because they do not look like they fit the Hibernate OGM concept. >> There are implementations (Redisson: https://github.com/mrniko/redisson, >> Spring Data Redis: >> https://github.com/spring-projects/spring-data-redis/tree/master/src/main/java/org/springframework/data/redis/support/collections) >> that cover the Redis-backed Collections etc. >> > > People will use Redis for the powers of its specific features, so I think > they should be accessible in one way or the other. Not everything can be > mapped to/exposed through JPA in a sensible way. I guess one needs to look > at it on a case-by-case basis, maybe it makes sense to somehow expose the > native API for certain advanced cases. > >> >> > * Our option system could be used to expose specific settings such as >> TTL for given entities >> Great! Redis supports TTL's, I was not aware of that OGM feature >> > > OGM does not directly support this, but it provides an extensible option > system which dialects can leverage to let the user configure specific > settings and apply these when talking to the datastore. E.g. one could > imagine something like this: > > @TTL(value=7, unit=TimeUnit.DAY) > @Entity > class LogRecord { ... } > > @TTL would be a Redis-backend specific setting based on said option > system, and the Redis backend would apply this setting when writing tuples > of the LogRecord entity. You can find examples with > @ReadPreference/@WriteConcern in the MongoDB backend. Of course that's not > required for a first version, but would make for a nice addition in later > iterations. > > >> >> > * Is there any form of transaction support in Redis we could leverage? >> Yes. Redis supports transactions with the MULTI/EXEC/DISCARD commands. >> Transactions require a connection affinity and imply the use of multiple >> connections. That's nothing bad per se, just a thing to be aware of. >> > > Ah, that's interesting. So it should be possible to link this with the > JPA/JTA transaction APIs? > > >> >> Thanks and best regards, Mark >> >> >> Gunnar Morling <gun...@hibernate.org> schrieb am Di., 21. Juli 2015 um >> 13:03 Uhr: >> >>> Hi Mark, >>> >>> Many thanks for giving this a try; Your work looks impressive, it's very >>> advanced for an initial dialect contribution, covering many advanced cases >>> already. So kudos for that! >>> >>> Please find some remarks and questions below. >>> >>> Cheers, >>> >>> --Gunnar >>> >>> >>> 2015-07-17 17:16 GMT+02:00 Mark Paluch <mpal...@paluch.biz>: >>> >>>> Hi there, >>>> >>>> I created a Redis module for Hibernate OGM, based on OGM 4.2. Redis is a >>>> Key-Value store supporting Lists, Hashes (Maps), Sorted Sets, ... >>>> >>>> I took the Infinispan/CouchDB approach storing entities and >>>> associations as >>>> JSON documents within Redis Hashes. A Map<Object, Map<Object, Object>> >>>> corresponds with a Redis Hash. >>>> >>>> The entity table maps to the Hash Id, the entity Id to the key of the >>>> hash >>>> and the set of columns is enclosed within a JSON (currently). Supported >>>> features are: >>>> >>>> * Composite Id's >>>> * Embedded associations >>>> * External associations >>>> >>> >>> * I saw you create a hash per entity type, using the ids as keys within >>> this hash and a JSON document modelling the properties of a given entity >>> instance. >>> >>> Have you considered to store JSON documents on the "top-level" instead? >>> This would require to encode the table name into the key as well (similar >>> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is >>> that I am wondering whether it may be a source of contention and/or whether >>> it may pose scaling issues if all entities of one type are stored within a >>> single hash. >>> >>> Using JSON for the actual entity contents seems reasonable and a common >>> approach in the Redis world from what I've read, so +1 for doing this from >>> me. How would the "more native" approach look like you describe below? >>> Would we have to serialise everything ourselves into String/byte[]? If so, >>> I am not sure whether that's of much usefulness actually and I'd be >>> inclined to go the JSON route exclusively. As you say, it also works nicely >>> for hierarchical structures. >>> >>> * The id is used as key and also part of the JSON value. Could it be >>> removed from the latter? >>> >>> * Do you have any ideas how Redis-specific structures could be exposed >>> which do not map exactly to entities, e.g. Lists stored directly via some >>> key? >>> >>> * Our option system could be used to expose specific settings such as >>> TTL for given entities >>> >>> * Is there any form of transaction support in Redis we could leverage? >>> >>> and I guess many more, that I'm currently not aware of :-) >>>> >>>> There are various ways (JSON, Java-serialized simulating Documents) how >>>> to >>>> store data within Redis. >>>> Another approach could be storing Tuples within Redis more native, since >>>> they are in the end Maps, somehow. However, mapping tuples to Redis fit >>>> a >>>> native storage but raises questions about Id's, limited data type >>>> support >>>> (there is no equivalent for nested lists/maps). So the more native >>>> approach >>>> would require distributing data amongst various data types, depending on >>>> the supported features, and would require more IO's. >>>> >>>> My impl can be used to interoperate with other clients from the C, Ruby, >>>> Python, Node.js, ... worlds. The JSON is clean and does not look >>>> awkward on >>>> the first look. >>>> >>>> The current code can be found below >>>> https://github.com/mp911de/hibernate-ogm/tree/redis. Most TCK tests >>>> pass, >>>> only some 15 tests fail. Mostly due to enum and JTA support. Couldn't >>>> figure out, how to fix it, so need your help to figure it out. >>>> >>> >>> I'll take a closer look at your change asap. Could you create a pull >>> request from your branch against the upstream repo? Then we can have a >>> discussion right there. >>> >>> Hope, I made you hungry for the Redis OGM module. Let's start a >>>> discussion >>>> based on my proposal for inclusion and let's see, where we end up. >>>> >>> >>> Yep, it looks very promising and I think it'd be great to have support >>> for Redis in OGM :) We'll need your continued help though to make it happen >>> and evolve it in the future as Redis itself and OGM core advance. >>> >>> Best regards, Mark >>>> _______________________________________________ >>>> hibernate-dev mailing list >>>> hibernate-dev@lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>>> >>> >>> _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
