We cannot guarantee that field_1 is always address. In Form 1, field_1
might be address while in another form it might be string or number
whatever. Thinking about designing the storage for Google Forms and it's
data entries.
Re "you could force each field to be a String and do the transformation at
a client level."
Forcing means serialize all data into a string right? In the example JSON
mentioned in previous email, it will transformed to something like
{ field_1: "\{country: \"US\", province: \"CA\", city: \"New York\",
address: \"Street Address\"\}" }
Then not able to do the aggregation.
On Fri, Sep 19, 2014 at 5:50 PM, David Pilato <[email protected]> wrote:
> I don't get it.
>
> If field_1.country is a String why you can not aggregate on it?
>
> --
> *David Pilato* | Technical Advocate | *elasticsearch.com
> <http://elasticsearch.com>*
> [email protected]
> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr
> <https://twitter.com/elasticsearchfr> | @scrutmydocs
> <http://twitter.com/scrutmydocs>
> <https://twitter.com/scrutmydocs>
>
>
>
> Le 19 septembre 2014 à 08:27:19, Michael Chen ([email protected]) a
> écrit:
>
> Thanks David. Based on the system behavior, having all type as string is
> fine for queries. But for the aggregation level it might be trouble. For
> example a type of address is a complex JSON object:
>
> { field_1: { country: "US", province: "CA", city: "New York", address:
> "Street Address"} }
>
> If we transform this type into any form of string, and trying to aggregate
> based on country/state, it will be VERY hard even not possible.
>
> On Fri, Sep 19, 2014 at 2:15 PM, David Pilato <[email protected]> wrote:
>
>> You could have one type per form although the cluster state will be
>> very big.
>> But you should test that option.
>>
>> Or if you don't really search for numbers as numbers (I mean with Range
>> queries/filters), you could force each field to be a String and do the
>> transformation at a client level.
>>
>> My 2 cents
>>
>>
>> --
>> * David Pilato* | Technical Advocate | *elasticsearch.com
>> <http://elasticsearch.com>*
>> [email protected]
>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr
>> <https://twitter.com/elasticsearchfr> | @scrutmydocs
>> <http://twitter.com/scrutmydocs>
>> <https://twitter.com/scrutmydocs>
>>
>>
>>
>> Le 19 septembre 2014 à 05:31:38, Michael Chen ([email protected]) a
>> écrit:
>>
>> Hi,
>>
>> We have a system very much like Google Forms, which allow users to design
>> their own forms with various fields (single line text, paragraph, number,
>> address etc, you can imagine that.) Without any doubt, it's running on top
>> of MongoDB. Now it has 120K forms with nearly 10 million entries.
>>
>> Recently we found a performance bottleneck on the query. After we have
>> done every possible performance tuning on mongoDB side, we decide to index
>> the form entries into elastic search. And there is a trouble:
>>
>> Given there is a Form A, has field_1 as string type, field_2 as number,
>> the data entry might look like: { field_1: "hello", field_2: 100}
>>
>> Form B could be field_1 as number, field_2 as number, the date entry will
>> look like { field_1: 100, field_2: "hello form"}
>>
>> We have successfully create an index "entries" in ES, and can index the
>> first entry successfully. But the second one failed for an obvious reason:
>> type mismatch.
>>
>> I am not sure how to deal with this problem. I definitely don't want to
>> create 120K indices for every single form. And I am not sure it's doable to
>> write custom transform script to change the index type identical across all
>> entries.
>>
>> Any suggestion? Much appreciate any response.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/4a6d47d7-ae0e-44f5-bd3a-756ea94e3899%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/4a6d47d7-ae0e-44f5-bd3a-756ea94e3899%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/etPan.541bc9fd.66334873.18d1%40MacBook-Air-de-David.local
>> <https://groups.google.com/d/msgid/elasticsearch/etPan.541bc9fd.66334873.18d1%40MacBook-Air-de-David.local?utm_medium=email&utm_source=footer>.
>>
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Michael Chen
> --------------------------------
> Blog: http://michael.nona.name
> GTalk/Twitter/Facebook/Yahoo/Skype: mechiland
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAAzF%2BWj0jFb642x_hgko%2Bs9mfM3MCNqeS5Jr7TxtWq5XXAPvXg%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAAzF%2BWj0jFb642x_hgko%2Bs9mfM3MCNqeS5Jr7TxtWq5XXAPvXg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/etPan.541bfc4e.1f16e9e8.18d1%40MacBook-Air-de-David.local
> <https://groups.google.com/d/msgid/elasticsearch/etPan.541bfc4e.1f16e9e8.18d1%40MacBook-Air-de-David.local?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
Michael Chen
--------------------------------
Blog: http://michael.nona.name
GTalk/Twitter/Facebook/Yahoo/Skype: mechiland
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAAzF%2BWjYweFv5RtR5diezwE4pfuCEd1%2BGKf%3DpH-JfUMmSjeKrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
