It is possible to do what you want, but it is not simple to write.
Scikit-learn could definitely benefit from an example showing this sort of
thing, or from a better API to help the user do it, as suggested at
https://github.com/scikit-learn/scikit-learn/issues/2034. There you will
find a lengthy comment where I give an example very similar to yours (but
with fields as attributes rather than dict keys).
On 21 June 2014 09:10, Abijith Kp <[email protected]> wrote:
> What would be the advantage for using a shared vocabulary for Count
> Vectorizer??
>
> When I read about FeatureUnion, what I understood was that, the given list
> of transformers would process the given data set completely. Could we use
> it to selectively process different features?? Or is my understanding of
> the concept not clear??
>
> Regards,
> Abijith
>
>
> On Sat, Jun 21, 2014 at 7:12 PM, Andy <[email protected]> wrote:
>
>> Yes, you can use CountVectorizer.
>> Do you want the different features to share the same vocabulary?
>> To use the Count Vectorizer, you probably have to either get all the
>> values (for a shared vocabulary)
>> or learn one CountVectorizer per key (you could use FeatureUnion for
>> that).
>>
>> So there is a little bit of code to write to handle the fact that you
>> have multiple text fields.
>>
>> Hth,
>> Andy
>>
>>
>>
>> On 06/21/2014 03:35 PM, Abijith Kp wrote:
>>
>> Hi,
>>
>>
>> Initially, one of my feature list looks like: {"a":"3", "b":"random1",
>> "c":"", "d":"random2 text"}.
>> The random text contains names of people, email ids, some description,
>> numbers and goes on.
>>
>> When I used DictVectorizer, I could not get an accurate clustering.
>>
>> I wanted know if I could get any method similar to DictVectorizer,
>> which could process a dictionary of string features, correctly.
>>
>> Regards,
>> Abijith
>>
>>
>> On Sat, Jun 21, 2014 at 6:51 PM, Andy <[email protected]> wrote:
>>
>>> Hi Abijith.
>>>
>>> It depends on how you want to interpret the strings.
>>> If they are texts and you want to interpret them based on their content,
>>> Brians suggestion is the right one.
>>> If you want to consider each possible string as a distinct feature, the
>>> OneHotEncoder would be the right choice.
>>>
>>> Could you give an example of what the strings and the semantics of the
>>> strings are?
>>>
>>> Andy
>>>
>>>
>>>
>>>
>>> On 06/20/2014 06:05 PM, Abijith Kp wrote:
>>>
>>> Can anyone help me with the problem of dealing with feature which
>>> are both strings of varying length(say from 0 to 100-150 characters) and
>>> numbers?
>>>
>>> What will be the most widely used techniques in such kind of
>>> situations? And can it be solved using only scikit-learn?
>>>
>>> PS: Initially I have to convert a json file to a feature's list, and
>>> then use it.
>>>
>>> Any help is appreciated.
>>>
>>> Regards,
>>> Abijith
>>>
>>> --
>>> Abijith KP
>>> github.com/abijith-kp
>>> kpabijith.wordpress.com
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
>>> Find What Matters Most in Your Big Data with HPCC Systems
>>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>>> Leverages Graph Analysis for Fast Processing & Easy Data
>>> Explorationhttp://p.sf.net/sfu/hpccsystems
>>>
>>>
>>>
>>> _______________________________________________
>>> Scikit-learn-general mailing
>>> [email protected]https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
>>> Find What Matters Most in Your Big Data with HPCC Systems
>>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>>> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
>>> http://p.sf.net/sfu/hpccsystems
>>> _______________________________________________
>>> Scikit-learn-general mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>>
>>
>>
>> --
>> Abijith KP
>> github.com/abijith-kp
>> kpabijith.wordpress.com
>>
>>
>> ------------------------------------------------------------------------------
>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
>> Find What Matters Most in Your Big Data with HPCC Systems
>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>> Leverages Graph Analysis for Fast Processing & Easy Data
>> Explorationhttp://p.sf.net/sfu/hpccsystems
>>
>>
>>
>> _______________________________________________
>> Scikit-learn-general mailing
>> [email protected]https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
>> Find What Matters Most in Your Big Data with HPCC Systems
>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
>> http://p.sf.net/sfu/hpccsystems
>> _______________________________________________
>> Scikit-learn-general mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> --
> Abijith KP
> github.com/abijith-kp
> kpabijith.wordpress.com
>
>
> ------------------------------------------------------------------------------
> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
> Find What Matters Most in Your Big Data with HPCC Systems
> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
> http://p.sf.net/sfu/hpccsystems
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
