hhafner
Tue, 22 Sep 2009 12:06:06 -0700
Craig, thanks for your suggestions. I will try to code the largest category with the highest number. The method with my own dummies worked well all in all, except for the last category. To avoid multiple positive values, I imputed one category after the other (starting with the largest). If a observation had a 1 for one dummy, I made no further imputations for the other dummies for this observation (RESTRICT-Statement in IVEware; further imputations only if all previous dummies are zero).
Hans-Peter Craig Newgard <newga...@ohsu.edu> 22.09.2009 05:21 An "hhaf...@statistik-hessen.de" <hhaf...@statistik-hessen.de>, "impute@lists.utsouthwestern.edu" <impute@lists.utsouthwestern.edu> Kopie Thema RE: [Impute] IVEware: Imputation of categorical variables with many categories Hans-Peter, Not sure if you've found a response to your question below yet, but I have been through similar scenarios with IVEware before. My suggestion would be to keep the primary (polytomous) categorical terms in the MI code, as this allows IVEware to create dummies, while still recognizing that they are mutually exclusive categories. If you create your own dummies, you run the risk of imputing positive values for multiple dummies on the same observation. A few other things you could try to improve efficiency include: making sure the largest category (reference) is coded with the highest number in the term (default reference in IVEware) and examine small categories within the term and consider collapsing categories if appropriate. I've found that many IVEware MI models routinely require 24+ hours to run with such terms included. If these suggestions still fail to increase the MI efficiency, you could also consider running parallel chains of MI, using separate MI models for subjects within each category of the original term (this is commonly used for interaction terms and works best if each category has an adequate number of observations and minimal missing data). Craig ________________________________________ From: impute-boun...@lists.utsouthwestern.edu [impute-boun...@lists.utsouthwestern.edu] On Behalf Of hhaf...@statistik-hessen.de [hhaf...@statistik-hessen.de] Sent: Monday, September 21, 2009 1:45 AM To: impute@lists.utsouthwestern.edu Subject: [Impute] IVEware: Imputation of categorical variables with many categories Hello, I have a dataset with about 50.000 records and 30 variables. Among the variables are 2 categorical with many categories: Federal state (16 categories) and branch of economic activity (80 - 100 categories). Since I want to produce a synthetic dataset, I double the dataset by replacing all values of one variable with missings. Now to my problem with IVEware: If I want to impute for example the federal state, after 5-6 hours still the first iteration is running, so it takes too long. My second attempt: I compute dummies for the 16 federal states. At first I impute the state having the most units, then the one with the second most units and so on. All in all this works well, but for the last state there are only 20-30 units remaining (original data: 358 units). I tried to swap the order of the smallest and the second smallest state: This didn't solve the problem. Now the second smallest state has by far too few units in the synthetic dataset. Does anyone have any further suggestions how one can handle categorical variables with many values in IVEware? Kind regards Hans-Peter Hafner STATISTIK HESSEN ----------- Hessisches Statistisches Landesamt Rheinstraße 35/37 65175 Wiesbaden Internet: http://www.statistik-hessen.de Telefon: 0611 3802-815 Telefax: 0611 3802-890 E-Mail: hhaf...@statistik-hessen.de
_______________________________________________ Impute mailing list Impute@lists.utsouthwestern.edu http://lists.utsouthwestern.edu/mailman/listinfo/impute