Re: [Ifeffit] Athena: Data cannot be imported by using plugin PFBL12C
Dear Matt, Thanks for the reply and script. I will try to use Larch, instead of athena with Ifeffit, process the data by a proper method, and conduct a study for dataformat. Best regards, Yuji Hiraoka >I have to admit I find this all sort of disappointing. > >First, the PFBL12C is a really poor way to represent EXAFS data. It uses >a hard-to-parse and hard-to-interpret format and has many unlabeled >columns. It appears to include this really odd notion that recording >"angle" is somehow better than recording energy, and saves a >ridiculuously over-sampled spectrum that needs smoothing because of the >high noise per recorded point. It is, in short, a really poor way to >represent EXAFS data. We've seen it for many years, we've written papers >and published supported code for better representations of EXAFS data. > Still, we get questions about using data recorded in such files. Data >files this complex need a way to convert them into something useful. > >Second, the limitations you are seeing are due to using the outdated, >no-longer-supported (for several years now actually) Ifeffit library. >These data can be read and processed without problem with Larch, the >replacement for Ifeffit. > >Stop using Ifeffit. Use Larch instead. Really. > >An example script to use larch for your `fail.dat` file is listed below. >It is somewhat complicated because the raw data file is such a poor >representation of EXAFS data. > >--Matt > ># larch script to read, smooth and process data in the horrible PFBL12C >format ># read raw data, extract angle, i0, sum of fluorescence >datafile = 'fail.dat' >rawdat = read_ascii(datafile) > >def degrees2eV(angle, dspace=1.63747): >from scipy.constants import hbar, c, e >hbarc = 1.e10*hbar*c/e >return (pi*hbarc/dspace) / sin(angle*pi/180.0) >#enddef > >rawdat.angle = rawdat.data[0, :] # angle in degrees. yikes! >rawdat.energy = degrees2eV(rawdat.angle) >rawdat.i0 = rawdat.data[42, :] >rawdat.fluor = rawdat.data[3:42, :].sum(axis=0) > ># smooth data with Savitzky-Golay (data is ~evenly sampled in angle) >smoothed = group(energy=rawdat.energy) >smoothed.i0= savitzky_golay(rawdat.i0,5, 1) >smoothed.fluor = savitzky_golay(rawdat.fluor, 5, 1) > >newplot(smoothed.energy, rawdat.fluor, win=3) >plot(smoothed.energy, smoothed.fluor, zorder=50, linewidth=4, win=3) > >### create "normal" XAFS energies, finely spaced over edge, >### but spaced evenly in k for the EXAFS > >e0= 8980.0 # nominal Cu K edge >emin = smoothed.energy.min() >emax = smoothed.energy.max() - 5.0 >epre1 = e0 - 20.0 >epre2 = e0 + 20.0 >kmin = etok(20.0) >kmax = etok(emax-e0) > >def make_range(start, stop, step): >npts = max(2, 1 + int(0.1 + abs(stop-start)/abs(step))) >return linspace(start, stop, npts) >#enddef > >en_new = concatenate((make_range(emin, epre1, 5), > make_range(epre1+0.5, epre2-0.5, 0.5), > e0 + ktoe(make_range(kmin, kmax, 0.05 > ># now interpolate smoothed data onto this energy grid >dat = group(energy=en_new, filename=datafile) >dat.i0= interp(smoothed.energy, smoothed.i0,dat.energy) >dat.fluor = interp(smoothed.energy, smoothed.fluor, dat.energy) >dat.mu= dat.fluor / dat.i0 > ># run autobk >autobk(dat, rbkg=0.8, kweight=1) > ># plot data >plot_mu(dat, show_norm=True) >plot_chik(dat, kweight=1, win=2) > >### > > >html-part included links--- >http://dat.mu > >---text/plain--- >___ >Ifeffit mailing list >Ifeffit@millenia.cars.aps.anl.gov >http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit >Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit ___ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
Re: [Ifeffit] Athena: Data cannot be imported by using plugin PFBL12C
Dear Robert, Thanks for the reply and advice. I will try to use the larch, instead of athena with Ifeffit, and process the data by a proper method. Best regards, Yuji Hiraoka > Hi Yuri, > >You are probably running into the memory limit. This has been discussed a >number of times >on this mailing list, e.g.: > >https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg03931.html > >https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg05673.html > >Bruce's answer in the 2nd link provides some suggestions. > >Note, if I delete 1500 rows or so from your fail file, I am able to import it. >This seems to be similar to the success.dat file which also appears to be >truncated. > >Using a column labelled '20' as Io seems to work >[img:daokbeheloophpeb.png] > >-R. > >On 11/14/2017 12:59 AM, 平岡 裕治 wrote: > Dear all, > >I tried to import a data of multi-element detector at SPring-8 by using >plugin PFBL12C to Athena, but it was not successful. >if the number of rows in the file is about 2000[success.dat], it can be >successfully imported. But for the data over about 4000 rows[fail.dat], >it seems that several data at first of both converted energy and >intensity are substituted by 0. column6.png shows intensity of ch6 vs >energy_attained and I0.png shows intensity of I0 vs energy_attained. you >can see this behavior. As a result, intensity cannot divide by I0, and >data cannot be imported. > >if someone knows approach to solve this, could you teach me. > >Best regards, >Yuji Hiraoka > > >___ >Ifeffit mailing list >mailto:Ifeffit@millenia.cars.aps.anl.gov >http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit >Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit > > >---text/plain--- >___ >Ifeffit mailing list >Ifeffit@millenia.cars.aps.anl.gov >http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit >Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit > >---text/plain--- >___ >Ifeffit mailing list >Ifeffit@millenia.cars.aps.anl.gov >http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit >Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit ___ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
Re: [Ifeffit] Athena: Data cannot be imported by using plugin PFBL12C
I have to admit I find this all sort of disappointing. First, the PFBL12C is a really poor way to represent EXAFS data. It uses a hard-to-parse and hard-to-interpret format and has many unlabeled columns. It appears to include this really odd notion that recording "angle" is somehow better than recording energy, and saves a ridiculuously over-sampled spectrum that needs smoothing because of the high noise per recorded point. It is, in short, a really poor way to represent EXAFS data. We've seen it for many years, we've written papers and published supported code for better representations of EXAFS data. Still, we get questions about using data recorded in such files. Data files this complex need a way to convert them into something useful. Second, the limitations you are seeing are due to using the outdated, no-longer-supported (for several years now actually) Ifeffit library. These data can be read and processed without problem with Larch, the replacement for Ifeffit. Stop using Ifeffit. Use Larch instead. Really. An example script to use larch for your `fail.dat` file is listed below. It is somewhat complicated because the raw data file is such a poor representation of EXAFS data. --Matt # larch script to read, smooth and process data in the horrible PFBL12C format # read raw data, extract angle, i0, sum of fluorescence datafile = 'fail.dat' rawdat = read_ascii(datafile) def degrees2eV(angle, dspace=1.63747): from scipy.constants import hbar, c, e hbarc = 1.e10*hbar*c/e return (pi*hbarc/dspace) / sin(angle*pi/180.0) #enddef rawdat.angle = rawdat.data[0, :] # angle in degrees. yikes! rawdat.energy = degrees2eV(rawdat.angle) rawdat.i0 = rawdat.data[42, :] rawdat.fluor = rawdat.data[3:42, :].sum(axis=0) # smooth data with Savitzky-Golay (data is ~evenly sampled in angle) smoothed = group(energy=rawdat.energy) smoothed.i0= savitzky_golay(rawdat.i0,5, 1) smoothed.fluor = savitzky_golay(rawdat.fluor, 5, 1) newplot(smoothed.energy, rawdat.fluor, win=3) plot(smoothed.energy, smoothed.fluor, zorder=50, linewidth=4, win=3) ### create "normal" XAFS energies, finely spaced over edge, ### but spaced evenly in k for the EXAFS e0= 8980.0 # nominal Cu K edge emin = smoothed.energy.min() emax = smoothed.energy.max() - 5.0 epre1 = e0 - 20.0 epre2 = e0 + 20.0 kmin = etok(20.0) kmax = etok(emax-e0) def make_range(start, stop, step): npts = max(2, 1 + int(0.1 + abs(stop-start)/abs(step))) return linspace(start, stop, npts) #enddef en_new = concatenate((make_range(emin, epre1, 5), make_range(epre1+0.5, epre2-0.5, 0.5), e0 + ktoe(make_range(kmin, kmax, 0.05 # now interpolate smoothed data onto this energy grid dat = group(energy=en_new, filename=datafile) dat.i0= interp(smoothed.energy, smoothed.i0,dat.energy) dat.fluor = interp(smoothed.energy, smoothed.fluor, dat.energy) dat.mu= dat.fluor / dat.i0 # run autobk autobk(dat, rbkg=0.8, kweight=1) # plot data plot_mu(dat, show_norm=True) plot_chik(dat, kweight=1, win=2) ### ___ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
Re: [Ifeffit] Athena: Data cannot be imported by using plugin PFBL12C
Hi Yuri, You are probably running into the memory limit. This has been discussed a number of times on this mailing list, e.g.: https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg03931.html https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg05673.html Bruce's answer in the 2nd link provides some suggestions. Note, if I delete 1500 rows or so from your fail file, I am able to import it. This seems to be similar to the success.dat file which also appears to be truncated. Using a column labelled '20' as Io seems to work -R. On 11/14/2017 12:59 AM, 平岡 裕治 wrote: Dear all, I tried to import a data of multi-element detector at SPring-8 by using plugin PFBL12C to Athena, but it was not successful. if the number of rows in the file is about 2000[success.dat], it can be successfully imported. But for the data over about 4000 rows[fail.dat], it seems that several data at first of both converted energy and intensity are substituted by 0. column6.png shows intensity of ch6 vs energy_attained and I0.png shows intensity of I0 vs energy_attained. you can see this behavior. As a result, intensity cannot divide by I0, and data cannot be imported. if someone knows approach to solve this, could you teach me. Best regards, Yuji Hiraoka ___ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit ___ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
