Il agree, Richard, but we are not really looking for accuracy here, we are looking for a general trend. The method is approximative and, as Jean-Claude mentions rightfully, it suffers a terrible language and domain bias. In other words, it is plagued by a strong underestimation. Whether ORBi's compliance level is 70, 80 or 90% is not a major concern to me (even though I would love it to reach 100%!), I must admit. I am satisfied to observe that it is very high and not in the 15-30% range which is what happens when a mandate is not being enforced by a link to assessment procedures.
Bernard > Le 21 sept. 2014 à 10:51, "Richard Poynder" <ri...@richardpoynder.co.uk> a > écrit : > > As a layperson I would certainly be interested to know what margin of error > levels we can assume the “Web of Science and/or in Scopus” approach has. I am > conscious, for instance, that some of the reports by UK universities into > RCUK compliance mention using Web of Science, but they all appear keen to > stress that they have serious concerns about data accuracy. > > A list of RCUK compliance reports, by the way, can be found here: > http://goo.gl/Yi3twT > > There is also a very informative blog post on the topic of monitoring open > access mandates/policies by Cameron Neylon here: http://goo.gl/Y02S87 > > Richard Poynder > > > > > From: goal-boun...@eprints.org [mailto:goal-boun...@eprints.org] On Behalf Of > Jean-Claude Guédon > Sent: 20 September 2014 23:27 > To: Global Open Access List (Successor of AmSci) > Subject: [GOAL] Re: Fwd: The Open Access Interviews: Paul Royster > > Extremely good answer, Bernard! > > It is also very good to clarify the fact that the 90% figure is calculated > against the baseline of a combined WoS_Scopus search. However, and this was > part of my difficulties with Stevan's argument, I suspect that in SSH, in a > French-speaking university, many publish in French-language journals that do > not appear in either list. This means that, for Liège, the baseline works > from one year to the next, but if you want to compare Liège's mandate and its > effectiveness (which, once again, I agree, is - from common sense - the best) > with another kind of mandate in an English-speaking university, the baselines > will not be comparable. > > If, furthermore, you imagine two universities that not only differ > linguistically, but also differ in the relative weight of disciplines in > research output - say one heavily slanted STM and the other heavily slanted > SSH, this too will affect the baseline simply by virtue of the fact that SSH > publications are not as well covered by WoS and Scopus as are STM > publications. > > In conclusion, the baseline is OK for comparisons of a mandate's > effectiveness longitudinally, of for comparison purposes of two successive, > but different, mandates, assuming the institution remains pretty much the > same over time in terms of mix of research emphases; it is far more > questionable across institutions, especially when different languages are > involved (but not only). > > Incidentally, what proportion of papers deposited in ORBI do not appear in > either WoS or Scopus? That too would be interesting to know as it might help > Stevan refine his baseline and thus make it more convincing. > > Finally, given that all universities require, an annual assessment of > performance, including a bibliography of publications in the completed year, > would it be difficult to compare the repository's holding against the > publications of the researchers as declared by them? Knowing researchers, > every last little scrap of paper will be minutely listed in the yearly > assessment forms... <image001.png> > > -- > > Jean-Claude Guédon > Professeur titulaire > Littérature comparée > Université de Montréal > Le samedi 20 septembre 2014 à 19:10 +0200, Bernard Rentier - IMAP a écrit : > > Dear Richard, > > > Here are the answers: > > > 1. ORBi, the Liège University Repository, will soon (I believe) reach 90% > compliance. It is our target for 2014 and I hope we make it. > This figure comes from the calculation of the percentage of ULg papers that > can be found in Web of Science and/or in Scopus that are deposited in ORBi as > well (see method in http://eprints.soton.ac.uk/340294/) > It concerns one year at a time and it is not cumulative. Last May, the > compliance level for the publications of 2013 was already 73% and our figure > for 2012 is in the 80% range. > > > 2. Only a small proportion of ULg papers are in CC-BY. > This is simply because, in order to publish in the journal of their choice (I > haven’t tried to do anything against that!), our authors, in the great > centuries-old tradition, give away their rights to the publisher. We have no > control on that. > Later on, there is no way for them to CC-BY the same text (in fact, we are > preparing ORBi 2.0, that will offer a CC-BY choice). > For now, we are aiming at free access and we are not yet fighting hard for > re-use rights. We shall move progressively in this direction of course, while > the publishing mores evolve… > In other words, I agree that we have free access, not a full fledge open > access yet. It is not a failure, it is our objective to gain confidence first. > Unfortunately, even if we have established in-house rules for evaluation, > external evaluations are still based on traditional indicators such as the > highly and rightfully criticized but widely used Impact Factor and the like. > In these conditions, today we cannot sacrifice our researchers — singularly > the young ones — in the overall competition for jobs and funds, on the altar > of « pure » Open Access. > > > Best wishes > > > Bernard Rentier > Rector, University of Liège, Belgium > > > > > > > > Le 19 sept. 2014 à 21:52, Richard Poynder <ri...@richardpoynder.co.uk> a > écrit : > > Dear Bernard, > > I have two questions if I may: > > 1. You say that Liège is getting close to 90% compliance. Can you > explain how you know that, and how you calculate compliance levels? I ask > this because the consistent theme coming through from UK universities with > regard to compliance to the RCUK OA mandate is that they simply do not know > how many research outputs their faculty produce each year. If that is right, > what systems does Liège have in place to enable it to produce a comprehensive > list of research outputs that UK universities apparently do not have? > > 2. Does Liège track the licences attached to the deposits in its > repository? If so, can you provide some stats, especially the number of items > that are available CC-BY (which we are now told is required before a deposit > can be characterised as being open access? > > Thank you. > > > Richard Poynder > > > From: goal-boun...@eprints.org [mailto:goal-boun...@eprints.org] On Behalf Of > brent...@ulg.ac.be > Sent: 19 September 2014 18:46 > To: Global Open Access List (Successor of AmSci) > Subject: [GOAL] Re: Fwd: The Open Access Interviews: Paul Royster > > "Liège does not mandate anything, so far as I know; it only looks into the > local repository (Orbi) to see what is in it, and it does so to assess > performance or respond to requests for promotions or grant submissions." (JC. > Guédon) > > > > Oh no, Jean-Claude, Liège mandates everything. > It is a real mandate and it took me a while to get almost every ULg > researcher to realise that it is to his/her benefit. > Linking the deposits to personal in-house assessment was the trick to get the > mandate enforced in the first place. As well as a few positive incentives and > a lot of time consuming persuasion (but it was well worth it). > Last Wednesday, the Liège University Board has put an ultimate touch of > wisdom on its mandate by adding "immediately upon acceptance, even in > restricted access" in the official procedure. Actually, a nice but to some > extent useless addition because, with time (the mandate was imposed in 2007), > ULg authors have become so convinced of the increase in readership and > citations that two thirds of them make their deposits between the date of > acceptance and the date of publication. > All this explains why we are getting close to 90% compliance, an outstanding > result, I believe. > > > > > > Le 18 sept. 2014 à 23:40, Jean-Claude Guédon > <jean.claude.gue...@umontreal.ca> a écrit : > > > A reasonably quick response as I do not want to go into discursive tsunami > mode... > > 1. Stevan admits that his evaluation of compliance is an approximation, easy > to get, but not easy to correct. This approximation varies greatly from one > institution to another, one circumstance to another. For example, he admits > that language plays a role; he should further admit that the greater or > smaller proportion of SSH researchers in the research communities of various > institutions will also play a role. in short, comparing two institutions by > simply using WoS approximations appears rash and unacceptable to me, rather > than simply quick and dirty (which I would accept as a first approximation). > > The impact factor folly was mentioned because, by basing his approximation on > the WoS, Stevan reinforces the centrality of a partial and questionable tool > that is, at best, a research tool, not a management tool, and which stands > behind all the research assessment procedures presently used in universities, > laboratories, etc. > > 2. Stevan and I have long differed about OA's central target. He limits > himself to journal articles, as a first step; I do not. I do not because, in > the humanities and social sciences, limiting oneself to journal articles > would be limiting oneself to the less essential part of the archive we work > with, unlike natural scientists. > > Imagine a universe where a research metric would have been initially designed > around SSH disciplines and then extended as is to STM. In such a parallel > universe, books would be the currency of choice, and articles would look like > secondary, minor, productions, best left for later assessments. Then, one > prominent OA advocate named Stenan Harvard might argue that the only way to > proceed forward is to focus only on books, that this is OA's sole objective, > and that articles and the rest will be treated later... Imagine the reaction > of science researchers... > > 3. Liège does not mandate anything, so far as I know; it only looks into the > local repository (Orbi) to see what is in it, and it does so to assess > performance or respond to requests for promotions or grant submissions. If > books and book chapters are more difficult to treat than articles, then place > them in a dark archive with a button. This was the clever solution invented > by Stevan and I agree with it. > > 4. To obtain mandates, you need either faculty to vote a mandate on itself > (but few universities have done so), or you need administrators to impose a > mandate, but that is often viewed negatively by many of our colleagues. > Meanwhile, they are strongly incited to publish in "prestigious journals" > where prestige is "measured" by impact factors. From an average researcher's > perspective, one article in Nature, fully locked behind pay-walls, is what is > really valuable. Adding open access may be the cherry on the sundae, but it > is not the sundae. The result? OA, as of now, is not perceived to be directly > significant for successfully managing a career. > > On the other hand, the OA citation advantage has been fully recognized and > accepted by publishers. That is in part why they are finally embracing OA: > with high processing charges and the increased citation potential of OA, they > can increase revenues even more and satisfy their stakeholders. This is > especially true if funders, universities, libraries, etc., are willing to pay > for the APC's. This is the trap the UK fell into. > > 5. SSH authors are less interested in depositing articles than STM > researchers because, for SSH researchers, articles have far less importance > than books (see above), and, arguably, book chapters. > > 6. I am not citing rationales for the status quo, and Stevan knows this well. > This must be the first time that I have ever been associated with the status > quo... Could it be that criticizing Stevan on one point could be seen by him > as fighting for the status? But that would be true only if Stevan were right > beyond the slightest doubt. Hmmmmmmmmmm! > > I personally think he is right on some points and not so right on others. > > Also, I am simply trying to think about reasons why OA has been so hard to > achieve so far, and, in doing so, I have come to two conclusions: too narrow > an objective and too rigid an approach can both be counter-productive. > > This said, trying to have a method to compare deposit rates in various > institutional and mandate circumstances would be very useful. I support > Stevan's general objective in this regard; I simply object to the validity of > the method he suggests. Alas, I have little to suggest beyond my critique. > > I also suggest that a better understanding of the sociology of research (not > the sociology of knowledge) is crucial to move forward. > > Finally, I expect that if I saw Stevan self-archive his abundant scientific > production, I would be awed by the lightning speed of his keystrokes. But are > they everybody's keystrokes? > > Jean-Claude Guédon > > > > -- > > Jean-Claude Guédon > Professeur titulaire > Littérature comparée > Université de Montréal > Le jeudi 18 septembre 2014 à 12:28 -0400, Stevan Harnad a écrit : > On Wed, Sep 17, 2014 at 10:53 AM, Jean-Claude Guédon > <jean.claude.gue...@umontreal.ca> wrote: > Most interesting dialogue. > > I will focus on two points: > > 1. Using the Web of Science collection as a reference: this generates all > kinds of problems, particularly for disciplines that are not dominated and > skewed by the impact factor folly. This is true, for example, of most of the > social sciences and the humanities, especially when these publications are > not in English. > > > > The purpose of using WoS (or SCOPUS, or any other standardized index) as a > baseline for assessing OA repository success is to be able to estimate (and > compare) what percentage of an institution's total annual refereed journal > article output has been self-archived. > > > Raw total or annual deposit counts tell us neither (1) whether the deposits > are refereed journal articles nor (2) when the articles were published, nor > (most important of all) (3) what proportion of total annual refereed journal > article output is deposited. > > > Institutions do not know even know their total annual refereed journal > article output. (One of the (many) reasons for mandating self-archiving is in > order to get that information.) > > > The WoS (or SCOPUS, or other) standardized database provides the denominator > against which the deposits of those articles provide the numerator. > > > Once that ratio is known (for WoS articles, for example), it provides an > estimate of the proportion of total institutional article output deposited. > > > Anyone can then "correct" the ratio for their institution and discipline, if > they wish, by simply taking a (large enough) sample of total institutional > journal article output for a recent year and seeing what percentage of it is > in WoS! (This would obviously have to be done discipline by discipline; and > indeed the institutional totals should also be broken down and analyzed by > discipline.) > > > So if D/W, the WoS-deposit/total-WoS ratio = R, and w/s, the > WoS-indexed-portion/total-output-sample = c, then c can be used to upgrade W > to the estimate of total institutional article output, and the WoS deposit > ratio R can be compared to the deposit ratio for the non-WoS sample (which > must not, of course, be derived from the repository, but some other way!) to > get a non-WoS ratio of Rc. > > > My own prediction is that R and Rc will be quite similar, but if not, c can > also be used to correct R to better reflect both WoS and non-WoS output and > their relative sizes. > > > But R is still by far the easiest and fastest way to get an estimate of > institutional deposit percentages. > > > (As far as I can see, none of this has much to do with impact factor folly. > For non-English-language institutions, however, the non-WoS correction may be > more substantial.) > > > Stevan has also and long argued about limiting oneself to journal articles. I > have my own difficulties with this limitation because book chapters and > monographs are so important in the disciplines that I tend to work in. Also, > I regularly write in French as well as English, while reading articles in a > variety of languages. Most of the articles that are not in English are not in > the Web of Science. A better way to proceed would be to check if the journals > not in the WoS, and corresponding to deposited articles, are peer-reviewed. > The same could be done with book chapters. Incidentally, if I limited myself > to WoS publications for annual performance review, I would look rather bad. I > suspect I am not the only one in such a situation, while leading a fairly > honourable career in academe. > > > > Authors are welcome to deposit as much as they like: articles, chapters, > books, data, software. > > > But OA's primary target (and also its primary obstacle) is journal articles. > Ditto for OA mandates. > > > All disciplines, including the social sciences and humanities, in all > languages, write journal articles. This discussion is about the means of > measuring the success of an OA self-archiving mandate. It applies to all > journal articles (and refereed conference articles) in all disciplines. > > > There are problems with mandating book deposit, or even book chapter deposit, > so that is being left for later. > > > Nothing is being said about performance review except that the way to submit > journal articles should be stipulated to be repository deposit. > > 2. The issue of rules and regulations. It is absolutely true that a procedure > such as the one adopted at the Université de Liège and which Stevan aptly > summarizes as (with a couple of minor modifications): "henceforth the way to > submit refereedjournal article publications for annual performance review is > to deposit them in the [appropriate] IR ". > > > Liège does not mandate the deposit of books. > > However, obtaining this change of behaviour from an administration is no > small task. At the local, institutional, level, it corresponds to a > politically charged effort that requires having a number of committed OA > advocates working hard to push the idea. Stevan should know this from his own > experience in Montreal; he should also know that, presently, the Open Access > issue is not on the radar of most researchers. In scientific disciplines, > they tend to be mesmerized by impact factors without making the link between > this obsession and the OA advantage, partly because enough controversies have > surrounded this issue to maintain a general feeling of uncertainty and doubt. > In the social sciences and humanities where the citation rates are far less > "meaningful" - I put quotation marks here to underscore the uncertainty > surrounding the meaning of citation numbers: visibility, prestige, quality? - > the benefits of self-archiving one's articles in open access are less obvious > to researchers, especially if they do not adopt a global perspective on the > importance of the "grand conversation" needed to produce knowledge in an > optimal manner, but rather intend to manage and protect their career. > > > > I am not sure what is the point of the above observations. I agree it has > been difficult to get authors to deposit. That's why the OA movement has > turned to mandates, and now to ways of optimizing mandates so as to > facilitate and maximize success (i.e. deposit rates). And here we are just > talking about how to measure and compare those deposit rates between > institutions, and between mandates. > > > Nothing about impact factors, performance evaluation criteria, metrics, > discipline criteria or language differences. Just ways to induce journal > article authors to deposit them in their institutional repositories. > > > Saying all this is not saying that we should not remain committed to OA, far > from it; is is simply saying that the chances of success in reaching OA will > not be significantly improved by simply referring to "huge" benefits at the > cost of only a few extra keystrokes. This is rhetoric. The last time I > deposited an article of mine, given the procedure used in the depository I > was using, it took me close to half an hour to enter all the details required > by that depository - a depository organized by librarians, mainly for > information science specialists. All these details were legitimate and > potentially useful. However, while I was absolutely sure I was doing the > right thing, I could well understand why a colleague less sanguine about OA > than I am might push this task to the back burner. In fact, I did so myself > for several months. Shame on me, probably, but this is the reality of the > quotidian. > > > > I invite Jean-Claude to time me depositing an article in my institutional > repository (and I am not a fast typist)! It takes about two minutes. > > > In conclusion, i suspect that if Stevan focuses on such a narrowly-defined > target - journal articles in the STM disciplines - this is because he gambles > on the fact that making these disciplines fully OA would force the other > disciplines in the humanities and social sciences to follow suit sooner or > later. Perhaps, it is so, but perhaps it is not. Meanwhile, arguing in this > fashion tends to alienate practitioners of the humanities and the social > sciences, so that the alleged advantages of narrowly focusing on a > well-defined target are perhaps more than negatively compensated by the > neglect of SSH disciplines. yet, the latter constitute about half, if not > more, of the researchers in the world. > > > The target is journal articles in all disciplines. Not clear why SSH journal > article authors would be any more or less compliant with self-archiving > mandates than any other discipline. It has nothing to do with books, yet. > > > Yes, once journal articles are being self-archived universally, many other > things will follow. > > > I suggest that it may be more constructive to practice deposit keystrokes to > provide OA than to cite a-priori rationales for the status quo, Jean-Claude. > I bet you'll be up to speed after depositing just a few articles! > > > Stevan Harnad > Le mercredi 17 septembre 2014 à 07:07 -0400, Stevan Harnad a écrit : > > > > Begin forwarded message: > > > > From: Stevan Harnad <har...@ecs.soton.ac.uk> > > Subject: Re: The Open Access Interviews: Paul Royster > Date: September 16, 2014 at 5:28:48 PM GMT-4 > > To: jisc-repositor...@jiscmail.ac.uk > > > > On Sep 16, 2014, at 2:46 PM, Paul Royster <proyst...@unl.edu> wrote: > > > > At the risk of stirring up more sediment and further muddying the waters of > scholarly communications, > but in response to direct questions posed in this venue earlier this month, I > shall venture the following … > > Answers for Dr. Harnad > > (1) What percentage of Nebraska-Lincoln output of peer-revewed journal > articles (only) per year is > deposited in the N-L Repository? About 3 months ago I furnished your graduate > student (at least he > said he was your student) with 5 years of deposit data so he could compare it > to Web of Science > publication dates and arrive at some data-based figure for this. I cautioned > him that I felt Web of > Science to be a narrow and commercially skewed comparison sample, but I sent > the data anyway. > So I expect you will have an answer to this query before I do. If the news is > good, I hope you will > share it with this list; if not, then let your conscience be your guide. As > for benchmarking, I don’t believe > it is a competition, and every step in the direction of free scholarship is a > positive one. I hope when > they hand out the medals we at least get a ribbon for participation. > > > Thanks for reminding me! It was my post-doc, Yassine Gargouri, and I just > called him to ask about > the UNL results. He said he has the UNL data and will have the results of the > analysis in 2-3 weeks! > > > So the jury is still out. But many thanks for sending the data. Apparently > Sue was not aware that UNL > had provided those data (and I too had forgotten!). > > > > (2) Why doesn’t N-L adopt a self-archiving mandate? > I do not even attempt to explain the conduct of the black box that is my > university’s administration; > so in short, I cannot say why or why not. I can only say why I have not > campaigned for adoption of > such a mandate. My reasons have been purely personal and idiosyncratic, and > I do not hold them > up as a model for anyone else or as representing the thinking or attitude of > this university. Bluntly, > I have not sought to create a mandate because I feel there are enough > regulations and requirements > in effect here already. Instituting more rules brings further problems of > enforcement or compliance, > and it creates new categories of deviance. There are already too many rules: > we have to park in > designated areas; we have to drink Pepsi rather than Coke products; we have > to wear red on game > days; we can’t enter the building through the freight dock; etc. etc. etc. I > simply do not believe in > creating more rules and requirements, even if they are for our own good. The > Faculty Senate > voted to “endorse and recommend” our repository; I have not desired more than > that. But I am > concerned mainly with 1600 faculty on two campuses in one medium-sized > university town—not > with a universal solution to the worldwide scholarly communications crisis. I > see discussions lately > about “putting teeth” into mandated deposit rules, and I wonder—who is > intended to be bitten? > Apparently, the already-beleaguered faculty. > > > I agree that we are over-regulated! But I think that doing a few extra > keystrokes when a refereed > final draft is accepted for publication is really very little, and the > potential benefits are huge. Also, > there is some evidence as to how authors comply with a self-archiving mandate > — if it’s the right > self-archiving mandate, i.e., If the mandate simply indicates that henceforth > the way to submit refereed > journal article publications for annual performance review is to deposit them > in UNL’s IR (rather than > however they are being submitted currently) then UNL faculty will comply as > naturally as they did > when it was mandaed that submissions should be online rather than in hard > copy. It’s just a technological upgrade. > > > > > (3) Why do you lump together author-pays with author-self-archives? > I was not aware that I did this, so perhaps you are responding to Sue’s > catalog of various proposed > solutions—“author-pays OA, mandated self-archiving of manuscripts, CHORUS, > SHARE, and others”—as > all being “ineffectual or unsustainable initiatives to varying degrees.” I > feel we are strong believers and > even advocates for author self-archiving (so-called), and disdainful > non-advocates for author-pays models. > But I think we have become aware of the divergence of interests between the > global theoretics of the > open access “movements” on the one hand and the “boots-on-the-ground” > practicalities of managing > a local repository, even one with global reach, on the other. Crusades for > and controversies about > “open access” have come to seem far removed from what we actually do, and now > seem more of a > distraction than a help or guide. > > > I can understand that, from the library’s perspective: The library can’t > mandate self-archiving, can’t fund > author-pays, and can’t do anything about authors’ rights. But maybe, if you > look at the evidence that > mandates work, and become convinced, then the library could encourage the > administration… And > of course if self-archiving is mandated at UNL, then the library can help > with mediated self-archiving, > at least initially, as I pointed out to Sue (though it’s hardly necessary, > for a few keystrokes — certainly > a much smaller task than UNL’s current mediated deposit: tracking down the > PDF. checking the rights. etc.). > > > > We have been (and continue to be) constant supporters of “green” open access; > and we have appreciated > Dr. Harnad’s reliably indefatigable defenses of that cause against > innumerable critics, nay-sayers, and > “holier-than-thou” evangelists of competing approaches. I sympathize with his > weariness, I applaud his tirelessness, > and I do not wish to tax his patience further. I hope no part of this > response will be interpreted as attempting > to dispute, contradict, or diminish any of his points. I regret if these > answers are unsatisfactory or incomplete, > but that is all I can manage at this time. > > > Much appreciated, Paul! > > > Hope to have the UNL data for you soon, with a comparison with other IRs, > mandated and unmandated. > > > Best wishes, > > > Stevan > > > > > Paul Royster > Coordinator of Scholarly Communications > University of Nebraska–Lincoln > proys...@unl.edu > http://digitalcommons.unl.edu > > > > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > > > > > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > > > > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal > _______________________________________________ > GOAL mailing list > GOAL@eprints.org > http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal
_______________________________________________ GOAL mailing list GOAL@eprints.org http://mailman.ecs.soton.ac.uk/mailman/listinfo/goal
