Thank you for these all valuable information. I will check the translation table.
So far, I have managed to extract introduction part by making use of heading numbers. That is, mostly sections are numbered. For introduction I mostly encountered that it is: 1 Introduction (or INTRODUCTION) 1. Introduction (or INTRODUCTION) I. Introduction (or INTRODUCTION) I Introduction (or INTRODUCTION) So exploiting this information I retain the text until 2nd section heading appears. And then I again check the font type of this 2nd section heading text With the introduction heading font type. By doing so, as you mentioned I can grab many templates such as IEEExplore, Spring etc. Best regards, Mehmet From: [email protected] [mailto:[email protected]] On Behalf Of Peter Murray-Rust Sent: Saturday 8 November 2014 7:45 PM To: Mehmet Ali Abdulhayoglu Cc: [email protected] Subject: Re: Is sub-heading extraction possible? I've spent a lot of time on papers like this! Be aware that the typesetting almost certainly won't use Unicode by default. If you want to capture the maths you may need to use my translation tables in PDF2SVG. There are (I think) two solutions to the problem: * create a human-manageable template (e.g. one for ISIT, one for Springer, ...) I will be developing this over the next 1-2 months. The idea is that each journal (often publisher or publishing system) has a human creaed/editable template advising PDF2STM what to expect. * develop heuristics and adaptive systems. There are several complemntary and conflicting approaches: - whitespace - numbering - Font size and appearance - human-language meaning for some journals we find that A works, for another B... It may be possible to develop a system that works out the best set of rules for a journals. Perhaps a decision tree would be best? But it's a research area at best. P. On Sat, Nov 8, 2014 at 3:06 PM, Mehmet Ali Abdulhayoglu <[email protected]<mailto:[email protected]>> wrote: Hi again, Through your suggestions I came to point where I can detect introduction (background etc.) part and heading's font type. If there is a Font type difference between header and the text belonging to that heading this approach works well to get only introduction part. However, for some other pdfs as I attached, heading and the texts belonging to it have the same font type. For such cases I have tried to make use of alignment or indentation but I could not make it. First, is there a way of getting those features (alignment etc.). Second, for the pdf I attached, is there any other suggestion to get only text part belonging to introduction? Thanks. Best regards, Mehmet -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Peter Murray-Rust Sent: Thursday 6 November 2014 7:38 PM To: [email protected]<mailto:[email protected]> Subject: Re: Is sub-heading extraction possible? Greetings, In general there is NO automatic way - it depends on how the paper was produced and what semantics are used. Note that PDFs do NOT contain words, sentences, paragraphs, sections - only characters (and sometimes only pixels) We have been working on this for 3 years using PDFBox and adding code on top (see http://bitbucket.org/petermr/pdf2svg and related projects). There are two main steps: * break the paper into sections. The best strategy is usually to use horizontal whitespace coupled with the typography of the section headings. With most publishers this is possible, but some use special boxes. Note that the strategy will depend on whether the language is ISOLatin (i.e. L2R, Top2Bottom) or other. * identify the role of the sections. In general you have to know the language of the paper - we only do ones in English at present. We are working with the European Bioinfromatics Institute who have a mapping of phrases onto common concepts (e.g. Introduction could be "Introduction", "Background", etc.). This may only hold for biosciences. There are many complications which can make it more difficult... Follow us at http://contentmine.org where we are starting to do this on a large scale (starting with XML, but moving to PDF later). P. On Thu, Nov 6, 2014 at 4:56 PM, Mehmet Ali Abdulhayoglu < [email protected]<mailto:[email protected]>> wrote: > Hi all, > > When a text extraction is tried from a scientific paper in pdf format, > is it possible to detect Headings and sub-headings? More specifically, > is it possible to extract only introduction Part or conclusion part? > > Thanks in advance. > > Best, > Mehmet > -- Peter Murray-Rust Reader in Molecular Informatics Unilever Centre, Dep. Of Chemistry University of Cambridge CB2 1EW, UK +44-1223-763069<tel:%2B44-1223-763069> -- Peter Murray-Rust Reader in Molecular Informatics Unilever Centre, Dep. Of Chemistry University of Cambridge CB2 1EW, UK +44-1223-763069
