Hi Anthony
My interest lies in the question you raised - how to machine learn the structure of a paragraph (not a document yet), given parse trees of individual sentences. Doc2vec is one direction, but my personal preference is more explicit, structure-based. In my opinion, deep learning family of approaches leverages a huge training dataset they train from, but lacks representing of logical structure of a given document. On the other hand, a discourse tree of a paragraph is a good way to link individual parse trees in a structure to represent a paragraph of text, but lacks extensive knowledge for how n-grams form "meanings" in documents. Therefore I believe doc2vec and learning of discourse trees complement each other. To systematically learn discourse tree in addition to parse trees, we use tree kernel learning. It forms the space of all sub-trees of trees with abstract labels, and does SVM learning in it. We combine regular parse trees and links between sentences such as rhetoric relations. The application areas are: - answering multi-sentence questions - document-level classification, text style recognition, e.g. for security domain - where documents include the same words but need to be classified by style. - content generation where maintaining of rhetoric structure is important. >The generalizations could hurt the classification performance in some >tasks, but seem to be more useful when the target documents are larger. Yes, in this case discourse trees are less plausible. >It could also be possible to chose the "document" to be a single word as >well, reducing the underlying matrix to an array, does that make sense? Have not thought about it / have not tried it either >Therefore, we could also use document based vectors for mid to high-layer >tasks (doc cat, sentiment, profile etc..). What do you think? I think for doc classification yes, for sentiments - I am more skeptical, although it is the evaluation area of (Mikolov et al). Do you have a particular problem in mind? I can share code on git / papers on the above. Another way to look at deep learning for NLP : deep learning kind of takes science away from linguistics and makes it more like engineering, I am not sure it is a direction for openNLP? Regards Boris ________________________________ From: Anthony Beylerian <anthony.beyler...@gmail.com> Sent: Wednesday, June 29, 2016 11:24:02 AM To: dev@opennlp.apache.org Subject: Re: DeepLearning4J as a ML for OpenNLP Hi Boris, Thank you very much for sharing your experience with us! Is it possible to ask you for more information? I have only just recently used d4lj with some introductory material, however I have also felt doc2vec could also be quite useful, although my understanding of it is still limited. My current understanding is that doc2vec as an extension of word2vec, can capture a more generalized context (the document) instead of just focusing on the context of a single word, in order to provide features useful to classify that document. The advantage would be to better capture latent information that exist in the document (such as the order of words), instead of just averaging word vectors, or through other approaches on the document level (would love some feedback on this) The generalizations could hurt the classification performance in some tasks, but seem to be more useful when the target documents are larger. It could also be possible to chose the "document" to be a single word as well, reducing the underlying matrix to an array, does that make sense? Therefore, we could also use document based vectors for mid to high-layer tasks (doc cat, sentiment, profile etc..). What do you think? It would be fantastic to clarify, I believe that would also motivate more people to pitch in and better assist with this. Thanks, Anthony Hi William I have never heard of Features2Vec. I think for low-level tasks, pre-linguistic tasks such as text classification where we don't want to build models and have a one-fits-all solution, Word2Vec works well. I used it in industrial environment for text classification, some information extraction and content generation tasks. So I think it should also work for low-level OpenNLP tasks. Regards Boris ________________________________ From: William Colen <william.co...@gmail.com> Sent: Wednesday, June 29, 2016 4:43:25 AM To: dev@opennlp.apache.org Subject: Re: DeepLearning4J as a ML for OpenNLP Thank you, Boris. I am new to DeepLearning, so I have no idea the issues we would face. I was wondering if we can use Features2Vec instead of Word2Vec, does it make any sense? The idea was to use DL in low level NLP tasks where we don't have parse trees yet. 2016-06-29 6:34 GMT-03:00 Boris Galitsky <bgalit...@hotmail.com>: > Hi guys > > I should mention how we used DeepLearning4J for the OpenNLP.Similarity > project at > > https://github.com/bgalitsky/relevance-based-on-parse-trees > > > The main question is how word2vec models and linguistic information such > as part trees complement each other. In a word2vec approach any two words > can be compared. The weakness here is that when learning is based on > computing a distance between totally unrelated words like 'cat' and 'fly' > can be meaningless, uninformative and can corrupt a learning model. > > > In OpenNLP.Similarity component similarity is defined in terms of parse > trees. When word2vec is applied on top of parse trees and not as a > bag-of-words, we only compute the distance between the words with the same > semantic role, so the model becomes more accurate. > > > There's a paper on the way which does the assessment of relevance > improvent for > > > word2vec (bag-of-words) [traditional] vs word2vec (parse-trees) > > > Regards > > Boris > > [https://avatars3.githubusercontent.com/u/1051120?v=3&s=400]< > https://github.com/bgalitsky/relevance-based-on-parse-trees> > > bgalitsky/relevance-based-on-parse-trees< > https://github.com/bgalitsky/relevance-based-on-parse-trees> > github.com > Automatically exported from > code.google.com/p/relevance-based-on-parse-trees > > > > > ________________________________ > From: Anthony Beylerian <anthony.beyler...@gmail.com> > Sent: Wednesday, June 29, 2016 2:13:38 AM > To: dev@opennlp.apache.org > Subject: Re: DeepLearning4J as a ML for OpenNLP > > +1 would be willing to help out when possible >
