*IJCNLP-2017 Shared Task on Review Opinion Diversification* *First Call for Participation*
Website: https://sites.google.com/itbhu.ac.in/revopid-2017 <https://www.google.com/url?q=https%3A%2F%2Fsites.google.com%2Fitbhu.ac.in%2Frevopid-2017&sa=D&sntz=1&usg=AFQjCNGzlagZR-rmSiysbsF9JuMWEKumwQ> Contact email: [email protected] The shared task aims at producing, for each product, top-*k* reviews from a set of reviews such that the selected top-*k* reviews act as a summary of all the opinions expressed in the reviews set. The three independent subtasks incorporate three different ways of selecting the top-*k* reviews, based on helpfulness, representativeness and exhaustiveness of the opinions expressed in the review set. In the famous Asch Conformity experiment, individuals were asked to decide which of 2 sticks (which they were shown separately) was longer. The same task was then to be performed with a group of people (all of them actors, deliberately giving the wrong answer). The error rate leapt from 1% to 36.8% when the people around expressed the wrong perception. This goes to show how heavily can others’ opinions influence our own. For example, if on searching for 'iPhone reviews', we see results (ranked by, say, PageRank) that coincidentally happen to be against the product, then one might form an incorrect perception of the general opinion around the world regarding the smartphone. To avoid such a misconception, while summarizing documents, Opinion Diversification needs to be incorporated. As an introductory impetus to this approach, we propose this shared task, focusing on Product Reviews Summarization (in the form of a ranked list). Reviews have always played a crucial role for customers to select products informatively ever since information technology became a common part of life. Considering the large volume of reviews available at present, it becomes a difficult task for the customers to extract relevant information from huge amounts of data and they can often end up skipping some useful content, thereby making wrong choices. Thus, it is important to extract a representative set of reviews from a large set of data, while keeping all important content available in this representative set. *Task Description* The shared task consists of three independent subtasks. Participating systems are required to produce a top-*k* summarized ranking of reviews (one ranked list for each product for a given subtask) from amongst the given set of reviews. The redundancy of opinions expressed in the review corpus must be minimised, along with maximisation of a certain property. This property can be one of the following (one property corresponds to one subtask): 1) usefulness rating of the review 2) representativeness of the overall corpus of reviews 3) exhaustiveness of opinions expressed Participants are free to participate in one or more subtasks individually. Each subtask will be evaluated separately. *Definitions* *Review*: Review text and any other relevant metadata as may be deemed necessary to be used by the participating system, from the given data. *Corpus*: All the input reviews for a particular product. *Feature*: A ratable aspect of the product. *Perspective*: An ordered pair of an aspect and sentiment (towards that aspect) pair that appears in any review. *Subtask A (Usefulness Ranking)* Usefulness rating is a user-collected field in the provided training dataset. Given a corpus of reviews for a particular product, the goal is to rank the top-*k* of them, according to predicted usefulness rating, while simultaneously penalizing redundancy among the ranked list of reviews. An essential subsection of this task obviously includes predicting the usefulness rating for a particular review. Systems are advised to use the training corpus which has the actual usefulness rating of each review. Participants are free to choose their set of features for this supervised learning process. However, if any dataset other than the one provided is used for training purposes, it must be explicitly mentioned in the submission. *Subtask B (Representativeness Ranking)* Given a corpus of reviews for a particular product, the goal is to rank the top-*k* of them, so as to maximize representativeness of the ranked list, while simultaneously penalizing redundancy among the ranked list of reviews. The ranking should summarize the perspectives expressed in the reviews given as input, incorporating a trade-off between diversity and novelty. An ideal representation would be one that covers the popular perspectives expressed in the corpus, in proportion to their expression in the corpus (for that product), e.g. if 90 reviews claim that the iPhone cost is low, and 10 reviews claim that it is high, the former perspective should have 90% visibility in the final ranking and the latter should have 10% (or may even be ignored owing to low popularity) in the final ranking. The ranking should be such that for every i in 1<=i<=k, the top i reviews best represent the overall set of reviews for the product. That is, the #1 review should be the best single review to represent the overall opinion in the corpus; The combination of #1 and #2 reviews should be the best pair of reviews to represent the corpus, and so on. *Subtask C (Exhaustive Coverage Ranking)* Given a corpus of reviews for a particular product, the goal is to rank the top-*k* of them, so as to include the majority of popular perspectives in the corpus regarding the product, while simultaneously penalizing redundancy among the ranked list of reviews. This is similar to Subtask B, except that: In Subtask B, the final ranking is judged on the basis of how well the ranked list represents the most popular opinions in the review corpus, in proportion. In Subtask C, the final ranking is judged on the basis of the exhaustive coverage of the opinions in the final ranking. That means, most of the significant (not necessarily all very popular) perspectives should be covered regardless of their proportions of popularity in the review corpus, e.g. if 90 reviews claim that the iPhone cost is low, and 10 reviews claim that it is high, both perspectives should be more or less equally reflected in the final ranked list. *Data and Resources* The training, development and test data will be extracted and annotated from Amazon SNAP Review Dataset and will be available on the website according to the given schedule. *Evaluation* Evaluation scripts will be made available on the website. nDCG <https://www.google.com/url?q=https%3A%2F%2Fwww.kaggle.com%2Fwiki%2FNormalizedDiscountedCumulativeGain&sa=D&sntz=1&usg=AFQjCNGoFiB-B1qnByi-o9I8XkwmOEsMaw> (normalized Discounted Cumulative Gain) is tentatively the primary measure of evaluation. Being an introductory task, we will evaluate the system submissions on a wide range of measures for experimental reasons. These secondary evaluations will not reflect in the scoring of participating systems. More details can be found on the website. *Invitation* We invite participation from all researchers and practitioners. The organizers rely, as is usual in shared tasks, on the honesty of all participants who might have some prior knowledge of part of the data that will eventually be used for evaluation, not to unfairly use such knowledge. The only exceptions (to participation) are the members of the organizing team, who cannot submit a system. The organizing chair will serve as an authority to resolve any disputes concerning ethical issues or completeness of system descriptions. *Timeline* Shared Task Website Ready: *May 1, 2017* First Call for Participants Ready: *May 1, 2017* Registration Begins: *May 15, 2017* Release of Training Data: *May 15, 2017* Dryrun: Release of Development Set: *July 20, 2017* Dryrun: Submission on Development Set: *July 26, 2017* Dryrun: Release of Scores: *July 27, 2017* Registration Ends: *August 18, 2017* Release of Test Set: *August 21, 2017* Submission of Systems: *August 28, 2017* System Results: *September 5, 2017* System Description Paper Due: *September 15, 2017* Notification of Acceptance: *September 30, 2017* Camera-Ready Deadline: *October 10, 2017* See http://sites.google.com/revopid-2017 <http://www.google.com/url?q=http%3A%2F%2Fsites.google.com%2Frevopid-2017&sa=D&sntz=1&usg=AFQjCNFY7YT4i5eTwBlrkRn5oRygzv8P7g> for more information.
_______________________________________________ Mt-list site list [email protected] http://lists.eamt.org/mailman/listinfo/mt-list
