Even checkstyle can check for licenses. It can handle non-java files. Since we are using it for enforcing style, we can use it for this as well.
On Wed, Sep 23, 2015 at 4:57 PM, Thomas Weise <[email protected]> wrote: > We should look at hooking up rat, if that's the commonly used checker for > ASF releases. > > Before doing that, we should update all files to the expected header, and > add the headers to the non-java files. > > Chetan, do you know if the license:format can handle the non-java files, > too? > > > > > On Wed, Sep 23, 2015 at 4:33 PM, Chetan Narsude <[email protected]> > wrote: > > > Yes Ram, this is handled by license plugin already. Do you think we > should > > hook up the license:check with the pull request? > > > > -- > > Chetan > > > > > > On Wed, Sep 23, 2015 at 4:17 PM, Munagala Ramanath <[email protected]> > > wrote: > > > > > If we have a solution that we are happy with, there is no need to > resort > > to > > > my script. > > > > > > I wrote the script to address what I thought was a one-time need as > > > described in the message > > > from Thomas. Our files have a lot of quirks in the the way the > copyrights > > > notices are present, > > > for example, I found these variants (among others): > > > > > > /* > > > * Copyright (c) 2012-2013 DataTorrent, Inc. > > > * All Rights Reserved. > > > */ > > > > > > /* > > > * Copyright (c) 2014 DataTorrent, Inc. ALL Rights Reserved. > > > */ > > > > > > /** > > > * Put your copyright and license info here. > > > */ > > > > > > A custom script that recognizes and deals with these variants will > likely > > > do a better job that > > > a standard plugin (for example, we need to preserve the original date, > we > > > cannot change everything > > > to 2015). I'm not suggesting it for regular use, just a one-time > > > conversion. > > > > > > So, to repeat, if we already have a solution we like, please disregard > my > > > script. > > > > > > Ram > > > > > > On Wed, Sep 23, 2015 at 4:06 PM, Chetan Narsude < > [email protected]> > > > wrote: > > > > > > > Ram, > > > > > > > > It does have a licensed header which violates the generally > followed > > > > licensing rules. The first line is blank (there is a issue open to be > > > > lenient in that case as I remember I saw sometime ago). There are a > few > > > > more files with the same viloation. Regardless, I am not saying that > > it's > > > > perfect. But why reinvent the wheel? You can work with the plugin > > writer > > > > like 100 others have done and make it better if you feel that it's > not > > > good > > > > enough. > > > > > > > > A better way to find all the files which violate the formatting is > to > > > > format the files which do not follow the rule, I get the following. > It > > > has > > > > a few more files than you mentioned: > > > > > > > > chetan@chubi:~/work/apex$ mvn license:format -Dlicense.skip=false > > > > [INFO] Scanning for projects... > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Reactor Build Order: > > > > [INFO] > > > > [INFO] Realtime Stream Processing Framework > > > > [INFO] DataTorrent API > > > > [INFO] Base Library > > > > [INFO] Buffer Server > > > > [INFO] Streaming Application Manager > > > > [INFO] Apex Application Maven archetype > > > > [INFO] Apex App Configuration Maven archetype > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Realtime Stream Processing Framework 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ > > dt-framework > > > > --- > > > > [INFO] Updating license headers... > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building DataTorrent API 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ dt-api > --- > > > > [INFO] Updating license headers... > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/api/src/main/java/com/datatorrent/api/annotation/Name.java > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Base Library 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ dt-common > > --- > > > > [INFO] Updating license headers... > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/common/src/main/java/com/datatorrent/common/metric/SingleMetricAggregator.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/common/src/main/java/com/datatorrent/common/metric/sum/LongSumAggregator.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/common/src/main/java/com/datatorrent/common/metric/sum/DoubleSumAggregator.java > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Buffer Server 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ > > > dt-bufferserver > > > > --- > > > > [INFO] Updating license headers... > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/bufferserver/src/main/java/com/datatorrent/bufferserver/util/VarInt.java > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Streaming Application Manager 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ dt-engine > > --- > > > > [INFO] Updating license headers... > > > > [WARNING] Unknown file extension: > > > > /home/chetan/work/apex/engine/src/main/scripts/dtcli > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/plan/logical/StreamCodecWrapperForPersistance.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/engine/Slider.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/stream/PartitionAwareSinkForPersistence.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/webapp/asm/FieldSignatureVisitor.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/webapp/asm/CompactAnnotationNode.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/webapp/asm/CompactFieldNode.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/api/AppDataSource.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/main/java/com/datatorrent/stram/util/PubSubWebSocketServlet.java > > > > [INFO] Updating license header in: > > > > > > > > > > > > > > /home/chetan/work/apex/engine/src/test/java/com/datatorrent/stram/plan/StreamPersistanceTests.java > > > > [WARNING] Unable to find a comment style definition for some files. > You > > > may > > > > want to add a custom mapping for the relevant file extensions. > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Apex Application Maven archetype 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ > > > > apex-app-archetype --- > > > > [INFO] Updating license headers... > > > > [INFO] > > > > > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Building Apex App Configuration Maven archetype 3.2.0-SNAPSHOT > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] > > > > [INFO] --- license-maven-plugin:2.11:format (default-cli) @ > > > > apex-conf-archetype --- > > > > [INFO] Updating license headers... > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Reactor Summary: > > > > [INFO] > > > > [INFO] Realtime Stream Processing Framework ............... SUCCESS [ > > > > 0.889 s] > > > > [INFO] DataTorrent API .................................... SUCCESS [ > > > > 0.162 s] > > > > [INFO] Base Library ....................................... SUCCESS [ > > > > 0.093 s] > > > > [INFO] Buffer Server ...................................... SUCCESS [ > > > > 0.072 s] > > > > [INFO] Streaming Application Manager ...................... SUCCESS [ > > > > 0.408 s] > > > > [INFO] Apex Application Maven archetype ................... SUCCESS [ > > > > 0.233 s] > > > > [INFO] Apex App Configuration Maven archetype ............. SUCCESS [ > > > > 0.013 s] > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] BUILD SUCCESS > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > [INFO] Total time: 2.521 s > > > > [INFO] Finished at: 2015-09-23T16:00:02-07:00 > > > > [INFO] Final Memory: 13M/471M > > > > [INFO] > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2015 at 4:00 PM, Munagala Ramanath < > > [email protected]> > > > > wrote: > > > > > > > > > When I run that command, I get the same error about Name.java but > > that > > > > file > > > > > _does_ have a > > > > > license header. > > > > > > > > > > The files identified by my script as having issues are not shown: > > > > > > > > > > ============================================================ > > > > > 3 Java files with no comments > > > > > 0: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/main/java/com/datatorrent/stram/plan/logical/StreamCodecWrapperForPersistance.java > > > > > 1: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/main/java/com/datatorrent/stram/stream/PartitionAwareSinkForPersistence.java > > > > > 2: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/java/com/datatorrent/stram/plan/StreamPersistanceTests.java > > > > > ============================================================ > > > > > 4 Java files with no matching copyright comment > > > > > 0: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/resources/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/RandomNumberGenerator.java > > > > > 1: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/resources/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/StdoutOperator.java > > > > > 2: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/target/test-classes/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/RandomNumberGenerator.java > > > > > 3: > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/target/test-classes/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/StdoutOperator.java > > > > > > > > > > > > > > > Ram > > > > > > > > > > On Wed, Sep 23, 2015 at 3:46 PM, Thomas Weise < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Ram, > > > > > > > > > > > > You need to run > > > > > > > > > > > > mvn license:check -Dlicense.skip=false > > > > > > > > > > > > I get the following: > > > > > > > > > > > > [INFO] --- license-maven-plugin:2.11:check (default-cli) @ dt-api > > --- > > > > > > [INFO] Checking licenses... > > > > > > [WARNING] Missing header in: > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/hdev/devel/dt/Apex/api/src/main/java/com/datatorrent/api/annotation/Name.java > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2015 at 3:34 PM, Munagala Ramanath < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > It's in the output.txt file uploaded with my earlier message > but > > > here > > > > > is > > > > > > an > > > > > > > extract from the end of it: > > > > > > > > > > > > > > ============================================================ > > > > > > > 3 Java files with no comments > > > > > > > 0: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/main/java/com/datatorrent/stram/plan/logical/StreamCodecWrapperForPersistance.java > > > > > > > 1: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/main/java/com/datatorrent/stram/stream/PartitionAwareSinkForPersistence.java > > > > > > > 2: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/java/com/datatorrent/stram/plan/StreamPersistanceTests.java > > > > > > > ============================================================ > > > > > > > 4 Java files with no matching copyright comment > > > > > > > 0: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/resources/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/RandomNumberGenerator.java > > > > > > > 1: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/src/test/resources/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/StdoutOperator.java > > > > > > > 2: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/target/test-classes/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/RandomNumberGenerator.java > > > > > > > 3: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/ram/src/apex/apex/engine/target/test-classes/testAppPackage/mydtapp/src/main/java/com/example/mydtapp/StdoutOperator.java > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2015 at 2:44 PM, Chetan Narsude < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > which files are those? > > > > > > > > > > > > > > > > -- > > > > > > > > chetan > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2015 at 12:51 PM, Munagala Ramanath < > > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Not sure what it's checking but "*mvn license:check*" shows > > me > > > no > > > > > > list > > > > > > > of > > > > > > > > > files in core. > > > > > > > > > Yet, my script finds at least 3 java files with no comments > > at > > > > all. > > > > > > > > > > > > > > > > > > Ram > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2015 at 11:54 AM, Chetan Narsude < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > mvn license:check > > > > > > > > > > mvn license:format > > > > > > > > > > > > > > > > > > > > At the top level pom does magic for you. > > > > > > > > > > > > > > > > > > > > On Wednesday, September 23, 2015, Munagala Ramanath < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thomas and Chris: > > > > > > > > > > > > > > > > > > > > > > I wrote a Ruby script to automate this process. > > Currently, > > > it > > > > > > makes > > > > > > > > no > > > > > > > > > > > changes and just > > > > > > > > > > > reports what it finds. The script is attached. > > > > > > > > > > > > > > > > > > > > > > I ran it like this: ruby -w fix-license.rb -f > > > > ~/src/apex/apex > > > > > > > > > > > output.txt > > > > > > > > > > > The output is also attached. > > > > > > > > > > > > > > > > > > > > > > Let me know if you'd like me to enhance this to > actually > > > add > > > > > the > > > > > > > new > > > > > > > > > > > copyright notice to the top of > > > > > > > > > > > the files where it does detect some form of copyright > > > notice > > > > > > (there > > > > > > > > > will > > > > > > > > > > > still be a few files where > > > > > > > > > > > no such notice was found and those will need manual > > > > > > augmentation). > > > > > > > > > > > > > > > > > > > > > > Ram > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2015 at 9:30 AM, Chris Nauroth < > > > > > > > > > [email protected] > > > > > > > > > > > <javascript:_e(%7B%7D,'cvml','[email protected] > > > ');>> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Hi Thomas, > > > > > > > > > > >> > > > > > > > > > > >> I think you are correct. Here is the official Apache > > > > > statement > > > > > > on > > > > > > > > > what > > > > > > > > > > to > > > > > > > > > > >> do with the source headers: > > > > > > > > > > >> > > > > > > > > > > >> http://www.apache.org/legal/src-headers.html > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> I think your result (the second example below) matches > > up > > > > with > > > > > > > that. > > > > > > > > > > >> > > > > > > > > > > >> As a reminder, I have a work-in-progress patch for the > > > > > non-Java > > > > > > > > files > > > > > > > > > in > > > > > > > > > > >> Apex here: > > > > > > > > > > >> > > > > > > > > > > >> https://issues.apache.org/jira/browse/APEXCORE-1 > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> It would be nice if we could come up with a script to > > chop > > > > out > > > > > > the > > > > > > > > > > >> DataTorrent header and replace it with the ASF header. > > > > Then, > > > > > we > > > > > > > > could > > > > > > > > > > >> simply run that across *.java in both Apex and Malhar. > > > > > > > > > > >> > > > > > > > > > > >> --Chris Nauroth > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> On 9/22/15, 9:24 AM, "Thomas Weise" < > > > [email protected] > > > > > > > > > > >> <javascript:_e(%7B%7D,'cvml','[email protected] > > ');>> > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >> >Hit the wrong button... > > > > > > > > > > >> > > > > > > > > > > > >> >This question is about the license header for the > > source > > > > > files. > > > > > > > > > > Currently > > > > > > > > > > >> >we have: > > > > > > > > > > >> > > > > > > > > > > > >> >/** > > > > > > > > > > >> > * Copyright (C) 2015 DataTorrent, Inc. > > > > > > > > > > >> > * > > > > > > > > > > >> > * Licensed under the Apache License, Version 2.0 > (the > > > > > > > "License"); > > > > > > > > > > >> > * you may not use this file except in compliance > with > > > the > > > > > > > License. > > > > > > > > > > >> > * You may obtain a copy of the License at > > > > > > > > > > >> > * > > > > > > > > > > >> > * > http://www.apache.org/licenses/LICENSE-2.0 > > > > > > > > > > >> > * > > > > > > > > > > >> > * Unless required by applicable law or agreed to in > > > > writing, > > > > > > > > > software > > > > > > > > > > >> > * distributed under the License is distributed on an > > "AS > > > > IS" > > > > > > > > BASIS, > > > > > > > > > > >> > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, > either > > > > > express > > > > > > > or > > > > > > > > > > >> >implied. > > > > > > > > > > >> > * See the License for the specific language > governing > > > > > > > permissions > > > > > > > > > and > > > > > > > > > > >> > * limitations under the License. > > > > > > > > > > >> > */ > > > > > > > > > > >> > > > > > > > > > > > >> >The first line needs to go. And a reference to NOTICE > > > needs > > > > > to > > > > > > be > > > > > > > > > added > > > > > > > > > > >> as > > > > > > > > > > >> >shown below? > > > > > > > > > > >> > > > > > > > > > > > >> >/* > > > > > > > > > > >> > * Licensed to the Apache Software Foundation (ASF) > > under > > > > one > > > > > > or > > > > > > > > more > > > > > > > > > > >> > * contributor license agreements. See the NOTICE > file > > > > > > > distributed > > > > > > > > > > with > > > > > > > > > > >> > * this work for additional information regarding > > > copyright > > > > > > > > > ownership. > > > > > > > > > > >> > * The ASF licenses this file to You under the Apache > > > > > License, > > > > > > > > > Version > > > > > > > > > > >> 2.0 > > > > > > > > > > >> > * (the "License"); you may not use this file except > in > > > > > > > compliance > > > > > > > > > with > > > > > > > > > > >> > * the License. You may obtain a copy of the License > > at > > > > > > > > > > >> > * > > > > > > > > > > >> > * http://www.apache.org/licenses/LICENSE-2.0 > > > > > > > > > > >> > * > > > > > > > > > > >> > * Unless required by applicable law or agreed to in > > > > writing, > > > > > > > > > software > > > > > > > > > > >> > * distributed under the License is distributed on an > > "AS > > > > IS" > > > > > > > > BASIS, > > > > > > > > > > >> > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, > either > > > > > express > > > > > > > or > > > > > > > > > > >> >implied. > > > > > > > > > > >> > * See the License for the specific language > governing > > > > > > > permissions > > > > > > > > > and > > > > > > > > > > >> > * limitations under the License. > > > > > > > > > > >> > */ > > > > > > > > > > >> > > > > > > > > > > > >> >On Tue, Sep 22, 2015 at 9:19 AM, Thomas Weise < > > > > > > > > > [email protected] > > > > > > > > > > >> <javascript:_e(%7B%7D,'cvml','[email protected] > > ');>> > > > > > > > > > > >> >wrote: > > > > > > > > > > >> > > > > > > > > > > > >> >> There > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
