Can you take a look at the imports? There seems to be many class names? I also think you should try to shorten the package names. Maybe names like oac.comet (org.apache.catalina.comet) would already help.
And we really need visualize the dependencies ... This is however a great start. Kind regards, Peter Kriens On 30 jul 2011, at 16:02, Tiger Gui wrote: > Hi Peter, > > I changed the merge situation a little, > > FROM: > boolean condition4 = 4 * sameUB >= one.usedByList.size() + > two.usedByList.size(); > TO: > boolean condition4 = 3 * sameUB >= one.usedByList.size() + > two.usedByList.size(); > > Try the analyse and split algorithm in Tomcat application again, and > we got another report, in the attach file. It seems that this bundle > structure is more reasonable > > 2011/7/30 Tiger Gui <tigergui1...@gmail.com>: >> Hi Peter, >> >> I have generated the report for Tomcat too, in the attach file, and >> improve the report as you wish, just see the comments >> >> 2011/7/29 Peter Kriens <peter.kri...@aqute.biz>: >>> Now we're getting somewhere! Could you: >>> >>> 1) Please rename names like round1MergeBundle8 to just R18 or so ... >> >> I have renamed round1MergeBundle8 to just R1B8, BTW, you should know >> that users can change the bundle name as he wish in our tools, >> OSGiMaker has supplies the feature already, we have a GUI bundle edit >> editor >> >>> 2) The imports look like they sometimes contain classes? And it would be >>> nice if you put a </br> after each one of them (maybe combine the ones with >>> the same prefix?) >> >> No, the imports are all package, i add a "<br />" every 5 imports, if >> we add a "<br/>" after each of them, it will be a long list, not very >> comfortable to read >> >>> 3) Can you sort on the number of contains entries? I.e. biggest bundle first >> >> Yeah, i have sorted it already, the biggest first >> >>> 4) Could you link the bundles with an <a href="..">R18</a>? >> >> Yeah, i have add the http link already in the report >> >>> 5) Could you make the names of the >> >> ??? >> >> >>> 6) Get rid of the .*? >> >> It is easy to get rid of the .*, but i think it make things more >> clear, make us know, the bundles includes all the classes in the >> packages >> >>> 7) Also generate one of these for catalina/Tomcat? >>> >>> Actually, this spring one is pretty neat. >>> >>> I am a bit curious about the jdbc stuff. I assume all the packages in >>> org.springframework.jdbc11 are strongly connected? What are the packages >>> that refer to com.ibm.websphere.rsadapter and com.mchange.v2.c3p0, etc? >>> These are the VERY expensive dependencies and I wonder how they get merged >>> in so badly in this bundle. >>> >>> Similar to the orm bundle, who is the culprit (class?) that points to >>> hibernate? You see why this is bad? In a good design the >>> org.springframework.orm should not be strongly connected to something like >>> hibernate. The hibernate should be a bridge package but not have internal >>> references. >> >>> >>> Interesting! Kind regards, >>> >>> Peter Kriens >>> >>> >>> >>> >>> On 29 jul 2011, at 11:32, Tiger Gui wrote: >>> >>>> Hi Peter, >>>> >>>> To visualize things, i have a simple review to JUNG, it seems that it >>>> is good solution for display graph or tree in a canvas, but it is a >>>> problem that JUNG is developed for AWT/SWING, not suitable for Eclipse >>>> SWT/JFace environment, so, i will continue searching for the best >>>> solution, may be we can even consider Eclipse GMF technology >>>> >>>> >>>> To report things: >>>> I have changed the report to a HTML file in the attach file, it's >>>> still Spring's OSGi split report, it looks much better :-) >>>> >>>> 2011/7/29 Peter Kriens <peter.kri...@aqute.biz>: >>>>> Rule #1, as long as I see you're engaged you do not have to worry about >>>>> the evaluation. So do not do things for my sake, do them because you >>>>> believe it is the best way forward. I think Tinkerpop and JUNG are a >>>>> great way to visualize, but do not take my word. I have the greatest >>>>> respect for people who do not listen to me because they are convinced >>>>> they know better. That out of the way. >>>>> >>>>> This whole things is very much about understanding and our visual brains >>>>> is by far the most powerful tool on earth to understand things. And >>>>> you're not taking advantage of it :-) We are looking for structure here, >>>>> the names are only relevant later. So to see structure, tables are >>>>> perfect. Which means you have to have short names. Just using the last >>>>> part of a package (com.springframework.jca, take jca). I.e. a name like >>>>> round1MergeBundle8 is ok to call b8. You can always later provide a list >>>>> of translations. This is also the reason why we need to use packages, >>>>> classes are too many off. Visualization is everything in this phase >>>>> because we need to find the rules of the structure. >>>>> >>>>> So lets retry. First give me a list of all groups (named g1 .. gn) and >>>>> their package content >>>>> >>>>> >>>>> Group Contains Uses Used By >>>>> External >>>>> g3 com.springframework.{jca,jdbc,...}, g4, g5, g7 g2 >>>>> org.w3.xml, javax.persistence, >>>>> ... >>>>> com.oracle.g11.{impl,whatever} >>>>> >>>>> It might be easiest to generate HTML so you can use tables. Don't make it >>>>> look fancy, just use the tools to get insight in the structure: >>>>> >>>>> <html> >>>>> <body> >>>>> <table> >>>>> <tr> >>>>> <th>Group</th> >>>>> <th>Contains</th> >>>>> <th>Uses</th> >>>>> <th>Used By</th> >>>>> <th>External</th> >>>>> </tr> >>>>> <tr> >>>>> <td>g3</td> >>>>> <td>com.springframework.{jca,jdbc,...},...</td> >>>>> <td>g4, g5, g7</td> >>>>> <td>g2</td> >>>>> <td>org.w3.xml, >>>>> javax.persistence,com.oracle.g11.{impl,whatever}</td> >>>>> </tr> >>>>> </table> >>>>> </body> >>>>> </html> >>>>> >>>>> Shortening the names is everything or else your eyes drown in a sea of >>>>> characters. >>>>> >>>>> Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>> On 29 jul 2011, at 05:16, Tiger Gui wrote: >>>>> >>>>>> Hi Peter, >>>>>> >>>>>> Thank you for your reply first, please see the following comments >>>>>> >>>>>> 2011/7/28 Peter Kriens <peter.kri...@aqute.biz>: >>>>>>> It is good that you use a real project now. What I like to see how many >>>>>>> "bundles" we have after the first round, where you group strongly >>>>>>> connected packages. This should already simplify because the nr of >>>>>>> entities will be smaller. >>>>>>> >>>>>>> I need to see a real case. Step 1 is ok, but maybe you can add merging >>>>>>> of bundles that have identical usesExternal (imports). >>>>>> >>>>>> Yes, I do merge the bundles that have identical usesExternal >>>>>> (imports), i described in the former mail, >>>>>> >>>>>> We define sameUE: it menas the number of two bundles have same >>>>>> usedExternal external package rely elements. >>>>>> >>>>>> boolean condition2 = 3 * sameUE >= one.usedExternalList.size() + >>>>>> two.usedExternalList.size(); >>>>>> >>>>>> if(condition2 == true) i will merge bundles one and two,In the process >>>>>> of real case test(Spring and Tomcat), i found that it's hardly to find >>>>>> two bundles which have exactly the same usesExternal items, we should >>>>>> also merge bundles who have *proportional* usesExternal items. In my >>>>>> test case, it works good. >>>>>> >>>>>>> And make sure you ignore all java.* to make the info smaller (in OSGi, >>>>>>> java is always provided by the VM). Can you just print out something >>>>>>> like >>>>>> >>>>>> Yes, i have considered this situation, in the whole merge process, we >>>>>> do not consider jdk supplies java.* classes. >>>>>> >>>>>>> >>>>>>> name usesInternal usesExternal >>>>>>> g1 g2 a,b,c >>>>>>> g2 d,e,f >>>>>>> >>>>>> >>>>>> Let's take example of Spring, this report is like this: >>>>>> Name --- >>>>>> usesInternal >>>>>> org.springframework.jms2 >>>>>> ---round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5 >>>>>> org.springframework.jdbc11 --- >>>>>> round1MergeBundle8,mergedBundle9 >>>>>> org.springframework.orm13 >>>>>> --- >>>>>> round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3 >>>>>> org.springframework.context14 >>>>>> --- round1MergeBundle8,round1MergeBundle1,mergedBundle9 >>>>>> org.springframework.scheduling15 >>>>>> --- >>>>>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11 >>>>>> org.springframework.jca22 >>>>>> --- >>>>>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15 >>>>>> round1MergeBundle1 >>>>>> --- >>>>>> round1MergeBundle8,mergedBundle9,org.springframework.metadata7 >>>>>> round1MergeBundle3 >>>>>> --- >>>>>> round1MergeBundle8,org.springframework.context14,round1MergeBundle5 >>>>>> round1MergeBundle4 --- >>>>>> round1MergeBundle8,org.springframework.context14 >>>>>> round1MergeBundle5 >>>>>> --- >>>>>> round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9 >>>>>> mergedBundle9 >>>>>> --- >>>>>> round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7 >>>>>> org.springframework.metadata7 --- none >>>>>> round1MergeBundle8 --- none >>>>>> >>>>>> As usesExternal element list is too long, i abridged them, you can get >>>>>> the report details with usesExternal elements in attach file >>>>>> "bundles_relation.txt", and get the bundles details in attach file >>>>>> "SpringSplitTest.analyse" >>>>>> >>>>>> >>>>>>> I hope we see one big bundle which is the core and then have to find >>>>>>> rules to classify the remaining bundles. I expect there are the >>>>>>> following categories: >>>>>>> >>>>>>> core implementation classes, lots of strongly connected >>>>>>> packages >>>>>>> api api classes, do not refer to core, very few imports >>>>>>> bridge refer strongly to core and have expensive imports >>>>>>> >>>>>>> >>>>>>> At this stage, the trick is to do some work by hand until you find you >>>>>>> really understand the problem. >>>>>>> >>>>>>> It would be perfect if you could take a look at Tinkerpop and JUNG. I >>>>>>> think it would be quite easy to visualize the graph of dependencies. >>>>>> >>>>>> You mean my next step is developing a Tinkerpop or JUNG graphical view >>>>>> for this report for user to adjust the bundles details manully ? Am i >>>>>> right ? I will start learn Jung and start this job soon >>>>>> >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 26 jul 2011, at 16:24, Tiger Gui wrote: >>>>>>> >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> This is the whole application split algorithm here. After application >>>>>>>> source code analyse algorithm described here[1], we can know each >>>>>>>> package/class use which packages/classes and be used by which >>>>>>>> packages/classes. Now, we just discuss package here, we treat package >>>>>>>> as a single atom, each package has three important attributes, usedBy, >>>>>>>> usesExternal and usesInternal, just like below: >>>>>>>> >>>>>>>> <package name="org.apache.catalina.deploy" >>>>>>>> sources="/E:/GSoC/gsoc2011osgi/runtime-New_configuration/TomcatJava/bin" >>>>>>>> size="30" usedBy="8" usesInternal="11" usesExternal="14" layer="6" >>>>>>>> cycle="org.apache.catalina et al."> >>>>>>>> <packageRef name="org.apache.catalina.core" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.startup" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.deploy" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.authenticator" >>>>>>>> type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.mbeans" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.connector" type="usedBy"/> >>>>>>>> <packageRef name="org.apache.catalina.realm" type="usedBy"/> >>>>>>>> <packageRef name="java.lang" type="usesExternal"/> >>>>>>>> <packageRef name="java.io" type="usesExternal"/> >>>>>>>> <packageRef name="org.apache.catalina.deploy" >>>>>>>> type="usesInternal"/> >>>>>>>> <packageRef name="org.apache.catalina.util" type="usesInternal"/> >>>>>>>> <packageRef name="java.util" type="usesExternal"/> >>>>>>>> <packageRef name="org.apache.juli.logging" type="usesInternal"/> >>>>>>>> <packageRef name="org.apache.tomcat.util.res" >>>>>>>> type="usesInternal"/> >>>>>>>> <packageRef name="java.beans" type="usesExternal"/> >>>>>>>> <packageRef name="(default package)" type="usesExternal"/> >>>>>>>> <packageRef name="org.apache.catalina" type="usesInternal"/> >>>>>>>> <packageRef name="org.apache.catalina.mbeans" >>>>>>>> type="usesInternal"/> >>>>>>>> <packageRef name="javax.management" type="usesExternal"/> >>>>>>>> <packageRef name="namingResources" type="usesExternal"/> >>>>>>>> <packageRef name="javax.naming" type="usesExternal"/> >>>>>>>> <packageRef name="org.apache.naming" type="usesInternal"/> >>>>>>>> <packageRef name="java.lang.reflect" type="usesExternal"/> >>>>>>>> <packageRef name="javax.servlet" type="usesInternal"/> >>>>>>>> <packageRef name="javax.servlet.annotation" type="usesInternal"/> >>>>>>>> <packageRef name="org.apache.catalina.order" type="usesExternal"/> >>>>>>>> <packageRef name="java.net" type="usesExternal"/> >>>>>>>> <packageRef name="webXml" type="usesExternal"/> >>>>>>>> <packageRef name="webXml.version" type="usesExternal"/> >>>>>>>> <packageRef name="webxml" type="usesExternal"/> >>>>>>>> <packageRef name="org.apache.catalina.core" type="usesInternal"/> >>>>>>>> <packageRef name="javax.servlet.descriptor" type="usesInternal"/> >>>>>>>> </package> >>>>>>>> >>>>>>>> usedBy means who use current package; >>>>>>>> usesInternal means which packages current package use in current >>>>>>>> application source code; >>>>>>>> usesExternal means which packages current pakcage use not in current >>>>>>>> application's source code. >>>>>>>> >>>>>>>> Now, we start split the whole application in to bundles, according to >>>>>>>> above algorithm source code analyse algorithm, we can also know >>>>>>>> package cycles in current application. >>>>>>>> >>>>>>>> 1. Treat each package cycle as a single bundle; >>>>>>>> 2. Treat each packages not in cycle as a single bundle; >>>>>>>> 3. Then we should decide which bundles can be merged into one new >>>>>>>> bundle; >>>>>>>> >>>>>>>> First round merge job: >>>>>>>> 4. If one bundle's all usesInternal elements are in the other bundle, >>>>>>>> these two bundles should be merged into a new bundle; >>>>>>>> >>>>>>>> Think about how to merge used by only bundle (bundle be used only by >>>>>>>> other bundle, it does not rely on any other bundle): >>>>>>>> Define two variable for used by only bundle: >>>>>>>> sameUB: it means the number of two bundles have same usedBy elements. >>>>>>>> sameUE: it menas the number of two bundles have same usedExternal >>>>>>>> external package rely elements. >>>>>>>> >>>>>>>> boolean condition1 = 4 * sameUB >= one.usedByList.size() + >>>>>>>> two.usedByList.size(); >>>>>>>> boolean condition2 = 2 * sameUE >= one.usedExternalList.size() + >>>>>>>> two.usedExternalList.size(); >>>>>>>> >>>>>>>> if condition1 or condition2 is true, we should merge these two usedBy >>>>>>>> only bundle. >>>>>>>> >>>>>>>> Merge the other bundles: >>>>>>>> 5.The core problem is how to decide two bundles(bundle one and bundle >>>>>>>> two) can be merged or not. >>>>>>>> >>>>>>>> Define 5 variables: >>>>>>>> uiNumber: the sum of bundle one's usesInternal elements in bundle two >>>>>>>> number and bundle two's usesInternal elements in bundle one number; >>>>>>>> >>>>>>>> ubNumber:the sum of bundle one's usedBy elements in bundle two number >>>>>>>> and bundle two's usedBy elements in bundle one number; >>>>>>>> >>>>>>>> sameUI: the same usesInternal number bundle one and two have >>>>>>>> >>>>>>>> sameUB: Be similar with used by only bundle's this variable >>>>>>>> >>>>>>>> sameUE: Be similar with used by only bundle's this variable >>>>>>>> >>>>>>>> Define these conditions: >>>>>>>> boolean condition1 = 2 * uiNumber >= one.usesInternalList.size() + >>>>>>>> two.usesInternalList.size(); >>>>>>>> boolean condition2 = 2 * ubNumber >= one.usedByList.size() + >>>>>>>> two.usedByList.size() ; >>>>>>>> boolean condition3 = 3.5 * sameUI >= one.usesInternalList.size() + >>>>>>>> two.usesInternalList.size(); >>>>>>>> boolean condition4 = 4 * sameUB >= one.usedByList.size() + >>>>>>>> two.usedByList.size(); >>>>>>>> boolean condition5 = 3 * sameUE >= one.usedExternalList.size() + >>>>>>>> two.usedExternalList.size(); >>>>>>>> >>>>>>>> If any above condition is true, these two bundles can be merged. But >>>>>>>> these are another problem, if bundle A can be merged with B, but it >>>>>>>> also can be merged with C, now, we should decide merge A with B or A >>>>>>>> with C. >>>>>>>> >>>>>>>> Define the follow attribute: >>>>>>>> int mergeFactor= 2 * (uiNumber + ubNumber) + 0.4 * (sameUI + sameUB) + >>>>>>>> 0.2 * sameUE - number; >>>>>>>> >>>>>>>> If A and B's mergeFactor is 20 and A and C's mergeFactor is 30, we >>>>>>>> should merge A and C. >>>>>>>> >>>>>>>> This is current merge algorithm in OSGiMaker, i will keep on improving >>>>>>>> it, use class relationship factors or etc. You can find the source >>>>>>>> code detail of this algorithm in class AnalyseJob of our project. >>>>>>>> >>>>>>>> The attach file is the analyse result document OSGiMaker analyse >>>>>>>> Tomcat's source code and split it into bundles, you can have a review. >>>>>>>> >>>>>>>> >>>>>>>> In fact, i did not want to bother you too much, but it seems that you >>>>>>>> have enough time to help me to improve it, this is a good thing. If >>>>>>>> you have any advises, please let me know, let's improving it together >>>>>>>> :-) >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>>> [1] >>>>>>>> http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm >>>>>>>> >>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>: >>>>>>>>> Well, I do not think I am eager but I have a hard time getting a >>>>>>>>> feeling where you are. You do not have to send reports, I want to see >>>>>>>>> intermediate results and discuss issues if there are any. As I said >>>>>>>>> earlier, it is not clear to me yet what the best algorithm is so that >>>>>>>>> need to be worked out before we do the gui stuff. >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> Peter Kriens >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 26 jul 2011, at 11:24, Tiger Gui wrote: >>>>>>>>> >>>>>>>>>> In my schedule, i will report current status to you tomorrow as i >>>>>>>>>> think i can get a usable version today, the whole analyse and split >>>>>>>>>> algorithm is complex i have to organize a document to describe it >>>>>>>>>> clearly. As you are really eager to see its progress, it is OK, i >>>>>>>>>> will >>>>>>>>>> start the report now >>>>>>>>>> >>>>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>: >>>>>>>>>>> If that is the case you have to provide more on going feedback. >>>>>>>>>>> What happened to the analysis by hand? >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> >>>>>>>>>>> Peter Kriens >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 26 jul 2011, at 11:02, Tiger Gui wrote: >>>>>>>>>>> >>>>>>>>>>>> No, Peter, i am really working hard for this project, you can check >>>>>>>>>>>> the progress here[1]. Now, this tool can analyse a whole project >>>>>>>>>>>> and >>>>>>>>>>>> export the its analyse result to bundles( I test it in Spring and >>>>>>>>>>>> Tomcat project), if possible, you can install it in your Eclipse >>>>>>>>>>>> and >>>>>>>>>>>> have a trial of it. But, i am still improving the split algorithm >>>>>>>>>>>> as >>>>>>>>>>>> the current algorithm is not working perfect. >>>>>>>>>>>> >>>>>>>>>>>> I will supply a document about the current status of this project >>>>>>>>>>>> and >>>>>>>>>>>> a simple guide for you to have a trial of it. I am really working >>>>>>>>>>>> very >>>>>>>>>>>> hard for it these days :-( >>>>>>>>>>>> >>>>>>>>>>>> [1] http://code.google.com/p/osgimaker/updates/list >>>>>>>>>>>> >>>>>>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>: >>>>>>>>>>>>> I am getting the feeling that you're not working very hard on >>>>>>>>>>>>> this project and only does something just for the evaluations ... >>>>>>>>>>>>> >>>>>>>>>>>>> Peter Kriens >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best Regards >>>>>>>>>>>> ---------------------------------------------------- >>>>>>>>>>>> Tiger Gui [tigergui1...@gmail.com] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards >>>>>>>>>> ---------------------------------------------------- >>>>>>>>>> Tiger Gui [tigergui1...@gmail.com] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards >>>>>>>> ---------------------------------------------------- >>>>>>>> Tiger Gui [tigergui1...@gmail.com] >>>>>>>> <TomcatJava.analyse> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards >>>>>> ---------------------------------------------------- >>>>>> Tiger Gui [tigergui1...@gmail.com] >>>>>> <bundles_relation.txt><SpringSplitTest.analyse> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards >>>> ---------------------------------------------------- >>>> Tiger Gui [tigergui1...@gmail.com] >>>> <SpringSplitTest.html> >>> >>> >> >> >> >> -- >> Best Regards >> ---------------------------------------------------- >> Tiger Gui [tigergui1...@gmail.com] >> > > > > -- > Best Regards > ---------------------------------------------------- > Tiger Gui [tigergui1...@gmail.com] > <TomcatJava.html>