Hi Peter,
I changed the merge situation a little,
FROM:
boolean condition4 = 4 * sameUB >= one.usedByList.size() +
two.usedByList.size();
TO:
boolean condition4 = 3 * sameUB >= one.usedByList.size() +
two.usedByList.size();
Try the analyse and split algorithm in Tomcat application again, and
we got another report, in the attach file. It seems that this bundle
structure is more reasonable
2011/7/30 Tiger Gui <tigergui1...@gmail.com>:
> Hi Peter,
>
> I have generated the report for Tomcat too, in the attach file, and
> improve the report as you wish, just see the comments
>
> 2011/7/29 Peter Kriens <peter.kri...@aqute.biz>:
>> Now we're getting somewhere! Could you:
>>
>> 1) Please rename names like round1MergeBundle8 to just R18 or so ...
>
> I have renamed round1MergeBundle8 to just R1B8, BTW, you should know
> that users can change the bundle name as he wish in our tools,
> OSGiMaker has supplies the feature already, we have a GUI bundle edit
> editor
>
>> 2) The imports look like they sometimes contain classes? And it would be
>> nice if you put a </br> after each one of them (maybe combine the ones with
>> the same prefix?)
>
> No, the imports are all package, i add a "<br />" every 5 imports, if
> we add a "<br/>" after each of them, it will be a long list, not very
> comfortable to read
>
>> 3) Can you sort on the number of contains entries? I.e. biggest bundle first
>
> Yeah, i have sorted it already, the biggest first
>
>> 4) Could you link the bundles with an <a href="..">R18</a>?
>
> Yeah, i have add the http link already in the report
>
>> 5) Could you make the names of the
>
> ???
>
>
>> 6) Get rid of the .*?
>
> It is easy to get rid of the .*, but i think it make things more
> clear, make us know, the bundles includes all the classes in the
> packages
>
>> 7) Also generate one of these for catalina/Tomcat?
>>
>> Actually, this spring one is pretty neat.
>>
>> I am a bit curious about the jdbc stuff. I assume all the packages in
>> org.springframework.jdbc11 are strongly connected? What are the packages
>> that refer to com.ibm.websphere.rsadapter and com.mchange.v2.c3p0, etc?
>> These are the VERY expensive dependencies and I wonder how they get merged
>> in so badly in this bundle.
>>
>> Similar to the orm bundle, who is the culprit (class?) that points to
>> hibernate? You see why this is bad? In a good design the
>> org.springframework.orm should not be strongly connected to something like
>> hibernate. The hibernate should be a bridge package but not have internal
>> references.
>
>>
>> Interesting! Kind regards,
>>
>> Peter Kriens
>>
>>
>>
>>
>> On 29 jul 2011, at 11:32, Tiger Gui wrote:
>>
>>> Hi Peter,
>>>
>>> To visualize things, i have a simple review to JUNG, it seems that it
>>> is good solution for display graph or tree in a canvas, but it is a
>>> problem that JUNG is developed for AWT/SWING, not suitable for Eclipse
>>> SWT/JFace environment, so, i will continue searching for the best
>>> solution, may be we can even consider Eclipse GMF technology
>>>
>>>
>>> To report things:
>>> I have changed the report to a HTML file in the attach file, it's
>>> still Spring's OSGi split report, it looks much better :-)
>>>
>>> 2011/7/29 Peter Kriens <peter.kri...@aqute.biz>:
>>>> Rule #1, as long as I see you're engaged you do not have to worry about
>>>> the evaluation. So do not do things for my sake, do them because you
>>>> believe it is the best way forward. I think Tinkerpop and JUNG are a great
>>>> way to visualize, but do not take my word. I have the greatest respect for
>>>> people who do not listen to me because they are convinced they know
>>>> better. That out of the way.
>>>>
>>>> This whole things is very much about understanding and our visual brains
>>>> is by far the most powerful tool on earth to understand things. And you're
>>>> not taking advantage of it :-) We are looking for structure here, the
>>>> names are only relevant later. So to see structure, tables are perfect.
>>>> Which means you have to have short names. Just using the last part of a
>>>> package (com.springframework.jca, take jca). I.e. a name like
>>>> round1MergeBundle8 is ok to call b8. You can always later provide a list
>>>> of translations. This is also the reason why we need to use packages,
>>>> classes are too many off. Visualization is everything in this phase
>>>> because we need to find the rules of the structure.
>>>>
>>>> So lets retry. First give me a list of all groups (named g1 .. gn) and
>>>> their package content
>>>>
>>>>
>>>> Group Contains Uses Used By
>>>> External
>>>> g3 com.springframework.{jca,jdbc,...}, g4, g5, g7 g2
>>>> org.w3.xml, javax.persistence,
>>>> ...
>>>> com.oracle.g11.{impl,whatever}
>>>>
>>>> It might be easiest to generate HTML so you can use tables. Don't make it
>>>> look fancy, just use the tools to get insight in the structure:
>>>>
>>>> <html>
>>>> <body>
>>>> <table>
>>>> <tr>
>>>> <th>Group</th>
>>>> <th>Contains</th>
>>>> <th>Uses</th>
>>>> <th>Used By</th>
>>>> <th>External</th>
>>>> </tr>
>>>> <tr>
>>>> <td>g3</td>
>>>> <td>com.springframework.{jca,jdbc,...},...</td>
>>>> <td>g4, g5, g7</td>
>>>> <td>g2</td>
>>>> <td>org.w3.xml,
>>>> javax.persistence,com.oracle.g11.{impl,whatever}</td>
>>>> </tr>
>>>> </table>
>>>> </body>
>>>> </html>
>>>>
>>>> Shortening the names is everything or else your eyes drown in a sea of
>>>> characters.
>>>>
>>>> Kind regards,
>>>>
>>>> Peter Kriens
>>>>
>>>>
>>>>
>>>> On 29 jul 2011, at 05:16, Tiger Gui wrote:
>>>>
>>>>> Hi Peter,
>>>>>
>>>>> Thank you for your reply first, please see the following comments
>>>>>
>>>>> 2011/7/28 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>> It is good that you use a real project now. What I like to see how many
>>>>>> "bundles" we have after the first round, where you group strongly
>>>>>> connected packages. This should already simplify because the nr of
>>>>>> entities will be smaller.
>>>>>>
>>>>>> I need to see a real case. Step 1 is ok, but maybe you can add merging
>>>>>> of bundles that have identical usesExternal (imports).
>>>>>
>>>>> Yes, I do merge the bundles that have identical usesExternal
>>>>> (imports), i described in the former mail,
>>>>>
>>>>> We define sameUE: it menas the number of two bundles have same
>>>>> usedExternal external package rely elements.
>>>>>
>>>>> boolean condition2 = 3 * sameUE >= one.usedExternalList.size() +
>>>>> two.usedExternalList.size();
>>>>>
>>>>> if(condition2 == true) i will merge bundles one and two,In the process
>>>>> of real case test(Spring and Tomcat), i found that it's hardly to find
>>>>> two bundles which have exactly the same usesExternal items, we should
>>>>> also merge bundles who have *proportional* usesExternal items. In my
>>>>> test case, it works good.
>>>>>
>>>>>> And make sure you ignore all java.* to make the info smaller (in OSGi,
>>>>>> java is always provided by the VM). Can you just print out something like
>>>>>
>>>>> Yes, i have considered this situation, in the whole merge process, we
>>>>> do not consider jdk supplies java.* classes.
>>>>>
>>>>>>
>>>>>> name usesInternal usesExternal
>>>>>> g1 g2 a,b,c
>>>>>> g2 d,e,f
>>>>>>
>>>>>
>>>>> Let's take example of Spring, this report is like this:
>>>>> Name ---
>>>>> usesInternal
>>>>> org.springframework.jms2
>>>>> ---round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5
>>>>> org.springframework.jdbc11 ---
>>>>> round1MergeBundle8,mergedBundle9
>>>>> org.springframework.orm13
>>>>> ---
>>>>> round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3
>>>>> org.springframework.context14
>>>>> --- round1MergeBundle8,round1MergeBundle1,mergedBundle9
>>>>> org.springframework.scheduling15
>>>>> ---
>>>>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11
>>>>> org.springframework.jca22
>>>>> ---
>>>>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15
>>>>> round1MergeBundle1
>>>>> ---
>>>>> round1MergeBundle8,mergedBundle9,org.springframework.metadata7
>>>>> round1MergeBundle3
>>>>> ---
>>>>> round1MergeBundle8,org.springframework.context14,round1MergeBundle5
>>>>> round1MergeBundle4 ---
>>>>> round1MergeBundle8,org.springframework.context14
>>>>> round1MergeBundle5
>>>>> ---
>>>>> round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9
>>>>> mergedBundle9
>>>>> ---
>>>>> round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7
>>>>> org.springframework.metadata7 --- none
>>>>> round1MergeBundle8 --- none
>>>>>
>>>>> As usesExternal element list is too long, i abridged them, you can get
>>>>> the report details with usesExternal elements in attach file
>>>>> "bundles_relation.txt", and get the bundles details in attach file
>>>>> "SpringSplitTest.analyse"
>>>>>
>>>>>
>>>>>> I hope we see one big bundle which is the core and then have to find
>>>>>> rules to classify the remaining bundles. I expect there are the
>>>>>> following categories:
>>>>>>
>>>>>> core implementation classes, lots of strongly connected
>>>>>> packages
>>>>>> api api classes, do not refer to core, very few imports
>>>>>> bridge refer strongly to core and have expensive imports
>>>>>>
>>>>>>
>>>>>> At this stage, the trick is to do some work by hand until you find you
>>>>>> really understand the problem.
>>>>>>
>>>>>> It would be perfect if you could take a look at Tinkerpop and JUNG. I
>>>>>> think it would be quite easy to visualize the graph of dependencies.
>>>>>
>>>>> You mean my next step is developing a Tinkerpop or JUNG graphical view
>>>>> for this report for user to adjust the bundles details manully ? Am i
>>>>> right ? I will start learn Jung and start this job soon
>>>>>
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> Peter Kriens
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 26 jul 2011, at 16:24, Tiger Gui wrote:
>>>>>>
>>>>>>> Hi Peter,
>>>>>>>
>>>>>>> This is the whole application split algorithm here. After application
>>>>>>> source code analyse algorithm described here[1], we can know each
>>>>>>> package/class use which packages/classes and be used by which
>>>>>>> packages/classes. Now, we just discuss package here, we treat package
>>>>>>> as a single atom, each package has three important attributes, usedBy,
>>>>>>> usesExternal and usesInternal, just like below:
>>>>>>>
>>>>>>> <package name="org.apache.catalina.deploy"
>>>>>>> sources="/E:/GSoC/gsoc2011osgi/runtime-New_configuration/TomcatJava/bin"
>>>>>>> size="30" usedBy="8" usesInternal="11" usesExternal="14" layer="6"
>>>>>>> cycle="org.apache.catalina et al.">
>>>>>>> <packageRef name="org.apache.catalina.core" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.startup" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.deploy" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.authenticator"
>>>>>>> type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.mbeans" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.connector" type="usedBy"/>
>>>>>>> <packageRef name="org.apache.catalina.realm" type="usedBy"/>
>>>>>>> <packageRef name="java.lang" type="usesExternal"/>
>>>>>>> <packageRef name="java.io" type="usesExternal"/>
>>>>>>> <packageRef name="org.apache.catalina.deploy" type="usesInternal"/>
>>>>>>> <packageRef name="org.apache.catalina.util" type="usesInternal"/>
>>>>>>> <packageRef name="java.util" type="usesExternal"/>
>>>>>>> <packageRef name="org.apache.juli.logging" type="usesInternal"/>
>>>>>>> <packageRef name="org.apache.tomcat.util.res" type="usesInternal"/>
>>>>>>> <packageRef name="java.beans" type="usesExternal"/>
>>>>>>> <packageRef name="(default package)" type="usesExternal"/>
>>>>>>> <packageRef name="org.apache.catalina" type="usesInternal"/>
>>>>>>> <packageRef name="org.apache.catalina.mbeans" type="usesInternal"/>
>>>>>>> <packageRef name="javax.management" type="usesExternal"/>
>>>>>>> <packageRef name="namingResources" type="usesExternal"/>
>>>>>>> <packageRef name="javax.naming" type="usesExternal"/>
>>>>>>> <packageRef name="org.apache.naming" type="usesInternal"/>
>>>>>>> <packageRef name="java.lang.reflect" type="usesExternal"/>
>>>>>>> <packageRef name="javax.servlet" type="usesInternal"/>
>>>>>>> <packageRef name="javax.servlet.annotation" type="usesInternal"/>
>>>>>>> <packageRef name="org.apache.catalina.order" type="usesExternal"/>
>>>>>>> <packageRef name="java.net" type="usesExternal"/>
>>>>>>> <packageRef name="webXml" type="usesExternal"/>
>>>>>>> <packageRef name="webXml.version" type="usesExternal"/>
>>>>>>> <packageRef name="webxml" type="usesExternal"/>
>>>>>>> <packageRef name="org.apache.catalina.core" type="usesInternal"/>
>>>>>>> <packageRef name="javax.servlet.descriptor" type="usesInternal"/>
>>>>>>> </package>
>>>>>>>
>>>>>>> usedBy means who use current package;
>>>>>>> usesInternal means which packages current package use in current
>>>>>>> application source code;
>>>>>>> usesExternal means which packages current pakcage use not in current
>>>>>>> application's source code.
>>>>>>>
>>>>>>> Now, we start split the whole application in to bundles, according to
>>>>>>> above algorithm source code analyse algorithm, we can also know
>>>>>>> package cycles in current application.
>>>>>>>
>>>>>>> 1. Treat each package cycle as a single bundle;
>>>>>>> 2. Treat each packages not in cycle as a single bundle;
>>>>>>> 3. Then we should decide which bundles can be merged into one new
>>>>>>> bundle;
>>>>>>>
>>>>>>> First round merge job:
>>>>>>> 4. If one bundle's all usesInternal elements are in the other bundle,
>>>>>>> these two bundles should be merged into a new bundle;
>>>>>>>
>>>>>>> Think about how to merge used by only bundle (bundle be used only by
>>>>>>> other bundle, it does not rely on any other bundle):
>>>>>>> Define two variable for used by only bundle:
>>>>>>> sameUB: it means the number of two bundles have same usedBy elements.
>>>>>>> sameUE: it menas the number of two bundles have same usedExternal
>>>>>>> external package rely elements.
>>>>>>>
>>>>>>> boolean condition1 = 4 * sameUB >= one.usedByList.size() +
>>>>>>> two.usedByList.size();
>>>>>>> boolean condition2 = 2 * sameUE >= one.usedExternalList.size() +
>>>>>>> two.usedExternalList.size();
>>>>>>>
>>>>>>> if condition1 or condition2 is true, we should merge these two usedBy
>>>>>>> only bundle.
>>>>>>>
>>>>>>> Merge the other bundles:
>>>>>>> 5.The core problem is how to decide two bundles(bundle one and bundle
>>>>>>> two) can be merged or not.
>>>>>>>
>>>>>>> Define 5 variables:
>>>>>>> uiNumber: the sum of bundle one's usesInternal elements in bundle two
>>>>>>> number and bundle two's usesInternal elements in bundle one number;
>>>>>>>
>>>>>>> ubNumber:the sum of bundle one's usedBy elements in bundle two number
>>>>>>> and bundle two's usedBy elements in bundle one number;
>>>>>>>
>>>>>>> sameUI: the same usesInternal number bundle one and two have
>>>>>>>
>>>>>>> sameUB: Be similar with used by only bundle's this variable
>>>>>>>
>>>>>>> sameUE: Be similar with used by only bundle's this variable
>>>>>>>
>>>>>>> Define these conditions:
>>>>>>> boolean condition1 = 2 * uiNumber >= one.usesInternalList.size() +
>>>>>>> two.usesInternalList.size();
>>>>>>> boolean condition2 = 2 * ubNumber >= one.usedByList.size() +
>>>>>>> two.usedByList.size() ;
>>>>>>> boolean condition3 = 3.5 * sameUI >= one.usesInternalList.size() +
>>>>>>> two.usesInternalList.size();
>>>>>>> boolean condition4 = 4 * sameUB >= one.usedByList.size() +
>>>>>>> two.usedByList.size();
>>>>>>> boolean condition5 = 3 * sameUE >= one.usedExternalList.size() +
>>>>>>> two.usedExternalList.size();
>>>>>>>
>>>>>>> If any above condition is true, these two bundles can be merged. But
>>>>>>> these are another problem, if bundle A can be merged with B, but it
>>>>>>> also can be merged with C, now, we should decide merge A with B or A
>>>>>>> with C.
>>>>>>>
>>>>>>> Define the follow attribute:
>>>>>>> int mergeFactor= 2 * (uiNumber + ubNumber) + 0.4 * (sameUI + sameUB) +
>>>>>>> 0.2 * sameUE - number;
>>>>>>>
>>>>>>> If A and B's mergeFactor is 20 and A and C's mergeFactor is 30, we
>>>>>>> should merge A and C.
>>>>>>>
>>>>>>> This is current merge algorithm in OSGiMaker, i will keep on improving
>>>>>>> it, use class relationship factors or etc. You can find the source
>>>>>>> code detail of this algorithm in class AnalyseJob of our project.
>>>>>>>
>>>>>>> The attach file is the analyse result document OSGiMaker analyse
>>>>>>> Tomcat's source code and split it into bundles, you can have a review.
>>>>>>>
>>>>>>>
>>>>>>> In fact, i did not want to bother you too much, but it seems that you
>>>>>>> have enough time to help me to improve it, this is a good thing. If
>>>>>>> you have any advises, please let me know, let's improving it together
>>>>>>> :-)
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>> [1]
>>>>>>> http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm
>>>>>>>
>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>>>> Well, I do not think I am eager but I have a hard time getting a
>>>>>>>> feeling where you are. You do not have to send reports, I want to see
>>>>>>>> intermediate results and discuss issues if there are any. As I said
>>>>>>>> earlier, it is not clear to me yet what the best algorithm is so that
>>>>>>>> need to be worked out before we do the gui stuff.
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>>
>>>>>>>> Peter Kriens
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26 jul 2011, at 11:24, Tiger Gui wrote:
>>>>>>>>
>>>>>>>>> In my schedule, i will report current status to you tomorrow as i
>>>>>>>>> think i can get a usable version today, the whole analyse and split
>>>>>>>>> algorithm is complex i have to organize a document to describe it
>>>>>>>>> clearly. As you are really eager to see its progress, it is OK, i will
>>>>>>>>> start the report now
>>>>>>>>>
>>>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>>>>>> If that is the case you have to provide more on going feedback. What
>>>>>>>>>> happened to the analysis by hand?
>>>>>>>>>>
>>>>>>>>>> Kind regards,
>>>>>>>>>>
>>>>>>>>>> Peter Kriens
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 26 jul 2011, at 11:02, Tiger Gui wrote:
>>>>>>>>>>
>>>>>>>>>>> No, Peter, i am really working hard for this project, you can check
>>>>>>>>>>> the progress here[1]. Now, this tool can analyse a whole project and
>>>>>>>>>>> export the its analyse result to bundles( I test it in Spring and
>>>>>>>>>>> Tomcat project), if possible, you can install it in your Eclipse and
>>>>>>>>>>> have a trial of it. But, i am still improving the split algorithm as
>>>>>>>>>>> the current algorithm is not working perfect.
>>>>>>>>>>>
>>>>>>>>>>> I will supply a document about the current status of this project
>>>>>>>>>>> and
>>>>>>>>>>> a simple guide for you to have a trial of it. I am really working
>>>>>>>>>>> very
>>>>>>>>>>> hard for it these days :-(
>>>>>>>>>>>
>>>>>>>>>>> [1] http://code.google.com/p/osgimaker/updates/list
>>>>>>>>>>>
>>>>>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>>>>>>>> I am getting the feeling that you're not working very hard on this
>>>>>>>>>>>> project and only does something just for the evaluations ...
>>>>>>>>>>>>
>>>>>>>>>>>> Peter Kriens
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards
>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards
>>>>>>>>> ----------------------------------------------------
>>>>>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> ----------------------------------------------------
>>>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>>>> <TomcatJava.analyse>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> ----------------------------------------------------
>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>> <bundles_relation.txt><SpringSplitTest.analyse>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> ----------------------------------------------------
>>> Tiger Gui [tigergui1...@gmail.com]
>>> <SpringSplitTest.html>
>>
>>
>
>
>
> --
> Best Regards
> ----------------------------------------------------
> Tiger Gui [tigergui1...@gmail.com]
>
--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
