Hi Ruth,
I do not know exact data set size, as we are in development stage at the
moment. What I know is that we need to prepare for:
- unknown no of categories (tests made on 100 000 - then the error
eccured)
- at least 2*suppliers number catalogs
- around 200 different manufacturers
- around 10,20 suppliters
- at the moment I am working with fetures, it's possible that I will
split number of categories.
My market is automotive industry. As you know, parts for such a
companies are very well categorised. Those criteria basis on:
a) Car selection
1)Make
2)Model
3)Year
4)Engine
5)Body Type
b) Part selection
-)Number
-)Placement
-)Vendor
-)Price
-)Supplier
-)Location
-)Material
1)Type (i.e. engine parts)
2)Sub-type (i.e bearings)
3)Part (i.e. crankshaft bearing)
4)Part instances
+ information about replacements
+ information about related parts
So this is only regarding categorisation. At the beggining I wanted to
put it all in one tree ( without criteria marked by "-)"). Just like in
www.rockauto.com
Now I do think about putting there more criteria to allow users browse
for what do they need. See example:
www.autopartswarehouse.com
Part catalog information will be integrated next year (soonest). Now we
need to prepare Phase0 project. As you see, with such amount of data we
need to fix issues that do not bother when playing with medium size shop
selling clothes.
Anyway I think that I could improve my working solution and pass it to
you. It will be easier for maintance for myself. Thats why my question
was regarding improvement of my code.
Hope this brings some light on the topic.
Dnia 2010-08-17, wto o godzinie 11:33 -0400, Ruth Hoffman pisze:
> Hi Micha:
>
> Could I ask how big are your data sets? The reason I ask is that
> several
> years ago, prior to 9.04 I had an out-of-memory problem that was
> never
> resolved. The database tables that I was querying had between 4 - 12
> million records. My database (and OFBiz usage of the same) went away
> before I couldn't do any further investigation into the problem.
>
> So, I'm still curious about OFBiz support for really large data sets.
>
> Regards,
> Ruth
>
> Michał Cukierman wrote:
> > Thanks for the quick answer.
> >
> > I have allocated 2GB of memory, so it's not a case. The problem is
> that
> > I work on huge data sets. Version is 10.4 but 9.04 is has the same
> code.
> > I have ommited those informations 'couse as you see - it's not
> relevant
> > to the topic.
> >
> > I don't understand your last assumption. Maybe it will someting in
> my
> > understanding?
> >
> > For now I think the best would be to improve the code I have
> proposed as
> > it fixes:
> > 1) My issues
> > 2) Coding way of selecting everything to get a couple of categories.
> >
> > So, thats why I asked about:
> >
> >
> >> 1) How to implement Right/Left joins on DynamicViewEntity
> >>
> > (I was able to find out how to creat inner join only)
> >
> >>> 2) How to implement my whereString
> >>>
> > (Haven't found in a code such a use case)
> >
> >
> > Regards,
> > Michał
> >
> >
> >
> >
> > Dnia 2010-08-17, wto o godzinie 07:54 -0700, BJ Freeman pisze:
> >
> >> as far as performance how much memory have you allocated to ofbiz?
> >> what version of ofbiz are you using?
> >> There is a top Category that should be checked for first.
> >> have not been in that part of the code for a while.
> >>
> >> Michał Cukierman sent the following on 8/17/2010 7:41 AM:
> >>
> >>> Hi,
> >>>
> >>> I have a problem with performence (and Out of Memmory Issue) with
> >>> catalog management module. There is a section invoked after
> >>> Choose Top Category
> >>>
> >>> Collection<GenericValue> allCategories =
> >>> delegator.findList("ProductCategory", null, null, null, null,
> false);
> >>> for (GenericValue curCat: allCategories) {
> >>> Collection<GenericValue> parentCats =
> >>> curCat.getRelatedCache("CurrentProductCategoryRollup");
> >>>
> >>> if (parentCats.isEmpty())
> >>> results.add(curCat);
> >>> }
> >>>
> >>> As we see. To get categories without parents we need to fetch for
> all
> >>> categories and for each of them, find a parent.
> >>>
> >>> I do not know Ofbiz api very well for now, so I have implemented:
> >>> EntityWhereString where =
> >>>
> EntityCondition.makeConditionWhere("PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID
> >>> not in" +
> >>> " (select PRODUCT_CATEGORY_ROLLUP.PRODUCT_CATEGORY_ID from
> >>> PRODUCT_CATEGORY_ROLLUP)");
> >>> Set<String> fieldsToSelect = FastSet.newInstance();
> >>> fieldsToSelect.add("productCategoryId");
> >>> results.addAll(delegator.findList("ProductCategory",
> >>> where,
> >>> fieldsToSelect, null, null, false));
> >>>
> >>>
> >>> I simply do invoke a SQL query in the database:
> >>>
> >>> select PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID from PRODUCT_CATEGORY
> where
> >>> PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID not in
> >>> (select PRODUCT_CATEGORY_ROLLUP.PRODUCT_CATEGORY_ID from
> >>> PRODUCT_CATEGORY_ROLLUP);
> >>>
> >>> Is there a way ( for sure is ) to implement it in a better manner?
> For
> >>> now it works on my system, but after improving the query it could
> be
> >>> passed to a production.
> >>>
> >>> So I got two questions in fact:
> >>>
> >>> 1) How to implement Right/Left joins on DynamicViewEntity
> >>> 2) How to implement my whereString
> >>>
> >>>
> >>> Thank you in advance,
> >>>
> >>>
> >>>
> >
> >
> >
>
