Froogle Data Export
-------------------
Key: OFBIZ-276
URL: http://issues.apache.org/jira/browse/OFBIZ-276
Project: OFBiz (The Open for Business Project)
Issue Type: New Feature
Components: product
Affects Versions: SVN trunk
Reporter: Marco Risaliti
Priority: Minor
Attached is a _very_ rough outline of how I think a froogle export would go.
This needs a lot of work still, but I'm out of time, so hopefully someone else
can take it up.
All Comments Work Log Change History Sort Order:
Comment by Andrew Sykes [16/May/06 01:43 PM] [ Permlink ]
A rough outline
Comment by rohit [17/May/06 04:44 AM] [ Permlink ]
hi,
i am very new to java and ofbiz. i have been using the oscommerce shopping cart
for more than 3 years, but fell for the amazing functionality that ofbiz
provides. Though oscommerce performs a very small part of the what ofbiz can it
still has some very useful and essential functions for webmaster and web
marketeers. i am attaching a froogle data exporter from oscommerce site. The
file works independent of the oscommerce installation, i mean one need not have
oscommerce installed to get it working. It works excellently and i used to set
a cron object such that the file was run once a week.
i hope this file will give an idea of how to develop a froogle data exporter
for ofbiz. if i am not wrong we can also tweek this file to retreive the data
from ofbiz database and export to froogle. this might be a temporary solution
and may require one to have php installed, but i guess it will save valuable
time.
any comments or suggestions are welcome.
rohit
Comment by rohit [17/May/06 04:49 AM] [ Permlink ]
if one needs more information about the file attached, please visit
http://www.oscommerce.com/community/contributions,3876/category,all/search,froogle.
oscommerce is a basic shoppinh cart, but with a very vibrant community, the
feature and modules are very useful to any webmaster. if ofbiz could include
some ideas from oscommerce, it could improve even better.
rohit
Comment by Andrew Sykes [17/May/06 06:39 AM] [ Permlink ]
Rohit,
This would be a VERY messy approach, and I'm not entirely convinced it would
even work with OFBiz
The code I've supplied is more than half of what is required to get a froogle
export working.
I have set it up as a service as that allows it to be scheduled using the
JobScheduler.
If you are looking for somewhere to start with OFBiz code, this is as good a
place as any :-)
Comment by David Yoon [17/May/06 01:19 PM] [ Permlink ]
Hi guys, I'm also working on this issue, although I'm making something a bit
more generic so you could add multiple comparison shopping sites. So far I've
configured it to work (update, export, upload) for froogle, and shopAmex.
Instead of recursing through all the product_categories, I was thinking of
using the inventory_item table as the base, since most of the cs sites want the
products that you upload to be available. currently I'm flattening the product,
inventory_item, category, price, tables into a single export_product table
which contains all the fields necessary for the various cs feed sites. The
update for this table is taking quite a long time though (like 30 mins, since
there are 22k products in the database). Any suggestions?
d.
Comment by Andrew Sykes [17/May/06 04:39 PM] [ Permlink ]
David,
I'm not sure it's a good idea to use the inventory_item table as it's not
actually necessary to use inventory management to run an OFBiz ecommerce site.
Some users just won't care about inventory (e.g. JIT ordering). While others
will have a thirdparty application for inventory management.
Andrew
Comment by Si Chen [18/May/06 11:07 AM] [ Permlink ]
Hey everybody -
I'd like to help. Sorry I got to the party late. We did this once before too,
but it got entombed in some custom code that's no longer worth extracting. Here
is what I'd suggest:
1. Andrew' approach is mostly correctly, but I'd rather use productStoreId than
webSiteId. Andrew, you are actually using webSiteId to get productStoreId, but
productStoreId is more direct. Also very important: you need to pull all your
variant products as well, which I don't see, or people would miss most of your
real physical products. I definitely don't think InventoryItem is what you
should be using, by the way.
2. You'll need to configure some links for the products and probabily also an
"atc=___" tag for tracking marketing campaigns (so you know if froogle or
whatever actually made you sales!) I'd suggest a .properties file.
3. The best way to render the output is actually using the content manager to
create a FTL template of froogle's layout and then render against data. Look at
the DhlServices.java or the GLExportServices in our Financials module. It's
much easier this way than hardcoding formats into Java and more maintainable as
well should the formats change (as they inevitably do.)
Let me know if you need any help...
Comment by Andrew Sykes [18/May/06 12:15 PM] [ Permlink ]
The reason for using websiteId is that you need it further down to create the
links properly. Some comments in my code would have helped here :-)
I'm not sure that people would necessarily want to spam froogle with all their
variants, (for example, imagine a clothes store). Also, variants do not
necesssarily belong to any category, so that might lead to problems.
I totally agree about rendering, again I apologise for not adding this to my
comments.
Comment by Si Chen [18/May/06 12:37 PM] [ Permlink ]
Andrew,
I think you're right about the websiteId then. So you set the URL as well in
WebSite, like in the standardContentPrefix field?
Why don't we make variants/no variants a flag so people can choose?
No problem about the comments.
Anybody want to work on this and submit one we can put into OFBiz?
Si
Comment by Si Chen [18/May/06 12:41 PM] [ Permlink ]
Rohit,
So here's a question about your PHP file, totally off topic, but I'm just
curious about osCommerce: How does this patch work? You put it somewhere, and
somebody downloads the main osCommerce and then this patch? So it's tested with
a particular module...but not necessarily others? Do you just put it in your
www/ directory and it runs?
Now, what about licensing--there's no license terms in the header, so what do
we assume it's licensed under? GPL? Apache?
Please don't get me wrong--I'm not saying it's bad--I'm just curious.
Thanks,
Si
Comment by Ray Barlow [18/May/06 12:53 PM] [ Permlink ]
I have already done some mods to the version Andrew posted, and thrown a screen
in front of it. I had registered a Froogle account the weekend before with a
view to working on this so the whole thread has come at a good time.
The mods I've made already include output created using the DataFile class so I
won't rush to change that to FTL rendering. I'll try and polish a little more
in the next few days and then post it back here although I doubt that it will
be fit for commiting at that stage.
Comment by rohit [18/May/06 12:58 PM] [ Permlink ]
Hi Si,
Thanks for asking. Just to give you an idea Oscommerce is an open source
shopping cart based on PHP. The software and its modules like the file attached
are under GLP license.
Oscommerce has a basic module and the community members can create new
functionality and share with other. The froogle exporteer file is almost
independed on oscommerce files, though it depends on the database structure of
oscommerce. One can simply upload the file to a webserver and set a cron object
for it to run weekly. that is it.
i mentioned this file, to explore the possibility of something like this for
ofbiz, something thats very easy to install and run and that new users as well
as existing users of ofbiz can easy use for there websites.
i have using oscommerce for 3 years and believe me it does not do even 10% of
what ofbiz can do, but the ability of community to develop new contributions
such as froogle exporter, etc and the relative ease of integrating these
contributions into existing webstores, is where i guess oscommerce takes the
lead. i am still learning ofbiz and i hope i am proved wrong. And i did like to
add that you must visit oscommerce.com and take a look on how the 100K
community members have contributed to it.
Rohit
Comment by Andrew Sykes [18/May/06 01:49 PM] [ Permlink ]
I'm glad to hear that someone has been spurred into action by my contribution,
thanks for looking into this Ray. You may want to contact Ian Gilbert (if you
haven't already) as he was also planning to do something with it, perhaps you
guys could collaborate?
I think Si's suggestion of an optional include variants (may) be useful - can
you give an example scenario please Si? Also one thing to consider, there is a
filter category (whose name escapes me) that specifies which products to show
on a site, would this mess up variant posting?
I think it would also be good to add an optional categoryId parameter to the
service definition and a recurse[Y/N] flag. This would mean that without a
categoryId posting would start from the browse root, with a categoryId posting
would start from the specified category and if recurse==N posting would be
exclusively for that category.
In my additional comments on the user list, I suggested using the DataFile
stuff, as that was the first thing that sprung to mind, I really don't see any
objection to carrying on down that route, although someone else might know
different...
Has anyone considered whether setting the Froogle category in the CategoryAttr
is a good approach, I'd be interested to hear any comments on this...
Also, what about the FTP part, it would be nice if OFBiz handled that directly,
anyone have any suggestioins?
Finally if FTP is integrated, we need somewhere to store the FTP address and
credentials - any suggestions?
Comment by Si Chen [18/May/06 02:22 PM] [ Permlink ]
Hey -
1. Why variants: if someone searches for "white shirt" on froogle and all you
had was the virtual "shirt" but not variants, you might not get listed.
2. No idea about the filter category.
3. Recurse - yeah probably a good idea too.
4. I still think the content template + render is better than DataFile, but I
could be convinced otherwise.
5. I think there are some FtpServices.java in ofbiz now. Not sure what it does
though.
Comment by Andrew Sykes [18/May/06 02:43 PM] [ Permlink ]
Si,
Sure that could be an issue, but would you really want to send
5 colours x 5 neck sizes x 5 chest sizes x 10 braiding colours
= 1250 products
(ok, a bit excessive, but you get the point I'm sure)
I'm guessing froogle would probably take quite a dim view of this especially if
they all had the same product image and description.
Also, having done a few clothing sites, they never have products this generic,
because...
A/ They'd never get a google rank
B/ The retailer would always find a more descriptive product name.
"White Shirt" is more likely to be a category name.
I've been trying to think of a scenario where variants would be useful, but I
just don't think product names would ever be generic enough to need this.
Comment by Si Chen [18/May/06 02:47 PM] [ Permlink ]
That's why it's a configurable parameter, so if you don't want it, you don't
have to use it. It's OK if you don't put it in--I can add it later if we decide
to use it.
Also: FTP address and credentials - use a .properties file in a config/
directory. That's how the freight shipping and payment processors are handled.
Comment by Si Chen [18/May/06 03:02 PM] [ Permlink ]
Rohit,
Thanks for your comments. I actually do know of oscommerce but wasn't quite
sure how it worked. Thanks for explaining it.
Well, we are definitely a community project as well and do indeed accept
contributions from a fairly large number of people. I think the difference you
find is not only a greater set of features but also a more formal framework and
process for building applications. Hopefully this will work for you. All this
stuff may seem intimidating at first, but you can definitely learn it (if I can
learn it, anybody can learn it), and once you know it, it will make your work
very easy.
Ok, end of my digressions.
Si
Comment by Ray Barlow [21/May/06 06:03 AM] [ Permlink ]
DataFile.patch adds support to write a DataFile with a header row.
Comment by Ray Barlow [21/May/06 06:10 AM] [ Permlink ]
I have added a menu on the catalog application called "DataIO" next to
"Configurations" and in there is a new screen for Froogle.
The screen contains comments about what is still required to be done which to
be honest still includes a fair amount to be a really cool implementation, but
this submission will at least generate the Froogle file. I've only run on the
demo data so far but will deploy it on one of my live sites asap and test
generating and submitting to Froogle for real.
Although not complete if this is acceptable then it would be nice to commit a
version sooner rather than wait for the holy grail of FTP automation etc.
Comment by rohit [21/May/06 10:28 PM] [ Permlink ]
Hi,
Can you please give some instructions on how to install it. I will surely test
it and give my feedback.
rohit
Comment by Ray Barlow [22/May/06 06:59 AM] [ Permlink ]
The two files I attached are created as SVN patch files. Depending on your
platform and tool depends on how you apply them and to be honest applying
patches is not something I've done a lot of. For Windows if your using tortoise
SVN (which is very good) try right click on the root folder and "Apply Patch"
from the sub menu. For the command line I think you'd use the "patch" command,
but for both I'd suggest a test folder and google would be a good option.
On a more general note I forgot to mention that I saw the "DataIO" section
being an area for control of other features being talked about like Yahoo,
Amazon and maybe even some simple catalogue data import/export routines.
Comment by rohit [24/May/06 09:23 AM] [ Permlink ]
hi Ray,
i am sorry but i could not get it working. i was initially using opentaps-0.9.0
but then tried it on SVN using the tortoise SVN client.
i will really appreciate if you could be more specific about how you went about
installing. For. e.g. in which folders are these files to be copied into and if
there is any code that needs to be edited manually.
i am new to ofbiz, hence a little more detailed instructions will help a lot.
thank,
rohit
Comment by rohit [24/May/06 11:43 PM] [ Permlink ]
hi Ray,
i was able to get the files install, but i am getting the following error, when
i try to use the froogle exporter:
3057047[ServiceEventHandler.java:317:ERROR]
---- exception report
----------------------------------------------------------
Service invocation error
Exception: org.ofbiz.service.GenericServiceException
Message: Service target threw an unexpected exception (null)
---- stack trace
---------------------------------------------------------------
org.ofbiz.service.GenericServiceException: Service target threw an unexpected
exception (null)
org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:113)
org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:534)
java.lang.NullPointerException
org.ofbiz.product.product.ProductExportServices.productsToCsv(ProductExportServices.java:73)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:324)
org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:101)
org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:534)
--------------------------------------------------------------------------------
3057047[ RequestHandler.java:283:ERROR] Request froogleExport caused an error
with the following message: Error calling event:
org.ofbiz.webapp.event.EventHandlerException: Service invocation error (null)
3057047[ RequestHandler.java:395:INFO ] [RequestHandler.doRequest]: Response is
a view.
3057047[ RequestHandler.java:547:INFO ] servletName=control, view=Froogle
3057109[ UtilXml.java:242:DEBUG] XML Read 0.062s:
file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
3057109[ ScreenFactory.java:120:INFO ] Got 2 screens in 0.062s from:
file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
3057281[ UtilXml.java:242:DEBUG] XML Read 0.156s:
file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
3057281[ ScreenFactory.java:120:INFO ] Got 13 screens in 0.156s from:
file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
3057328[ UtilXml.java:242:DEBUG] XML Read 0.031s:
file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
3057344[ ScreenFactory.java:120:INFO ] Got 8 screens in 0.047s from:
file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
3057578[ UtilXml.java:242:DEBUG] XML Read 0.047s:
file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
3057578[ ScreenFactory.java:120:INFO ] Got 9 screens in 0.047s from:
file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
3057750[ ServiceDispatcher.java:452:DEBUG] [[Sync service finished-
total:0.0,since last(Begin):0.0]] - 'catalog /
getProductCategoryAndLimitedMembers'
3057812[ ControlServlet.java:258:DEBUG] [[[froogleExport] Done rendering page,
Servlet Finished- total:1.125,since last([froogleExport] S...):1.125]]
3061609[ ControlServlet.java:98 :INFO ] The character encoding of the request
is: [null]. The character encoding we will use for the request and response is:
[UTF-8]
3061625[ ControlServlet.java:134:DEBUG] [[[froogleExport] Servlet Starting,
doing setup- total:0.0,since last(Begin):0.0]]
3061625[ ControlServlet.java:183:DEBUG] [[[froogleExport] Setup done, doing
Event(s) and View(s)- total:0.0,since last([froogleExport] S...):0.0]]
3061625[ RequestHandler.java:212:INFO ] [Processing Request]: froogleExport
3061672[ ServiceDispatcher.java:386:DEBUG] [[Sync service failed...-
total:0.0,since last(Begin):0.0]] - 'catalog / froogleExport'
3061687[ ServiceDispatcher.java:389:ERROR]
Please advise what might be wrong.
rohit
Comment by Ray Barlow [05/Jun/06 12:17 PM] [ Permlink ]
Attached is an improved version that has a little more error checking and also
uses drop down lists to reduce user entry errors, for data such as webSiteId.
The NPE exception Rohit was experiencing would have been due to an invalid
webSiteId entry.
Also added support for tracking codes if defined in the marketing module and a
check to skip null images for products.
It would be nice to see this reviewed and committed if possible. DataFile patch
still stands as submitted before as I've not changed anything in it that for
this update.
Comment by rohit [21/Jul/06 01:58 AM] [ Permlink ]
hi,
i have attached a new patch file to work with nre requirement of google base
format. This patch has been created on the latest SVN 424158.
Comment by rohit [21/Jul/06 02:00 AM] [ Permlink ]
This patch has all the changes. you need only this file to implement froogle on
ofbiz.
Comment by Marco Risaliti [13/Sep/06 04:12 PM] [ Permlink ]
May I have to move it to the new jira server ?
Thanks
Marco
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
