Scrapy shell loads different result when loading from browser

[Deddy Hermanus]

Hello. I am trying to scrap an online portal shopping site that's heavily 
uses Ajax. 
I manage to find the ajax link to get the data from a page that should be 
showing product directory. It's showing correctly on my chrome browser
however when I loaded the link through my scrapy shell the response data is 
empty. 

The ajax url call is :


https://ace.tokopedia.com/search/product/v3?full_domain=www.tokopedia.com&scheme=https&device=desktop&source=directory&page=1&fshop=1&rows=25&sc=8&start=0&ob=23&callback=angular.callbacks._0

Response  Browser :

{"header":{"total_data":795009,"process_time":0.020266475000000003},"data":{"redirection":{"redirect_url":"","department_id":0},"suggestions":{"suggestion":"","total_data":0},"catalogs":[],"products":[{"id":23129326,"name":"Barrons
 TOEFL iBT 13th Edition with 
CD","url":"https://www.tokopedia.com/bukutoefl/barrons-toefl-ibt-13th-edition-with-cd","image_url":"https://ecs7.tokopedia.net/img/cache/200-square/product-1/2016/1/4/23129326/23129326_ec4625c5-b25f-459a-8a61-c0486d82ca06.png","image_url_700":"https://ecs7.tokopedia.net/img/cache/700/product-1/2016/1/4/23129326/23129326_ec4625c5-b25f-459a-8a61-c0486d82ca06.png","price":"Rp
 150.000","shop":{"id":211477,"name":"Toko Buku 
TOEFL","url":"https://www.tokopedia.com/bukutoefl","is_gold":true,"location":"Malang","city":"Kota
 
Malang","reputation":"https://inbox.tokopedia.com/reputation/v1/badge/shop/211477","clover":"https://clover.tokopedia.com/badges/merchant/v1?shop_id=211477"},"wholesale_price":[],"condition":1,"department_id":889,"labels":[],"badges":[{"title":"Free
 
Return","image_url":"https://hermes.tokopedia.com/v1/prosecure/badge?product_id=23129326"},{"title":"Gold
 
Merchant","image_url":"https://ecs7.tokopedia.net/img/gold-active-large.png"}]},{"id":68278733,"name":"BCL
 MAIN YOUTUBE | Kursus Online Rahasia Youtube Adsense 
Sukses","url":"https://www.tokopedia.com/tokoselly/bcl-main-youtube-kursus-online-rahasia-youtube-adsense-sukses","image_url":"https://ecs7.tokopedia.net/img/cache/200-square/product-1/2016/9/22/12350568/12350568_34946969-a563-4a87-adb8-c58c242fd07f.png","image_url_700":"https://ecs7.tokopedia.net/img/cache/700/product-1/2016/9/22/12350568/12350568_34946969-a563-4a87-adb8-c58c242fd07f.png","price":"Rp
 13.950","shop":{"id":1464211,"name":"Pusat Produk 
Bagus","url":"https://www.tokopedia.com/tokoselly","is_gold":false,"location":"Jakarta","city":"Jakarta","reputation":"https://inbox.tokopedia.com/reputation/v1/badge/shop/1464211","clover":"https://clover.tokopedia.com/badges/merchant/v1?shop_id=1464211"},"wholesale_price":[],"condition":1,"department_id":844,"labels":[],"badges":[{"title":"Free
 
Return","image_url":"https://hermes.tokopedia.com/v1/prosecure/badge?product_id=68278733"}]},{"id":49096045,"name":"TEEMASTERY
 - RAHASIA PROFIT MILYARAN JUALA. KAOS 
!","url":"https://www.tokopedia.com/marketdigital/teemastery-rahasia-profit-milyaran-juala-kaos","image_url":"https://ecs7.tokopedia.net/img/cache/200-square/product-1/2016/6/19/595266/595266_ab23c9ce-1a7d-48e5-a6ad-e1e054453f98.jpg","image_url_700":"https://ecs7.tokopedia.net/img/cache/700/product-1/2016/6/19/595266/595266_ab23c9ce-1a7d-48e5-a6ad-e1e054453f98.jpg","price":"Rp
 
35.000","shop":{"id":595266,"name":"marketdigital","url":"https://www.tokopedia.com/marketdigital","is_gold":false,"location":"Bandung","city":"Kota
 
Bandung","reputation":"https://inbox.tokopedia.com/reputation/v1/badge/shop/595266","clover":"https://clover.tokopedia.com/badges/merchant/v1?shop_id=595266"},"wholesale_price":[],"condition":1,"department_id":844,"labels":[],"badges":[{"title":"Free
 
Return","image_url":"https://hermes.tokopedia.com/v1/prosecure/badge?product_id=49096045"}]},{"id":7079115,"name":"BILINGUAL
 BOOK MURAH FOR 
TODDLER","url":"https://www.tokopedia.com/irenesstore/bilingual-book-murah-for-toddler","image_url":"https://ecs12.tokopedia.net/newimg/cache/200-square/product-1/2014/12/3/78859/78859_16a6bb42-7ad5-11e4-968e-18ee4908a8c2.jpg","image_url_700":"https://ecs12.tokopedia.net/newimg/cache/700/product-1/2014/12/3/78859/78859_16a6bb42-7ad5-11e4-968e-18ee4908a8c2.jpg","price":"Rp
 4.600","shop":{"id":78859,"name":"irene\u0026#39;s 
store","url":"https://www.tokopedia.com/irenesstore","is_gold":false,"location":"Jakarta","city":"Jakarta","reputation":"https://inbox.tokopedia.com/reputation/v1/badge/shop/78859","clover":"https://clover.tokopedia.com/badges/merchant/v1?shop_id=78859"},"wholesale_price":[],"condition":1,"department_id":758,"labels":[],"badges":[{"title":"Free
 
Return","image_url":"https://hermes.tokopedia.com/v1/prosecure/badge?product_id=7079115"}]},{"id":39490864,"name":"Novel
 Diorama (Ari Nur) - Diorama Sepasang 
Albanna","url":"https://www.tokopedia.com/frenari/novel-diorama-ari-nur-diorama-sepasang-albanna","image_url":"https://ecs7.tokopedia.net/img/cache/200-square/product-1/2016/11/29/39490864/39490864_208bc427-36f1-41d0-a009-38384bed7832_391_461.jpg","image_url_700":"https://ecs7.tokopedia.net/img/cache/700/product-1/2016/11/29/39490864/39490864_208bc427-36f1-41d0-a009-38384bed7832_391_461.jpg","price":"Rp
 
45.000","shop":{"id":318613,"name":"Frenari","url":"https://www.tokopedia.com/frenari","is_gold":false,"location":"Yogyakarta","city":"Kota
 
Yogyakarta","reputation":"https://inbox.tokopedia.com/reputation/v1/badge/shop/318613","clover":"https://clover.tokopedia.com/badges/merchant/v1?shop_id=318613"},"wholesale_price":[],"condition":1,"department_id":752,"labels":[],"badges":[{"title":"Free
 
Return","image_url":"https://hermes.tokopedia.com/v1/prosecure/badge?product_id=39490864"}]}]}}



Response from scrapy shell

{"header":{"total_data":0,"process_time":0},"data":{"redirection":{
"redirect_url":"","department_id":0},"suggestions":{"suggestion":"",
"total_data":0},"catalogs":[],"products":[]}}


Note: I've changed my user agent ID to be the same as my browser and also 
sett my ROBOTSTXT_OBEY=False setting to false. 


Any help is appreciated. Thanks.


-- 
You received this message because you are subscribed to the Google Groups 
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.