Arni Sumarlidason created NUTCH-1481:
----------------------------------------
Summary: When using MySQL as storage unicode characters within
URLS cause nutch to fail
Key: NUTCH-1481
URL: https://issues.apache.org/jira/browse/NUTCH-1481
Project: Nutch
Issue Type: Bug
Components: crawldb
Affects Versions: 2.1
Environment: mysql 5.5.28 on centos
Reporter: Arni Sumarlidason
MySQL's (innodb) primary key / unique key is restricted to 767 bytes..
currently the url of a web page is used as a primary key in nutch storage.
when using latin1 character set on the 'id' column @ length 767
bytes/characters; unicode characters in urls cause jdbc to throw an exception,
java.io.IOException: java.sql.BatchUpdateException: Incorrect string value:
'\xE2\x80\x8' for column 'id' at row 1
when using utf8mb4 character set on the 'id' column @ length 190 characters /
760 bytes to fully support unicode characters; the field length becomes
insufficient
It may be better to use a hash of the url as the primary key instead of the url
itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
